Update Mastodon server instance

You might have already noticed I use Mastodon for my main fediverse handle @gytis@mastodon.lt. Current version is 4.0.2.

While typically updating Mastodon is quite straightforward, sometimes you might need to iterate Ruby version on the server as well.

That was visible from error thrown:

rbenv: version `2.6.1' is not installed (set by live/.ruby-version)

So the full upgrade sequence is following:

Check dependencies

Certain versions might require specific dependencies: v.3.4.0 mentions following:

Postgres requirement raised to at least 9.5 Node.js requirement raised to at least 12

Check them:

# node --version

# psql -V
psql (PostgreSQL) 10.17 (Ubuntu 10.17-0ubuntu0.18.04.1)

Retrieve Mastodon source updates

Switch or login to user which is running Mastodon:

su - mastodon

And execute following:

cd ~/live
git fetch --tags
git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)

You might get error here:

error: Your local changes to the following files would be overwritten by checkout:
Please commit your changes or stash them before you switch branches.

In that case reset git:

git reset --hard

And try checkout again.

When updating to v4.0.2 sources need to be retrieved using following (and not git pull):

git fetch && git checkout v4.0.2

Note that above git command uses checkout with dynamic value – the most recent branch number.

Update Ruby

cd ~/.rbenv/plugins/ruby-build
git pull
cd ~/.rbenv

Check what Ruby versions are available with rbenv versions and install:

rbenv install 3.0.4

This was also documented in issue 638.

Which version of Ruby is used can be seen in file live/.ruby-version. Mastodon version 3.1.3 recommends version 2.6.6. With Mastodon upgrade to 3.2.2 we are also upgrading Ruby to 2.7.2. This version is advised with Mastodon version 3.3.0 upgrade. Version to use with Mastodon v4.0.0 is 3.0.4.

Update Mastodon

cd ~/live

Update Ruby dependencies

bundle install --deployment --without development test

Possible error

When recently updating mastodon.lt to Mastodon version 2.8.2 I got following error:

Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.

Fix it by running:

cd ~/.rbenv
gem install bundler

And go back to previous folder:

cd ~/live

Update node.js dependencies

yarn install --pure-lockfile

Yarn error upgrading to version 3.1.1

When upgrading to version 3.1.1 got this warning about Yarn:

warning Your current version of Yarn is out of date. The latest version is "1.22.0", while you're on "1.21.1".
info To upgrade, run the following command:
$ sudo apt-get update && sudo apt-get install yarn

Interestingly, I also needed to fix Yarn repository key because of error:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://dl.yarnpkg.com/debian stable InRelease: The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>
W: Failed to fetch https://dl.yarnpkg.com/debian/dists/stable/InRelease  The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>
W: Some index files failed to download. They have been ignored, or old ones used instead.

Fix it by running:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -

The minimum supported Node.js version has been bumped from 8 to 10. To upgrade Node.js, run:

curl -sL https://deb.nodesource.com/setup_10.x | bash -
apt install nodejs

move-file error when upgrading to 3.1.4

Trying to run yarn install triggered this error:

$ yarn install --pure-lockfile
yarn install v1.22.4
[1/6] Validating package.json...
[2/6] Resolving packages...
[3/6] Fetching packages...
info fsevents@2.1.3: The platform "linux" is incompatible with this module.
info "fsevents@2.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
error move-file@2.0.0: The engine "node" is incompatible with this module. Expected version ">=10.17". Got "10.16.3"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
mastodon@social:~/live$ node -v
mastodon@social:~/live$ npm --version

To upgrade Node version following was done:

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs

And then npm used:

# npm cache clean -f
npm WARN using --force I sure hope you know what you are doing.
root@social:~# npm install -g n
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
+ n@6.5.1
added 1 package from 4 contributors in 1.02s

   │                                                                │
   │       New minor version of npm available! 6.9.0 → 6.14.5       │
   │   Changelog: https://github.com/npm/cli/releases/tag/v6.14.5   │
   │               Run npm install -g npm to update!                │
   │                                                                │

# npm install -g npm
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
/usr/local/bin/npx -> /usr/local/lib/node_modules/npm/bin/npx-cli.js
+ npm@6.14.5
added 32 packages from 17 contributors, removed 23 packages and updated 78 packages in 5.903s
root@social:~# npm install -g n
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
+ n@6.5.1
updated 1 package in 0.121s

# node -v

# n latest

  installing : node-v14.5.0
       mkdir : /usr/local/n/versions/node/14.5.0
       fetch : https://nodejs.org/dist/v14.5.0/node-v14.5.0-linux-x64.tar.xz
   installed : v14.5.0 (with npm 6.14.5)

root@social:~# n lts

  installing : node-v12.18.2
       mkdir : /usr/local/n/versions/node/12.18.2
       fetch : https://nodejs.org/dist/v12.18.2/node-v12.18.2-linux-x64.tar.xz
   installed : v12.18.2 (with npm 6.14.5)

# node --version

After this yarn install --pure-lockfile succeeded.

Back to Mastodon user (su - mastodon) and install dependencies:

bundle install
yarn install

Database schema updates

Typically use:

RAILS_ENV=production bundle exec rails db:migrate

Done with migration to v.3.4.1.

Upgrading to version 3.0.x or 4.0.x:

SKIP_POST_DEPLOYMENT_MIGRATIONS=true RAILS_ENV=production bundle exec rails db:migrate

Pre-compiling updated assets

RAILS_ENV=production bundle exec rails assets:precompile

When upgrading to v.3.4.3 it was required to update browserslist:

Compiled all packs in /home/mastodon/live/public/packs
Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db

Why you should do it regularly:

Just run suggested command and you'll be good to go:

npx browserslist@latest --update-db

Clear cache

RAILS_ENV=production bin/tootctl cache clear

Restart services

Mastodon runs in memory so you will need to restart it for any of the previous updates to take effect.

Switch to root (exit) or run in sudo:

systemctl restart mastodon-*.service

Or restart services separately:

systemctl restart mastodon-web
systemctl restart mastodon-sidekiq
systemctl restart mastodon-streaming

Finish database migrations

Now that the new code is running, we can finish the database migrations. This will run the post-deployment ones:

RAILS_ENV=production bundle exec rails db:migrate

In case warning appears:

$ RAILS_ENV=production bundle exec rails db:migrate
Your database collation is susceptible to index corruption.
  (This warning does not indicate that index corruption has occured and can be ignored)
  (To learn more, visit: https://docs.joinmastodon.org/admin/troubleshooting/index-corruption/)

Stop Mastodon services, and, as documentation suggests, run following to fix:

RAILS_ENV=production bin/tootctl maintenance fix-duplicates

Restart Mastodon services once again.

Clear remote media

As remote media is cached locally, it might consume large amount of space over time.

Previously this cache could be cleaned using:

RAILS_ENV=production bundle exec rails mastodon:media:remove_remote

But it was changed to work with following command:

RAILS_ENV=production bin/tootctl media remove

On my instance it has freed quite an amount:

Removed 53385 media attachments (approx. 25.2 GB)

Database migrations after upgrade to 3.0

Now that the new code is running, we can finish the database migrations. This will run the post-deployment ones:

RAILS_ENV=production bundle exec rails db:migrate

There we go – Mastodon 2.8.0 with polls functionality running fine 😊 Try out my poll about Go code retrieval 👍


Updated 2019-05-17: added possible error when updating to version 2.8.2.

Updated 2019-06-21: added clear cache command after updating to version 2.9.0.

Updated 2019-07-08: upgrading to version 2.9.2: changed ruby version from 2.6.1 to 2.6.3:

mastodon@server:~/.rbenv$ rbenv install 2.6.3
Downloading ruby-2.6.3.tar.bz2...
-> https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.bz2
Installing ruby-2.6.3...
Installed ruby-2.6.3 to /home/mastodon/.rbenv/versions/2.6.3

Updated 2019-09-01: upgrading to version 2.9.3 changed ruby version from 2.6.3 to 2.6.4:

mastodon@server:~/.rbenv$ rbenv install 2.6.4
Downloading ruby-2.6.4.tar.bz2...
-> https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.4.tar.bz2
Installing ruby-2.6.4...
Installed ruby-2.6.4 to /home/mastodon/.rbenv/versions/2.6.4

Updated 2019-10-12: upgrading to version 3.0.1: updates Ruby version to 2.6.5:

mastodon@server:~/.rbenv$ rbenv install 2.6.5
Downloading ruby-2.6.5.tar.bz2...
-> https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.5.tar.bz2
Installing ruby-2.6.5...
Installed ruby-2.6.5 to /home/mastodon/.rbenv/versions/2.6.5

Updated 2020-02-25: upgrading to version 3.1.1: Ruby version upgrade from 2.6.5 to 2.7.0. Yarn repository needed key update.

Updated 2020-03-31: Upgrade from 3.1.1 to 3.1.2 is usual, no surprises. Version of rbenv is the same – 2.7.0.

Updated 2020-04-27: Upgrade from 3.1.2 to 3.1.3 – using Ruby version 2.6.6 and added remote media delete command.

Updated 2020-04-27: Upgrade from 3.1.2 to 3.1.3 – using Ruby version 2.6.6 and added remote media delete command.

Updated 2020-07-03: Upgrade from 3.1.3 to 3.1.4: needed to update Node.js 10.16.3 to newer version.

Updated 2020-07-20: Upgrade from 3.1.4 to 3.1.5: use git fetch && git checkout v3.1.5 instead of git pull.

Updated 2020-10-15: Upgrade from 3.1.5 to 3.2.0: need to run database update and afterwards clear cache.

Updated 2020-10-22: Upgrade from 3.2.0 to 3.2.1: use git fetch && git checkout v3.2.1 instead of git pull.

Updated 2020-12-21: Upgrade from 3.2.1 to 3.2.2: updated Ruby version to 2.7.2.

Updated 2021-01-02: Upgrade from 3.2.2 to 3.3.0: database upgrade should be continued after Mastodon prosesses restart, added details to fix database indexes warning.

Updated 2021-06-21: Upgrade from 3.3.0 to 3.4.1: check dependencies, perform database migration.

Updated 2021-11-16: Upgrade from 3.4.1 to 3.4.3: use git fetch && git checkout v3.4.3 to retrieve sources. Other than that usual upgrade, just after assets:precompile also need to update browserslist.

Updated 2021-12-17: Upgrade from 3.4.3 to 3.4.4: use git fetch && git checkout v3.4.4 to retrieve sources.

Updated 2022-02-03: Upgrade from 3.4.4 to 3.4.6: use git fetch && git checkout v3.4.6 to retrieve sources.

Updated 2022-11-15: Upgrade from v3.5.3 to v4.0.2: using git fetch && git checkout and updated Ruby version to 3.0.4.

Comments? Reply to this post on fediverse and I will get mention at @gytis@mastodon.lt.

Tags: #mastodon #server #ruby #ubuntu

My name is Gytis Repečka, I am Solution Architect focusing on Data Vault automation. Experienced data professional with over 10 years of experience designing and implementing Data Warehouse solutions based on Teradata, Informatica PowerCenter, Oracle Data Integrator, PostgreSQL, AWS. I enjoy using, promoting and contributing to open source software and love communicating about tech to both advanced and non-tech people. Visit Inretio for consulting services. Mention me @gytis@mastodon.lt on Mastodon.