Update Mastodon server instance

You might have already noticed I use Mastodon for my main fediverse handle @gytis@fedi.lt. Current version is 4.2.8.

Attention! This guide is stale and remains here for archive purposes. Updated guide how to update Mastodon version is maintained at Inretio Community. Changed my mind: afer using XenForo (nice proprietary community software) for a while I have figured I like writing in Markdown much more, so this guide will be maintained on WriteFreely instance here.

While typically updating Mastodon is quite straightforward, sometimes you might need to iterate Ruby version on the server as well.

That was visible from error thrown:

rbenv: version `2.6.1' is not installed (set by live/.ruby-version)

So the full upgrade sequence is following:

Check dependencies

Certain versions might require specific dependencies: v.3.4.0 mentions following:

Postgres requirement raised to at least 9.5 Node.js requirement raised to at least 12

Check them:

$ node --version
v16.20.2

$ psql -V
psql (PostgreSQL) 16.1 (Ubuntu 16.1-1.pgdg22.04+1)

Retrieve Mastodon source updates

Switch or login to user which is running Mastodon:

sudo -u mastodon bash

And execute following:

cd ~/live
git fetch --tags
git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)

You might get error here:

error: Your local changes to the following files would be overwritten by checkout:
	.ruby-version
Please commit your changes or stash them before you switch branches.
Aborting

In that case reset git:

git reset --hard

And try checkout again.

When updating to v4.0.2 or v4.1.3 (also when noted in Release Notes) sources need to be retrieved using following (and not git pull):

git fetch && git checkout v4.2.8

Note that above git command uses checkout with dynamic value – the most recent branch number.

Update Ruby

cd ~/.rbenv/plugins/ruby-build
git pull
cd ~/.rbenv

Check what Ruby versions are available with rbenv versions and install:

RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 3.2.3

This was also documented in issue 638.

Which version of Ruby is used can be seen in file live/.ruby-version. Mastodon version 3.1.3 recommends version 2.6.6. With Mastodon upgrade to 3.2.2 we are also upgrading Ruby to 2.7.2. This version is advised with Mastodon version 3.3.0 upgrade. Version to use with Mastodon v4.0.0 is 3.0.4. Version to use with Mastodon v4.2.0 is 3.2.2. Mastodon version 4.2.4 and 4.2.5 recommends Ruby version 3.2.3.

When installing Ruby version 3.2.2 there was an error:

rbenv: cannot rehash: /home/mastodon/.rbenv/shims/.rbenv-shim exists

To solve run rm ~/.rbenv/shims/.rbenv-shim and then rbenv rehash. Then change version to 3.2.2 in ~/.rbenv/version.

Update Mastodon

cd ~/live

Update Ruby dependencies

bundle install

If your server has problems connecting to rubygems.org, follow this advice and add below lines to /etc/gai.conf:

# Low precedence for api.rubygems.org IPv6 addresses.
precedence  2a04:4e42::0/32  5

After this bundle install works smooth.

Possible error

When updating mastodon.lt to Mastodon version 2.8.2 I got following error:

Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.

Fix it by running:

cd ~/.rbenv
gem install bundler

And go back to previous folder:

cd ~/live

Update node.js dependencies

yarn install --frozen-lockfile

Note! When updating from v4.1.6 to v4.1.7 changed parameter from --pure-lockfile to --frozen-lockfile.

Yarn error upgrading to version 3.1.1

When upgrading to version 3.1.1 got this warning about Yarn:

warning Your current version of Yarn is out of date. The latest version is "1.22.0", while you're on "1.21.1".
info To upgrade, run the following command:
$ sudo apt-get update && sudo apt-get install yarn

Interestingly, I also needed to fix Yarn repository key because of error:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://dl.yarnpkg.com/debian stable InRelease: The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>
W: Failed to fetch https://dl.yarnpkg.com/debian/dists/stable/InRelease  The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>
W: Some index files failed to download. They have been ignored, or old ones used instead.

Fix it by running:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -

The minimum supported Node.js version has been bumped from 8 to 10. To upgrade Node.js, run:

curl -sL https://deb.nodesource.com/setup_10.x | bash -
apt install nodejs

move-file error when upgrading to 3.1.4

Trying to run yarn install triggered this error:

$ yarn install --pure-lockfile
yarn install v1.22.4
[1/6] Validating package.json...
[2/6] Resolving packages...
[3/6] Fetching packages...
info fsevents@2.1.3: The platform "linux" is incompatible with this module.
info "fsevents@2.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
error move-file@2.0.0: The engine "node" is incompatible with this module. Expected version ">=10.17". Got "10.16.3"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
mastodon@social:~/live$ node -v
v10.16.3
mastodon@social:~/live$ npm --version
6.9.0

To upgrade Node version following was done:

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs

And then npm used:

# npm cache clean -f
npm WARN using --force I sure hope you know what you are doing.
root@social:~# npm install -g n
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
+ n@6.5.1
added 1 package from 4 contributors in 1.02s


   ╭────────────────────────────────────────────────────────────────╮
   │                                                                │
   │       New minor version of npm available! 6.9.0 → 6.14.5       │
   │   Changelog: https://github.com/npm/cli/releases/tag/v6.14.5   │
   │               Run npm install -g npm to update!                │
   │                                                                │
   ╰────────────────────────────────────────────────────────────────╯


# npm install -g npm
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
/usr/local/bin/npx -> /usr/local/lib/node_modules/npm/bin/npx-cli.js
+ npm@6.14.5
added 32 packages from 17 contributors, removed 23 packages and updated 78 packages in 5.903s
root@social:~# npm install -g n
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
+ n@6.5.1
updated 1 package in 0.121s

# node -v
v10.16.3

# n latest

  installing : node-v14.5.0
       mkdir : /usr/local/n/versions/node/14.5.0
       fetch : https://nodejs.org/dist/v14.5.0/node-v14.5.0-linux-x64.tar.xz
   installed : v14.5.0 (with npm 6.14.5)

root@social:~# n lts

  installing : node-v12.18.2
       mkdir : /usr/local/n/versions/node/12.18.2
       fetch : https://nodejs.org/dist/v12.18.2/node-v12.18.2-linux-x64.tar.xz
   installed : v12.18.2 (with npm 6.14.5)

# node --version
v12.18.2

After this yarn install --pure-lockfile succeeded.

Back to Mastodon user (sudo -u mastodon bash) and install dependencies:

bundle install
yarn install

Database schema updates

Typically use:

RAILS_ENV=production bundle exec rails db:migrate

Done with migration to v.3.4.1.

Upgrading to version 3.0.x or 4.0.x:

SKIP_POST_DEPLOYMENT_MIGRATIONS=true RAILS_ENV=production bundle exec rails db:migrate

Pre-compiling updated assets

RAILS_ENV=production bundle exec rails assets:precompile

When upgrading to v.3.4.3 it was required to update browserslist:

Compiled all packs in /home/mastodon/live/public/packs
Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db

Why you should do it regularly:
https://github.com/browserslist/browserslist#browsers-data-updating

It fired up when upgrading to v4.1.2, so let's upgrade:

Just run suggested command and you'll be good to go:

npx update-browserslist-db@latest

Clear cache

RAILS_ENV=production bin/tootctl cache clear

Restart services

Mastodon runs in memory so you will need to restart it for any of the previous updates to take effect.

Switch to root (exit) or run in sudo:

systemctl restart mastodon-*.service

Or restart services separately:

sudo systemctl restart mastodon-web
sudo systemctl restart mastodon-sidekiq
sudo systemctl restart mastodon-streaming

Finish database migrations

Now that the new code is running, we can finish the database migrations. This will run the post-deployment ones:

RAILS_ENV=production bundle exec rails db:migrate

In case warning appears:

$ RAILS_ENV=production bundle exec rails db:migrate
Your database collation is susceptible to index corruption.
  (This warning does not indicate that index corruption has occured and can be ignored)
  (To learn more, visit: https://docs.joinmastodon.org/admin/troubleshooting/index-corruption/)

Stop Mastodon services, and, as documentation suggests, run following to fix:

RAILS_ENV=production bin/tootctl maintenance fix-duplicates

Restart Mastodon services once again.

Clear remote media

As remote media is cached locally, it might consume large amount of space over time.

Cache cleanup command was changed to work with following command:

RAILS_ENV=production bin/tootctl media remove

On my instance it has freed quite an amount:

Removed 53385 media attachments (approx. 25.2 GB)

Database migrations after upgrade to 3.0

Now that the new code is running, we can finish the database migrations. This will run the post-deployment ones:

RAILS_ENV=production bundle exec rails db:migrate

There we go – Mastodon 2.8.0 with polls functionality running fine 😊 Try out my poll about Go code retrieval 👍

Changelog

Comments? Reply to this post on fediverse and I will get mention at @gytis@fedi.lt

Tags: #mastodon #server #ruby #ubuntu #mastoadmin #linux #fediverse


My name is Gytis Repečka, I am Solution Architect and data professional. I enjoy using, promoting and contributing to open source software and love communicating about tech to both advanced and non-tech people. Visit Inretio for consulting services. Comment by mentioning me @gytisrepecka@social.gyt.is on Fediverse.