The Raspberry Pi Mastodon Instance: Do You Have Your Backups Sorted?
A while ago, I set up my own Mastodon instance on a Raspberry Pi. Why? Well, I think it’s important to reclaim basic communication infrastructure on the internet. And besides, setting up your social media node in a big distributed network is a good opportunity to learn about back-end architecture. The good news is, it’s relatively easy and rewarding! Once I had set up the Pi in April last year, I hardly had to think about it.
But, of course, things do go wrong. And that is what I want to talk about here. What happens when your self-hosted instance crashes? How hard is it to get it back up and running? In this article, I will share my experience and give a few points of advice for those who are curious about running a Raspberry Pi Mastodon instance. But for that, I need to say a bit about my setup.
What do I use?
Because I want to run everything myself, I am not making use of any ‘cloud’ computing services and am responsible for all server components. I had a Raspberry Pi 4 Model B (2018) lying around, which I connected to a standard router and its own USB-C power supply. For the router connection, I used an ethernet cable and Wi-Fi, on and off, depending on where the Pi was. Just configure the router to forward the ports used by Mastodon (80 for HTTP, 443 for HTTPS, 22 for SSH) to the local address of your Raspberry Pi (type hostname -I in the terminal to see it). And then, use the DNS settings for your domain name to forward the same ports to the public-facing IP address of your router.
The Raspberry Pi makes for a great server. It’s impressively fast for such a small, cheap computer and has 8 GB of RAM. That’s more than enough for a small Mastodon instance. The tiny fellow came with a 32 GB SD card (Kingston Canvas Select Plus) as its main system drive. More about that SD card in a minute, but I can already say that although it’s fine, it’s certainly not fancy or fast. My operating system is just the default Raspbian Linux pre-installed on the SD card. To get Mastodon up and running, I followed the ‘Preparing your machine’ and ‘Installing from source’ documentation on joinmastodon.org (in that order). It was relatively straightforward.
Keeping memory in mind
OK, let’s dive deeper into memory for a bit. Because it does matter if you are thinking of hosting a Mastodon instance yourself, and this is where I deviated from defaults.
Even if you only host a handful of accounts (like I do), your instance ends up accumulating a good number of gigabytes of data. These are stored on the one hand in a PostgreSQL database (for text statuses that make up your timeline, as well as all system preferences and settings) and on the other in a folder that is ~/live/public/system by default (this is where profile pictures, shared media, etc. go). As you can imagine, especially the latter folder gets rather bulky as your social network grows — some people love posting videos, don't they?
To prevent the Pi from running out of memory, I added a 1 TB Samsung Portable SSD connected to the Pi’s USB port. (In the picture above, the Pi is resting on the drive.) This is where the bulky /system/ folder now resides, overriding the default setting. You can change the location where Mastodon stores its media in the ~/live/.env.production file and may also need to add an alias to the Nginx configuration for Mastodon. In my case, this was in the .env.production file, set:
PAPERCLIP_ROOT_PATH= /media/mastodon/system
PAPERCLIP_ROOT_URL= /storage/systemand for the Nginx config file, add the following line to both the HTTP and HTTPS configuration:
location /storage {alias /media/mastodon; }In addition, you need to make sure that the external drive is mounted correctly and that the user mastodon can write to it.
When I first added the drive, my server was running happily, but any media upload produced a 500 error (if you get comparable errors, begin by running journalctl -u mastodon-web -f in the terminal, which shows you a live view of HTTP requests to and from your instance). In my case, the problem was that the mastodon user (which runs the software) wasn’t allowed to write to the mounted SSD drive! No wonder uploads failed. I fixed this by tailoring the drive’s entry in the /etc/fstab file (which defines Linux’s mounting points), giving all users read and write access, like so:
/dev/sda1 /media vfat rw,user 0 0These settings resulted from some trial and error; at least they work for my setup. If you think I’ve done something horribly wrong or unsafe, do let me know! But this memory solution does its job after more than a year of tooting and scrolling.
The /system/ folder takes up 52.7 GB (~5% of disk space), and it’s no longer growing rapidly, as I’ve configured the instance to clear cached media files after 31 days. Plenty of room there. On the other hand, my Postgres database still lives on the default location /var/lib/postgresql/11/main, which is on the SD card. (You’ll notice that my Postgres is version 11, which works fine.) By now, the Postgres database is 2.55 GB in size. Which, at 8% of a 32 GB system drive, is not a disaster, but it’s too bulky for my liking (at times, I only had 17% drive space left). I think that, one day, I want to move the Postgres database to the external SSD as well. But that’s for another time.
Adding an extra drive to your Raspberry Pi Mastodon instance can be a bit fiddly, and the exact settings will depend on the file system of your external drive. But it is almost always a good idea to do so (partly for the reason I will describe below).
But do you make backups?
As I said in the beginning, I want to talk about when things go wrong. When you host your own instance, you need to think about what you’d do if your server goes down. Over time, I’ve built an extensive social media network and use Mastodon almost daily to interact with friends. And I cannot rely on anyone else to guarantee it’ll all still be there tomorrow. In other words, if you don’t have a backup plan, you may lose the accounts hosted on your instance overnight: the posts, the pictures, and, crucially, the network the people on your instance built up.
This is why you should do a risk assessment before setting up a Raspberry Pi Mastodon instance. Ask yourself, what can go wrong given my configuration? How bad would it be? And what would I need to have rescued to recover functionality when things go badly wrong? To be sure, some things require patience (like a hiccup with your internet provider). But other things require you to restore your server from whatever may be left of it.
Do a risk assessment before you even begin setting up a Raspberry Pi Mastodon instance
Of all those contingencies, the main risk I wanted to prepare for was the failure of my SD card. Running Linux from an SD card is not a very good idea, given that SD cards are flaky. But hey, that’s what a Raspberry Pi does. So what would I need if the SD memory failed beyond repair, and I was forced to reinstall Linux and the Mastodon software from scratch?
To mitigate the risk, I came up with two things.¹ The first I already talked about, namely moving the /system/ folder to a much more reliable SSD drive. In case of an SD card failure, the media storage would be safe. Good. But what about the Postgres database? Those data are probably even more important, as they define the social network the instance has built up.
Eventually, I decided to automate a daily backup of the database. I wrote a shell script that would run each night (I added it to /etc/crontab). The script dumps the contents of the Postgres database (which, by default, is mastodon_production) into a backup file stored on the external SSD.
For this, I scripted the following command: sudo -u postgres pg_dump mastodon_production > /media/backup/mastodon_production.dump. For extra safety, the script would then gzip the file (with encryption) and send it via FTP to a different remote server I can access. On that server, I keep the three most recent .dump files. The same script deletes older .dump files automatically. (I might push the script to GitHub someday, but it’s nothing very special.)
Finally, last week, the risk became reality. The Pi’s SD card did fail on me. I was entirely locked out of my machine. Was my backup strategy sufficient?
Restoring from backups
The precise answer is: no. My backup strategy was not sufficient. But it got close enough, fortunately. Because I am back online, and that proves that it’s not too difficult to recover your Mastodon instance, even if you have to reinstall the Raspbian operating system and the Mastodon software from scratch. As long as you devise a backup strategy.
Was my backup strategy sufficient? No.
So, what happened? As far as I can tell, the SD card’s boot sector got corrupted when I powered down the Pi incorrectly. This is all too easy to do. Plenty of people have lost their data after a power cut. Unthinkingly, I clicked ‘shut down’ on the desktop when I wanted to disconnect the Pi to move it to a different location. Only afterward did I read a note I left myself last year on the best ways to shut down safely — I should only use sudo shutdown -h now in the terminal (the -h option tells the system to halt whatever it’s doing). Oops. In any case, I found I could no longer boot up my system and, accordingly, no longer run my Mastodon server.
What to do? Somewhat at a loss, I used a VirtualBox machine running Ubuntu on my Mac to mount the card as an external drive and tried to repair it with the standard disk tool and with the disk management programme fsck in the terminal. No luck. However, on Ubuntu, I was able to mount and access the partition that contained my /home/ folder, which meant I could still rescue some files I turned out to need for recovery but hadn’t backed up. Phew.
Which files did I need? I found the best answer to this question in Mastodon’s documentation, in a description of ‘Migrating to a new machine.’ Because that is what you have to do when you have to re-install Raspbian and the Mastodon software from scratch: you’re setting up a fresh machine. The following is a list of what, given my configuration, I needed to recover server functionality in a straightforward way²:
- dump of the Postgres database (by default, this is
mastodon_production) - copy of the
/system/folder (by default, this is~/live/public/system) - copy of the
~/live/.env.productionfile - copy of Nginx config (by default
/etc/nginx/sites-available/mastodon)
After the SD card crash, my external SSD contained (1) and (2), which are essential. And for (3) and (4), I was lucky I could still retrieve them from the card when I mounted it on the other Linux machine.
Those additional files contained the specific configuration settings of my instance, such as the provision for an external media folder, the details of the SMPT server used for notifications, and the ‘secrets’ for active browser sessions, two-factor authentication, and push notifications. Okay, it would not have been impossible to recover without them, but having all this information exactly as I needed certainly made life easier. It saved me heaps of time.
Now, that I would need those files in an emergency should have been no surprise to me, given that those were the files I had earlier tweaked to make my instance suit my needs! Nonetheless, I had forgotten to include them in my backup routine.
Don’t just make backups. Also practice restoring your data from that backup, and do this before you start relying on the backup
That’s one thing that was wrong with my initial backup plan. But at least equally important: I had never tested my backup before I needed it. This meant that when the real problem arose, I didn’t know how to restore things, even though I had the (most important) backup files. This made the situation needlessly stressful.
For me, this is a lesson for the future. If you’ve only satisfied that you have generated some backup files, you’ve left out the other half of the process for which you’re making those files to begin: the restoration process. Don’t just make backups. Also, practice restoring your data from that backup before you start relying on the backup. If I had done that, I would have noticed that my backup was insufficient. And if you know what to do when the actual problem arises, you can restore things much more efficiently and with less stress and sweat.
Conclusion
In this case, the story had a happy ending. I’m writing this in part because I know I’ll probably encounter this or a similar problem again, and it’s useful to have a manual to remind me of the steps I need to take. But of course, I hope it might help you too.
In particular, I hope it makes the prospect of running your own Mastodon instance on a Raspberry Pi less daunting. Because really, you too can experiment with taking back control of online communication. In the end, the risk posed by losing access to your machine is much less scary than it may seem at first. As long as you do a risk assessment before you start, implement a reliable backup routine, and take some time to practice restoring your machine from backup before there’s an urgent need to do so, you’ll be fine. If you do these things, you can trust your little Pi to serve you well.
¹ An alternative strategy I have never tried: Make regular clones of your SD card. If you have a cloned version, you can swap cards when the original gets corrupted, and you should be back on air. I didn’t opt for this because I thought regular cloning would be cumbersome compared to the Postgres dump. But perhaps I’m overlooking a good alternative here?
² The list in the official documentation is more extensive, but because I relied on default settings for some other things, I did not need to worry about various other files. But do read the ‘Migrating to a new machine’ article before you start using your server, and read the ‘Backing up your server’ article.
