Server-side Swift: Making Canopy (4/6)

Last time I talked about Swift on Linux from the Cloud provider side, this week is about the Linux side, and considering I expect my audience to have little to no server-side experience, we will be talking about Linux sub-systems.

What is Canopy?

Canopy has apps for macOS and iOS, and it’s available here.

Domain

I used AWS, and thus used Amazon’s Route 53 for DNS, this makes it easy to assign your Elastic IP (which we talked about in part 3). You don’t have to buy the domain through Amazon, but they make it easy and all the registrars cost about the same. It is easier to keep all this stuff together, so I’d recommend it.

You need a domain before you can enable HTTPS, thus at this point you should either have a throw away domain or have figured out what your server is going to be called.

HTTPS (i.e. SSL)

It would be irresponsible not to run your server without encrypting its traffic nowadays. But what does this involve? Fortunately—nowadays at least—not all that much.

As I said in week one, Swift can trivially consume C-libraries, which means that Swift has easy access to mature cryptographic libraries like OpenSSL. Perfect thus has HTTPS support out the box. Sadly you cannot just enable it, since you first need a private key file, and only certain providers on the Internet provide these. Most of them are not free, but since HTTPS has become a must have the Internet Security Research Group stepped up and provided a free service: Let’s Encrypt. It’s popular, and you’ll be hard pressed to find many guides for other services nowadays.

I will not outline all the steps since their guides are thorough and better than anything I could summarize, but it boils down to:

1. sudo apt-get install certbot
2. sudo certbot certonly --standalone -d api.example.com

No other web-server should be running when you do step 2. This will create your key and certificate pair and place them in /etc/letsencrypt

To use these with Perfect is simple:

Now if you swift run you’ll find your server is accessible at the https version of your domain.

We’re not done yet

Let’s Encrypt requires you to renew your certificate often. They email you a few days before hand so you can get to it. Renewing is simple enough:

certbot renew --standalone

However you will need to stop your server binary first since certbotmanages the renewal by running its own server on port 443. This can be mitigated by having your server host the files certbot generates during renewal, but I haven’t managed to get this working yet. I will update when I do.

Because humans are forgetful we want to automate this renewal. The best tool for this on Linux is the ever reliable cron. We can configure the cron daemon thus:

sudo crontab -e

Which opens the system text editor, add this at the end:

47 3 * * * cd /home/ubuntu && certbot renew --standalone

Which runs the renewal script every day at 3:47 AM UTC. The archaic and confusing syntax for cron is famous, basically, google it, that’s what I do when I need to edit the crontab. Every. Single. Time. Basically the numbers indicate the minute, hour, day, etc. to run the script, asterisks mean every time period of that column. Then there is the script itself.

certbot is smart and the script does nothing if the renewal is not yet necessary.

If you worry about how long is left this handy one-liner will tell you:

echo | openssl s_client -connect api.example.com:443 2>/dev/null | openssl x509 -noout -dates

Which will output something like:

notBefore=Oct 16 19:21:38 2018 GMT
notAfter=Jan 14 19:21:38 2019 GMT

Which here means you cannot renew before October 16th, and your HTTPS will stop working if not renewed by January 14th.

systemd

We’re after five-nines (99.999%) uptime. We’re unlikely to get it, but we should try. One step along this path is getting Linux to treat your server binary as a proper system-daemon. This gives us:

1. Your server binary will be started on startup, (so: after reboots, Amazon reboot your server for you occasionally!)
2. Your server binary will be restarted if it crashes

The second shouldn’t happen, since we’re using Swift and we never bang (force unwrapping those optionals is the surest way to crash, and you really don’t need to with pure Swift, if let those optionals, if you have any). Still it will, for example mine has been kernel terminated due to a file-descriptor leak. Not everything can be mitigated.

Like most things Linux, systemd is easy, but it’s interface is completely foreign to everything else you’ve ever used. First we edit a file:

sudo rmate /etc/systemd/system/foo.service

I really recommend rmate since it makes it trivial to edit remote files with TextMate on your Mac.

Here’s a template for your .service file (adapt the paths if you are building from source).

[Unit]
Description=Foo Backend
After=network.target
[Service]
Type=simple
WorkingDirectory=/home/ubuntu
ExecStart=/home/ubuntu/foo
Restart=always
StandardOutput=file:/var/log/foo.log
[Install]
WantedBy=multi-user.target

You can now start your server-binary with:

systemctl start foo

Other verbs you will want to know are stop and restart.

You can now update your server by rsyncing the sources (which then means remote running swift build) or cross compiled binary and asystemctl restart. For example here was my script while I was building server-side:

rsync --archive --human-readable --compress --verbose --delete \
      --exclude .build --exclude Carthage \
      . canopy:src
ssh ubuntu@canopy.codebasesaga.com \
     "swift build -c release && sudo systemctl restart canopy"

To have Ubuntu start your service on restarts you need to enable it:

systemctl enable foo

Logs

Using systemd also means any output from your service is now funneled to the system-log, the system logger ensures log output is timestamped and rotated. Rotation is key if you don’t want your instance to run out of disk space due to logging.

Before using systemd you would probably use screen to keep your service running and view logs. This is a bad use for screen, but is widely considered a fine choice for development.

Now you can view logs with:

journctl -fu foo

This “tails” the log so you can see new messages as they come in. You can easily do this remotely with a one-liner:

ssh foo "journctl -fu foo"

Graceful Shutdown

A graceful shutdown is important, otherwise updating your server risks data loss, or at the very least, any existing connections will be abruptly terminated.

Sadly neither Perfect, nor Vapor, nor Kitura (in fact none of the Swift solutions seemingly) support this out the box. Thus in order to gracefully shutdown we have to handle it ourselves.

systemctl sends SIGTERM (this is a UNIX signal, when you press ⌃C when a process is running in Terminal.app it sends SIGINT, SIGTERM and SIGKILL are other common signals) when stopping your service. Thus we can intercept this signal and run some custom code:

The above intercepts the signal and stops our web-server, which will exit the program. However it does not allow any existing connections to complete.

There is no trivial way to achieve this, I had to write custom code, and it made using Promises absolutely essential. The gist is:

1. Add new connection promises to a list (be thread safe!)
2. When SIGTERM, stop accepting new connections (stop() is premature here, it will close all existing connections), use a Bool (be thread safe!)
3. Wait on all existing connections (when() with PromiseKit)
4. stop()

systemctl gives you about 15 seconds (which should be plenty—if your routes are taking longer than a few milliseconds to complete, you need to do some debugging!) after which it kills your process.

Backups

You must prepare for catastrophic failure. When ssh’d into your server you’re really only a few keystrokes away from losing your whole instance. Fortunately whole instance back-ups are trivial with AWS. We can use instance-snapshotting.

I won’t describe the steps since Amazon do a fine job but once enabled you can have hourly, daily, whatever interval you like snapshots created and saved to your S3, they are incremental which means they take up only as much new space as has changed in your instance, which is likely just the changes in your database.

At any time you can spin up a new instance with a snapshot or you can restart an existing instance using a snapshot instead of its current storage.

As with any back up strategy test your recovery flow. I verified it during development after a bad migration, it’s quick and effective.

Really S3 isn’t proper backup storage, so you should transfer the snapshots to Amazon’s Glacier at semi regular intervals. In practice though S3 is probably fine for your back up needs. Much of the Internet depends on nowadays after all.

Be Careful with Updates

In part three we talked about enabling automatic security updates. What else should you update? IMO: just Swift.

It’s not worth upgrading anything on a server unless you need a new feature. Security updates are automatically installed, so you’re getting those. Any other changes are too risky. A single bug on the server can affect every single user you have, this is not apps anymore where thousands of instances of your app are running and bugs effect people sporadically, a server issue can take out the server and then boom, none of those thousands of instances are getting any data.

For the love of ghosts, don’t upgrade the kernel.

And, for cheeses, if you must upgrade something (eg. Swift), make a snapshot backup first!

However it’s probably (possibly) a good idea to upgrade Swift as soon as possible. Swift releases have loads of bug fixes and performance improvements that translate directly into your service’s reliability and your AWS costs.

Database Choice

The top dogs of database choices (MySQL, Postgres, etc.) can run the biggest APIs in the world. But they are totally overkill for most, and they are probably overkill for you.

I chose SQLite and I did this because SQLite is super, incredible amaze-balls.

In-Process

The top dogs all run in separate processes, which also means they can run on separate machines. This makes scaling simpler but it makes configuring them much, much more complex. You’re committing yourself to a continuous nag of database management that will never disappear.

There is no faffing around with SQLite; the database engine is compiled directly into your binary.

This of course means scaling your system to multiple computers would require migrating your database. But once you have scaling problems you’ve won, you’ve succeeded. Worry about winning once you’ve won. Scaling will likely happen much later. Anyway you can mitigate this problem by using an abstraction layer that can use multiple underlying database types, or by writing your queries via SQL strings.

Other Reasons

SQLite is small, fast and efficient, provided it fits your needs there are barely any compromises.

It is everywhere, a bunch of Apple’s stuff uses it on Mac and iOS. Android uses it. Playstation uses it. Your smart devices probably use it. Everyone uses it. They use it because it is small, portable and really, really great.

SQLite is designed so incredibly thoughfully, reading about it will make you feel ashamed at how you approach designing your own systems. I recommend their How to Corrupt an SQLite Database article; while the title may distrub you the article is actually extremely comforting: the lengths the team have gone to defend your data warm the heart.

SQLite is considered to be one of the most throughly tested systems in the world. Check it out, SQLite verifies correct behavior for storage device failure, for incorrect error reporting from the kernel, that your database is safe if out-of-memory conditions occur inside transactions. They also extensively verify that your database will be ok if your code crashes; this matters since the database engine is running in the same process as your code.

Your database is your service, thank ghosts for the SQLite developers, they are incredible people.

Considerations

Don’t take my word for it, SQLite are humble and honest about when you should use their database.

SQLite has three modes that its library may be compiled, so you should check in your code itself before the server goes “up” that you have the multi-threaded version.

precondition(sqlite3_threadsafe() == 2)

Checks like these will save your butt in the unlikely event that somehow you accidentally install the wrong sqlite version in the future.

The multithreaded mode means you can read and write to the database in multiple threads provided you use a separate connection instance in each thread. Essentially this means creating a database object for every route in your server.

Using SQLite

Perfect, Vapor etc. all provide facade/abstraction layers of varying greatness. You also more “Swifty” options like GRDB. I elected to write SQL queries as strings and use Perfet’s simple Swift wrapper to process the queries into my own objects. Like most of my decisions this was because of the relatively immature system that is Swift on Server and my desire to limit the amount of overhead that may be buggy, unfinished or immature. Also because it had been a while since I’d directly used any databases, and I wanted to get down in the trenches and fully understand how my systems worked. Knowing how technologies work pays dividends when it comes to getting further ahead later.

Here’s an example of the code (db is an instance member that is the database connection):

With hindsight I wish I had used GRDB since it fits Swift’s programmer-model much better.

I could write entire articles on how to use databases, I shall not, there are many, better articles out there. If you don’t want to read them, use the facade libraries that Perfect, Vapor, etc. provide, they handle table creation and type conversions for you.

Part 5

Part 5 is here.