Upgrading major Rails versions by just using your 92% test coverage

Software testing is like brushing your teeth. Generally not that fun to do, but a couple of years later when everything falls apart, you wish you had done a better job early on.

For us there were three good reasons to test. And it enabled us later on to perform major upgrades really quick and faultless. Read more to learn how!

As a developer, you are probably familiar with the feeling of writing tests. Often it comes as an afterthought. It is not that much fun to do and usually the advantages are not directly apparent.

It’s a crucial job though! At inventid we strive to keep our test coverage above 90%. No matter how dull the job sometimes can be. But there is a good reason why we stick with it: it made our lives much, much easier.

Actual reasons to test

As you can read in book X and blog Y, testing has a lot of advantages. However in the real production world, we need business cases for tests.

We started out ourselves by just writing tests since it felt good (you know, because you do what you were learned in class). Yet after a while, we figured out some cases where testing early was a great idea.

The first one was that as a developer you are the first actual user of your API. When testing you early spot areas which are difficult to test: these are often candidates where you missed some abstraction which would make your life easier. Even more, you make the life of the next person to touch your code MUCH easier.

Secondly it saves a lot of maintenance in the long run. We have a 92% test coverage and monitor it (along with code quality) rigorously (Thank you codeclimate!). This allows any developer to touch any piece of code and validate it did not change existing behaviour by running the test suite.

To be honest, by having such a high code coverage, we have almost eliminated click-testing for everything in our backends. Just when touching frontend code you will fire up your browser, otherwise rspec is the go-to guy. And as automated testing is much faster and less error-prone than manual testing, it saves time (and bugs) as well.

A third point is the most known one: ensure your code behaves fault-free. As we handle vast amount of money flowing through our systems, including automated invoicing and payouts (and all millions of corner cases related to dealing with cash), not making errors when paying out customers is a big deal. When you have less than 100% confidence in your billing systems, you’d better turn them off asap. Tested helped us getting this confidence.

Upgrading from Rails 4.x to 5.x by just using rspec

When Rails 5.x emerged, I dreaded the upgrade. First of all we used a gem called squeel which handles database abstractions over Active Record. This gem got badly maintained at some point. So for Rails 5.x we had to migrate away from it anyway.

The first step was simple: ditch the gem and find a new one. In the end we settled on baby_squeel, which was actively maintained and did not monkeypatch Active Record.

As the gem was replaced, almost any test would fail. However by gradually fixing syntax and running rspec in the background, more and more green slowly emerged in my dark terminal.

After a while, all tests started passing again. As all public endpoint were completely tested, I could start to commit the different parts and prepare a pull request. Only after having the PR in place, I clicked through the application to verify our test suite did not miss anything. It didn’t :)

The first step of upgrading to Rails 5.x came a while later. After switching the Rails gems with newer versions, it was time to run the test suite again. Since a complete test of all backends takes just 3 minutes, this is a process which you can do all day long.

As part of our procedure, I’d first eliminate all deprecations. These can otherwise be kept in, and stuff keeps getting worse and worse. Deprecations are generally easily fixed but can cause problems between different components (no source, just my personal experience). Next step is removing all errors which prevent tests from running (so not fixing anything broken, only failing tests). Once this step is done, you will have a clear overview on how everything is performing. If the code quality was good to start with, you should just see a few failing tests.

And here comes the main point of testing: the tests indicate the desired behaviour! So it’s easy to figure out whether a test broke in the upgrade (some did), or that some code was no longer performing to spec (the others did). By using rspec (where your test name is indicative of what something should be doing), you can start fixing code/tests to continue to conform to that behaviour.

The take aways

As we have experienced first hand, testing is still not much fun. Still, the payoffs are huge:

  • Be the first user of your API: you spot usability problems early on,
  • Ensure your code works: critical if you care about your business,
  • Easy maintenance: just fire up your test suite and you ensure any behaviour did not change.

In the end, our Rails upgrade took 1.5 days to complete code wise. Afterwards we just fired it up to a staging environment, where we did click-test most important parts. In the end, the upgrade went through in 3 days, including code reviews. Not bad!

Software Engineer. Lead software engineer @ Magnet.me, former CTO @ inventid.nl. General nerd. github.com/rogierslag