We divided the time of our rspec test suite by 9

When people ask me why I chose to build Pootsy with Ruby On Rails, I always say it’s because the community is so serious about testing. They built rspec, cucumber, … those are tools that became the standards of the industry.

We started writing tests from day one. Then, after one year and lots of work, our code coverage was over 70% of the back-end with nearly 600 tests. Tests were running on every commit and our Pull Requests could only be accepted once every one of them passed. It sounds good, but soon it took Codeship (our CI) nearly 18 minutes to run the complete test suite and it kept increasing fast.

A long duration for your test suite is bad for multiple reasons :

• as developer, especially when working on a “core” feature, you need to quickly know if you didn’t break anything, so you have to run every test. And wait.
• as a company, you can’t ship as fast as you used to because every new code needs to be fully tested. So you wait.
• everytime someone has to wait for something to complete, they’ll start working on something else and lose focus on what’s important : shipping features in production. If a test had failed, they would have to switch context, see what went wrong, fix it, then wait again until every test goes green.

Test is code

Just like code, it needs to be maintained, refactored and optimized. Just like code, it can grow into an out-of-control mess if you don’t treat it with care.

As features were shipped into production, our test suite accumulated technical debt because we didn’t maintain it enough (“it’s all right, tests still pass”) and, quite frankly, because none of us ever wrote a test suite of that scale.

So we started optimizing our tests.

Optimise the tests

Typically, a test would contain :

1. a Seed file, creating a predictable state of the database, allowing us to test a specific scenario
2. a Shared Context, allowing us to use (and load) some data in multiple test files
3. test files, where the magic happens. You describe a use case verify that it works.

After some investigation, it turned out that the slowest part of our tests were … the seeds. For every unit test, the seeds would be generated. The higher the number of tests, the slower the tests.

Here’s what we came up with :

load the test once

In our SharedContext files, instead of loading the seeds before all tests, we load them once ( before(:do) vs before(:all)). The big difference between before(:do) and before(:all) is that the first rollback after the test and we did not want a rollback for our seeds.

But that wasn’t enough. In our TestsHelper we added the following method:

put the seeds file in some “cache”

This method makes sure that we don’t load one seed file more than once. Repeat this change for all the SharedContext and you’re done.

We went from 18 minutes to 2 minutes for a full test suite run. That’s really amazing!

Now, feedback is immediate and we can focus on shipping great new features instead of waiting for tests to finish running.

Tests are great to test your code and be sure that the use case that is tested still works as expected as the codebase grows. But tests still are code, that you need to take care of if you don’t want them to get in the way of shipping new features in production. Every now and then, don’t forget to refactor and optimise them.

Kudos to Renaud for finding this out 👍