How we made our tests faster
First some background on our process and what led us to this whole idea that we need to make our tests run faster. We develop our software in an iterative, agile fashion that streamlines delivery to about twice a week. Release is not the exception, but the rule. We deliver small chunks of features and deliver them fairly regularly. This has the advantage of being able to observe how the change is used in production and quickly adapt the next phase to match the observation. No need to wait 6 months for a simple feature request or first iteration of a larger feature. This way we can also avoid developing features that won’t be needed in the 6 months when we would have finally released them.
All of this has implications for our development process:
- We merge to master on per-commit basis
- Every commit that is merged must be releasable (!)
- Every change must be implemented in a backwards compatible manner
- We must have good test coverage of our code
- Each commit must pass tests before it’s merged
The last point in turn implies that it is essential for the tests to run as fast as possible. Otherwise merging commits takes too long, leads to potential conflicts and/or delays, merging queues start forming and in case of failed tests, developers have to attend to commits while already working on some other (possibly unrelated) thing.
Of course not everything is covered by unit tests (we run those before merging every commit) or integration tests (run on schedule several times a day) and sometimes you just want to make sure you didn’t break a feature, so you want to run the ultimate, system level tests that we call feature tests as we use the cucumber notation. Those are very comprehensive, but also take a while to finish and so similar unhelpful situations may arise.
These are some of the reasons why we decided to improve how fast our tests run.
So how did we do that? These days, it goes without saying that it all started with a Zoom meeting and a Google spreadsheet 😀. At first we came up with what we thought would be an ideal outcome of our little hackathon. Multiple metrics were discussed, but the most important ones were these:
- merge commit in under 4 minutes (took up to 9 minutes before)
- run feature tests in about 15 minutes (took more than 40 minutes before)
After defining the goals, it was time to get into specifics on how to achieve them. A couple dozen ideas were produced— often very different approaches to solving different problems. Everything and anything was on the table. What sticks and what doesn’t will come out of the process automatically — and if we miss here and there, at least we will be all the wiser for it. Which means we cannot lose 💡! Don’t make the mistake of killing your hackathon with discussions about merits of every single idea — before it even starts.
Anyway, to point out some of the ideas:
- reuse Selenium browser instances in feature tests
- get rid of dead code (yes, we know)
- simplify database migration scripts
- parallelize test runs
- running tests against multiple databases at once
- use simpler, faster browsers than Selenium (Mechanize etc.)
- faster hardware 😂
- allow automatic code formatting within code review (shameless plug of the author)
Impromptu groups of like-minded people formed, working together on topics they considered important. Regular meetings to get up to speed were held. And then the results started rolling in. And here they are:
Commit merging times were reduced exactly to what we set out to achieve — around 4 minutes. This did not come without its challenges, as making things faster led to some issues (e.g. NPM packages installation — we had to “manually” lock this process to avoid conflicts).
Feature tests work is even more exciting, as from the initial runs it looks like they now run 4× faster — around 10 minutes down from 30–40!
One of the issues with parallelization that had to be resolved was in the tests themselves — you obviously cannot change the data and expect them to be in a specific state at the same time. The solution is to simply generate custom parts of the dataset for each test as it’s run. Another thing is that the Behave runner that we use to run our tests does not support running tests in parallel , so a quick hack/workaround using Make that does support paralellization was used, while a better solution is being considered.
One thing is for sure — we are now more aware of performance considerations and we also have more ideas to explore in the future. And we now know yet a bit more about our codebase than we did before.
Epilogue
Of course none of this would be possible, if we - the developers - were not given an opportunity to embark on this quest. An enlightened management is a bit of a necessity here, but I believe even in less-than-ideal conditions, given the right arguments, minds can be changed. Yes, the idea of a three-day test improvement marathon may sound a bit too fantastical at first. There are always things that need to be finished first. However that mindset, if applied long term, leads to large technical debt, long running tests, increased bug occurrence, lower productivity and sad devs 😟
I encourage you to break free and find time for hackathons like this. Smaller ones. Different ones. But find it. 😈 😇
