Tips to improve speed of your test suite

Published in

Appaloosa Store Engineering

4 min readJul 27, 2017

At Appaloosa, most of our tests are written with Rspec, but we still have a certain amount of legacy Test Unit tests. Our Rspec and Test-Unit suites take up to 10 minutes each, so a full test suite runs in around 20 minutes. That’s a lot.

We recently decided to invest time on improving our tests to gain time when developing.

https://www.flickr.com/photos/steamster/20170456241/

Recurring problems and how to avoid them

build vs build_stubbed vs create

When we looked at our test logs, too many SQL queries were made. We are using FactoryGirl and too often we create a record. Interacting with a database costs a lot of time. In our RSpec configuration file, we have the following setup:

This is why we changed to build or build_stubbed whenever possible.

From the documentation :

build_stubbed is the younger, more hip sibling to build; it instantiates and assigns attributes just like build, but that’s where the similarities end. It makes objects look like they’ve been persisted, creates associations with the build_stubbed strategy (whereas build still uses create), and stubs out a handful of methods that interact with the database and raises if you call them. This leads to much faster tests and reduces your test dependency on a database.

from : https://robots.thoughtbot.com/use-factory-girls-build-stubbed-for-a-faster-test

You should try that approach on any test that does not require a model to actually be inserted in your tests database. In those cases, it’s much faster.

before(:each) vs before(:all)

A before is the best place to setup multiple tests. But when your setup is particularly DB-heavy, with a lot of models being created, you should consider executing it once, for all of your assertions. By default before is before(:each). But careful when using before(:all) because it introduce a global state between your tests.

Avoid unnecessary let!

In Rspec you can use let.

Use let to define a memoized helper method. The value will be cached across multiple calls in the same example but not across examples.
Note that let is lazy-evaluated: it is not evaluated until the first time the method it defines is invoked. You can use let! to force the method’s invocation before each example.

Sometimes it’s better to put let code into a before(:all) block with all the requirements you need to avoid multiple calls.

Try to remove the bang when refactoring your test, it will avoid forcing invocation before each example.

Parallel tests

Michael Grosser made a gem called parallel_test based on parrallel. ParallelTests splits tests into even groups (by number of lines or runtime) and runs each group in a single process with its own database.

We tried using it but our project doesn’t fit easy with parallel_tests. We have lots of random failures (classes not loaded, HTTP calls not stubbed…). We spent a few hours trying to fix all of those failing tests but we stopped because we had always new surprises.

Useless factories associations

We noticed that we were creating way too much records with FactoryGirl. For example we have code like this :

When run in a Rails console, you immediately realize the problem:

That’s a lot of time-consuming DB calls. So we started removing these associations, set by hands inside test or turn them into traits:

We did that for most of our factories. This saved around 4–6min on the all test suite. Some factories remain unchanged but it can be tricky when they are used in tens of tests.

We also started using FactoryDoctor. FactoryDoctor is a tool developed by Vladimir Dementyev to inform you when you are creating useless data in tests. It’s not perfect but it will often inform you about useless associations or unneeded create.

Logs

We watched “Run test run” by Vladimir Dementyev and learned a lot. In his talk we recommend for watching, he mentioned another script: FactoryProf. FactoryProf is based on Stackprof. We used this tool before, but never for a test. The script showed promising results:

Half of the time is spent in DB tasks, but also 13% in writing logs. So we changed the log level in our config (in config/environment/test.rb). We launched FactoryProf again. Unsurprisingly, Logger::LogDevice#write was not present anymore. With this little tweak, we saved 2 minutes!

Conclusion

When it comes to speeding up tests, there is no silver bullet (like it’s often the case in software development). Our code base is more than 6 years old, and it’s quite a hurdle to fix something that was decided so long ago. We know the best we can do now is to fix the unoptimized tests as we meet them, but it’s a big task.

What’s interesting is that this task has led the team to a lot of discussions about how we test our code, how and when we rely on database, when and why we should mock…