How I made my tests run 70% faster

Benny Sitbon

Follow

Jan 2, 2020 · 6 min read

Many apps have test suites that run exponentially slower as business complexity grows. This is painful to developer productivity and resource efficiency.

In this piece, I’ll show you how I’ve made my tests run 70% faster by changing a few lines.

Why Tests Matter

Imagine I have a test suite that verifies all the outputs and side effects of my code. With this kind of cover, I can confidently refactor the code until I’m satisfied with its quality. Confidence allows for faster code writing, legacy code refactoring, and sustainable velocity. It’s critical for software development.

Untested legacy code is a swamp that slows dev teams to a halt. Eventually, it slows whole organizations and even bankrupts them. Thus, removing fear is critical to high-performing teams and companies.

I Heart TDD

As I mature as a programmer, I fall more and more in love with TDD. It reminds me of the scientific method — the successful result is defined before the experiment. So the results won’t affect your judgment.

TDD makes you focus on what’s important — inputs and outputs, not what’s going on inside. We often change the implementation of a class many times during development by breaking it into classes, reversing the logic, or use different data structures. With TDD, you’re confident the refactor didn’t change the outputs. This is why TDD is so powerful.

The Problem — The Tests Were Too Slow

So I got my test suite done — 45 beautiful, shiny RSpec/Minitest tests. But alas, they’re too slow — running at over 25 seconds.

When practicing TDD, you’re running your tests many times (via the red, green, refactor cycle). And 25 seconds is quite a long time to wait for results. This makes my day a lot more boring. What can I do?

Dig to Understand — Why So Slow?

There are two main reasons why my tests were so slow:

1. Active Record: Ruby on Rails makes it too easy to mix DB access inside your classes — this, in turn, leads you to write unit/integration test hybrids, which are significantly slower than unit tests. These are the tests where you test your class but also create objects in the DB in the process. DB access is much slower than executing code, and that, by itself, massively slows your tests.
2. Factory cascade: To create those complex objects in their tests, many RoR devs use the factory-bot gem. It’s a wonderful gem that improves the quality of life and helps to create concise tests by leveraging the Factory pattern. But potentially, it also leads to a phenomenon called factory cascade. A factory cascade is when a factory uses other factories, and those factories use other factories. In a cascade, you can easily find yourself making +30 DB inserts in a single test. (See this awesome post to learn more about the topic.)

So not only am I creating objects in the DB during my tests, I’m creating many of them — and that’s slow.

The test-prof Gem Comes to the Rescue

There’s this cool gem called test-prof. It has a handful of tools that can help you analyze your test suite and improve it. Since there was no heavy lifting in my code, I immediately suspected a factory cascade.

To understand how bad the cascade was, I used the factory profiler the gem provides.

Results of FactoryProf:

For 45 tests, 1,490 objects were created in the DB, which caused these 45 tests to spend over 15 seconds in object creation. That sounds like a ton of objects, right? It definitely is! Keep reading below to see how I cut 70% of the objects and the runtime.

The first thing I noticed is that I’m creating 152 locations, and it takes four seconds — but none of them are explicitly called from my test suite ( top-level is 0).

To stop creating locations like crazy, I had to see which factory is creating these locations for me.

It seems like the invoice factory is one culprit. Since I’m building a class that manipulates invoice data, you can assume I’m calling the invoice factory quite often.

FactoryBot.define do
  factory :invoice do
    some_dependency { "foo" }
    ...
    location

Aha! A prime suspect, indeed. In case you didn’t know — you don’t have to use FactoryBot's create method every time you need an object. You should use it only if you need the object to be persisted in the DB. The other options you can use to generate objects are:

• FactoryBot.build — This won’t persist the object in the DB, so it’s much faster than create, but it’ll persist the object associations in the DB. So it can cause cascades. Ouch.
• FactoryBot.build_stubbed — This one won’t persist the object in the DB, and it won’t persist the associations either. Nice. But there’s a catch: It’ll populate the id column and all fields defined in the factory. When using this method, you won’t be able use your associations without explicitly passing them to the factory — e.g., FactoryBot.build_stubbed(:location, address: address).

OK, let’s pass a stubbed location wherever I create an invoice:

let(:invoice) { FactoryBot.build_stubbed(:final_invoice, location: location) }
let(:location) { FactoryBot.build_stubbed(:location) }

And let’s see the results after using build_stubbed:

Wow. Stubbing locations has also greatly reduced the number of address and bank_accounts created. And we’ve already shaved 30% of the test’s runtime.

Let’s see what else we can tackle:

primary_reservation seems like a good candidate. Again, we see a factory that’s not called explicitly that takes a huge chunk of run time — eight seconds. Let’s see what factory could be the culprit here:

FactoryBot.define do
  factory :line_item do
    some_dependency { "foo" } 
    association :reservation, factory: :primary_reservation

Aha! It’s the line_item factory. Line items are the rows in the invoice — therefore they’re used quite a lot in this test suite. Let’s find the places that don’t need a reservation and assign nil to them:

let!(:line_item) { FactoryBot.create(:line_item, reservation: nil) }

And the locations that do need a reservation will get a stubbed reservation:

let(:primary_reservation) { FactoryBot.build_stubbed(:primary_reservation, location: location) }

Let’s see the results:

Bam! Another six seconds shaved off. Wow, the object creation dropped from 15+ seconds to four seconds — a decrease of over 70%.

Four seconds seems fast enough to stop at this point. Let’s push this commit.

To Sum It Up

• Be careful when using FactoryBot.create. It can cause cascades and slow your tests considerably.
• Always try to use build_stubbed whenever possible, and if that’s not good enough, try to use build.
• If you have to use create, use test-prof to understand how bad the cascade is and try to optimize it using this information.

Thanks for reading!

Resources

• “Red, Green, Refactor” by Codeacademy
• factory_bot
• “TestProf II: Factory therapy for your Ruby tests” by Martian Chronicles
• FactoryProf