Cut Your RSpec/Minitest Runtime With TestProf

How I made my tests run 70% faster

Benny Sitbon
Jan 2 · 6 min read
Photo by Lukas Blazek on Unsplash

Many apps have test suites that run exponentially slower as business complexity grows. This is painful to developer productivity and resource efficiency.

In this piece, I’ll show you how I’ve made my tests run 70% faster by changing a few lines.


Why Tests Matter

Untested legacy code is a swamp that slows dev teams to a halt. Eventually, it slows whole organizations and even bankrupts them. Thus, removing fear is critical to high-performing teams and companies.


I Heart TDD

TDD makes you focus on what’s important — inputs and outputs, not what’s going on inside. We often change the implementation of a class many times during development by breaking it into classes, reversing the logic, or use different data structures. With TDD, you’re confident the refactor didn’t change the outputs. This is why TDD is so powerful.


The Problem — The Tests Were Too Slow

When practicing TDD, you’re running your tests many times (via the red, green, refactor cycle). And 25 seconds is quite a long time to wait for results. This makes my day a lot more boring. What can I do?


Dig to Understand — Why So Slow?

  1. Active Record: Ruby on Rails makes it too easy to mix DB access inside your classes — this, in turn, leads you to write unit/integration test hybrids, which are significantly slower than unit tests. These are the tests where you test your class but also create objects in the DB in the process. DB access is much slower than executing code, and that, by itself, massively slows your tests.
  2. Factory cascade: To create those complex objects in their tests, many RoR devs use the factory-bot gem. It’s a wonderful gem that improves the quality of life and helps to create concise tests by leveraging the Factory pattern. But potentially, it also leads to a phenomenon called factory cascade. A factory cascade is when a factory uses other factories, and those factories use other factories. In a cascade, you can easily find yourself making +30 DB inserts in a single test. (See this awesome post to learn more about the topic.)

So not only am I creating objects in the DB during my tests, I’m creating many of them — and that’s slow.


The test-prof Gem Comes to the Rescue

To understand how bad the cascade was, I used the factory profiler the gem provides.

Results of FactoryProf:

Ouch!

For 45 tests, 1,490 objects were created in the DB, which caused these 45 tests to spend over 15 seconds in object creation. That sounds like a ton of objects, right? It definitely is! Keep reading below to see how I cut 70% of the objects and the runtime.

The first thing I noticed is that I’m creating 152 locations, and it takes four seconds — but none of them are explicitly called from my test suite ( top-level is 0).

To stop creating locations like crazy, I had to see which factory is creating these locations for me.

It seems like the invoice factory is one culprit. Since I’m building a class that manipulates invoice data, you can assume I’m calling the invoice factory quite often.

FactoryBot.define do
factory :invoice do
some_dependency { "foo" }
...
location

Aha! A prime suspect, indeed. In case you didn’t know — you don’t have to use FactoryBot's create method every time you need an object. You should use it only if you need the object to be persisted in the DB. The other options you can use to generate objects are:

  • FactoryBot.build — This won’t persist the object in the DB, so it’s much faster than create, but it’ll persist the object associations in the DB. So it can cause cascades. Ouch.
  • FactoryBot.build_stubbed — This one won’t persist the object in the DB, and it won’t persist the associations either. Nice. But there’s a catch: It’ll populate the id column and all fields defined in the factory. When using this method, you won’t be able use your associations without explicitly passing them to the factory — e.g., FactoryBot.build_stubbed(:location, address: address).

OK, let’s pass a stubbed location wherever I create an invoice:

let(:invoice) { FactoryBot.build_stubbed(:final_invoice, location: location) }
let(:location) { FactoryBot.build_stubbed(:location) }

And let’s see the results after using build_stubbed:

Wow. Stubbing locations has also greatly reduced the number of address and bank_accounts created. And we’ve already shaved 30% of the test’s runtime.

Let’s see what else we can tackle:

primary_reservation seems like a good candidate. Again, we see a factory that’s not called explicitly that takes a huge chunk of run time — eight seconds. Let’s see what factory could be the culprit here:

FactoryBot.define do
factory :line_item do
some_dependency { "foo" }
association :reservation, factory: :primary_reservation

Aha! It’s the line_item factory. Line items are the rows in the invoice — therefore they’re used quite a lot in this test suite. Let’s find the places that don’t need a reservation and assign nil to them:

let!(:line_item) { FactoryBot.create(:line_item, reservation: nil) }

And the locations that do need a reservation will get a stubbed reservation:

let(:primary_reservation) { FactoryBot.build_stubbed(:primary_reservation, location: location) }

Let’s see the results:

Bam! Another six seconds shaved off. Wow, the object creation dropped from 15+ seconds to four seconds — a decrease of over 70%.

Four seconds seems fast enough to stop at this point. Let’s push this commit.


To Sum It Up

  • Always try to use build_stubbed whenever possible, and if that’s not good enough, try to use build.
  • If you have to use create, use test-prof to understand how bad the cascade is and try to optimize it using this information.

Thanks for reading!


Better Programming

Advice for programmers.

Thanks to Shem Magnezi and Zack Shapiro

Benny Sitbon

Written by

Software Engineer, Payments @WeWork

Better Programming

Advice for programmers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade