Using RSpec’s seed flag to fix a flaky test

TLDR

If your flaky test is due to an extra record in the database (db) that shouldn’t be there and you want to know which test is inserting this record, then you can:

Add the following config to your spec_helper.rb (to provide us with helpful console output — see below):

config.before(:each) do |example|
  puts example
  puts "User count after test: #{User.count}"
end

Run your tests in the order they were run in the failing test run/build:

bundle exec spec --seed [seed number from failing test run/build]

And you’ll see console output like this:

# test name
<RSpec::Core::Example “logs in”>
# count of entries in db after test is run
User count after test is: 0 
<RSpec::Core::Example “makes payment”>
User count after test is: 0
# Aha! If the test below runs before the flaky test, 
# then it is likely the culprit: it is introducing a 
# record into the db that is not being cleaned up after it has run
<RSpec::Core::Example “accesses loan”>
User count after test is: 1

Intro

Hello! I’m Shen a Software Engineer at Funding Circle 👋😄. I’m also a career changer 💪🏾. One of the things I love about writing code is something I couldn’t find in any other career: the concept of automated testing.

I love all types of tests, even failing ones — and especially flaky ones. A flaky test is (my definition): a test that sometimes passes and sometimes fails, independent of changes to the code it is testing.

Most devs will likely have come across a flaky test at some time or another — the common scenario being: a dev tries to merge to master but a test unrelated to their PR change fails the build.

At FC, when we come across a flaky test, we copy and paste its build failure into our #flaky-tests slack channel (the idea being that someone with free time can fix them). So a few months ago, on a sunny Friday afternoon during our learning and development time, I armed myself with an almond croissant and a coffee and picked the latest flaky test in the channel to solve/fi️x ☀️ ☕️ 🥐

The flaky test

Here is the build failure error for the flaky test:

ActiveRecord::RecordNotUnique Exception: PG::UniqueViolation: 
ERROR: duplicate key value violates unique constraint "customers_pkey" 
DETAIL: Key (id)=(43957) already exists.

Translation: the test is trying to insert a Customer record into the db with a particular id (primary key), but there is already a customer in the db with the same primary key. The db should be empty at this point, but because it is not and because it can’t insert duplicate records, it throws an error.

So: we have a record in the db that is not supposed to be there. My first action was to check if we are cleaning the db correctly between test runs.

Database cleaner

Database cleaner is a commonly used gem in rails applications. Its sole responsibility is to make sure the db is clean in between test runs. Looking at our config for database_cleaner:

config.before(:each) do
  DatabaseCleaner.start
end

config.append_after(:each) do
  DatabaseCleaner.clean
end

It looks like database_cleaner is set up correctly — it records the changes that are made to the db before each test and then it cleans up/reverses those changes after each test.

seed

Hmm, so what else could the issue be? Well, so far we know the following:

• Our flaky test is complaining of an unexpected record in the db
• Another test is likely inserting a record into the db that somehow remains in the db after this test has run
• This other test must therefore be running before our failing flaky test

To identify this other test, one idea could be to analyse all the tests that ran before our flaky test in the failing build. But how can we tell which tests ran before our flaky test? At FC we set up RSpec to run tests in a random order (reasons why listed here) which complicates things. This is where RSpec’s “seed” flag comes in.

In every one of our build outputs, RSpec prints the seed number. You can see this for yourself in the console:

➜  rspec dummy_spec.rb --order random

Randomized with seed 55765
..

Finished in 0.00382 seconds (files took 0.12376 seconds to load)
2 examples, 0 failures

# the seed number!
Randomized with seed 55765 

This is RSpec’s way of saying: these tests have been run in a random order. If you want to run the tests in this specific order again, use this number (in this case, 55765)

Almost there…

Awesome! Along with a little extra config in my spec_helper.rb file:

config.before(:each) do |example|
  puts example
  puts "Customer count after test: #{Customer.count}"
end

I can now run the tests in the order they were run in the build and see the state of the db after each test.

After a few sips of coffee and watching the terminal: bingo!

<RSpec::Core::Example "logs in">
User count after test is: 0 
<RSpec::Core::Example "makes payment">
User count after test is: 0
# ...and so on until:
<RSpec::Core::Example "updates the transactions allocated_ref">
User count after test is: 1

We have the culprit: test “updates the transactions allocated_ref ” has left one Customer record in the db after it has run.

The fix

But why is this happening? 🤔 Looking into the spec file for this test, we use a “before all” block:

before(:all) { @customer = FactoryBot.create :customer }

I usually see this implementation in older codebases. This block sets up the db before a group (or specifically, a block) of tests. It’s a way of saving time; if all the tests in a block use the same db state, then we can set up the db once and let the block of tests run against the same state.

However: there is one unintended consequence of using the before all block in our codebase. Remember our database_cleaner config from earlier? It is set up to clean up after each test…

config.before(:each) do
  DatabaseCleaner.start
end

config.append_after(:each) do
  DatabaseCleaner.clean
end

… but it is not set up to clean up after a group of tests. This means the setup in the before all block in the culprit test is essentially invisible to database_cleaner; the customer record is put into the db and because db_cleaner doesn’t know about it, it doesn’t clean up (ie remove) this record after the test has run.

Aha!💡 So the before all block is the problem. The solution? Simple: replace the before all block with a “let” block:

let(:customer) { FactoryBot.create :customer }

In our group of tests, the “let” block is executed before each test and so the inserted customer record will be detected (and cleaned up) by database_cleaner 🙌

Kudos:

Thanks to my fellow engineer pals Vik, Andrew and Mark for reviewing my thoughts and offering feedback on this topic.

Thanks to Adam in the Talent team for encouraging me to give a talk about this topic at LRUG.