Retrying flaky tests
One thing you notice when you have end to end tests in your application is that they can become pretty brittle. You do want to rely on them however and you want to know that if a test has failed it is not a false positive.
Sometimes you just end up running the test suite multiple times to see that it actually passed. Let’s automate this!
There are actually two things we need to do: retry tests on certain failures, and log the tests that have been successful only after a few retries (we do want to improve those in the future).
For the first part we are going to use rspec-retry since it gives us the possibility to configure retries right out of our spec_helper.rb configuration.
Setting up retries
First let’s set up some tests that fail randomly, emulating an environment where a suite of tests is running against an actual deployed environment.
After running bundle install and rspec --init to set up our test suite, we add flakiness_spec.rb to simulate a distributed environment where things might fail without warning.
Running these a few times we see the different results. Let’s retry them on failure.
Logging retries at the end of the suite
We have set up the retries but we want to see all the retried tests in one place, at the end of the suite. In case we have thousands of tests we don’t want to scroll and look at each one.
For this we will create a custom rspec formatter.
Firstly we will mark the flaky tests so we can count them at the end:
Now we can add our formatter.
So a few things are happening here: first of all we have inherited from the ‘documentation’ formatter so that it will do the heavy lifting for us. We are using the :example_passed event in order to paint the passed example a different color if it is flaky. We are also using the inherited deep_summary method in order to add the examples that were flaky, which we can identify thanks to the metadata key we add to them in our Rspec.configure block. The rest is just formatting and coloring.
That is basically it. We can now configure our tests to run multiple times on failure and get a neat report of the once that are still brittle and need improvement.
You can check out the code on Github @ vlad-velciov/retrying_tests.
