Flaky Tests: Retrying Failed Tests with Cypress

6 min readJan 20, 2020

A couple of weeks ago I saw Ben Kucera’s tweet that introduces retry feature for failed Cypress tests. Since this was one of the top “must-have” features I’ve been waiting for, I wanted to give it a try and guide you through with it while providing some brief information at the same time. But before that, let’s answer the following “frequently asked questions”:

What is a “flaky” test?

According to Google’s definition; a flaky test is a test that exhibits both a passing and a failing result with the same code and the same configuration, periodically.

Is it a good strategy to retry failed tests?

It depends. If you are going to evaluate that related test case as “Passed” and if you don’t care about previously failed attempts’ detailed results and you don’t want to analyse the root cause in-depth then it’s definitely not a good strategy for you. So basically, you shouldn’t use retries just to sweep flaky tests under the rug, that should not be the goal, of course. The main purpose should be to understand the root cause and eliminate flaky tests by finding proper answers to questions like below:

- Is the retry count always the same for each and every run?
- Is the failure message always the same or does it also change depending on the attempts?
- Is the related flaky test case dependent on any other test cases?

Answers to those questions may give you some hints about if it is an infrastructural/environmental issue you are running through or it is a bug in the test code. If you’re lucky enough you can diagnose the cause of flakiness quickly but be prepared for some time-consuming research.

On which environments should I use “retry”?

It is always the best phase to spot a flaky test before you go on live. Test, staging, pre-prod or whatever you call for that environment, you can use retry feature at this stage. Personally, I would not suggest using this feature on production environment tests cause flaky tests may mean flaky user experience which may result in more serious user-oriented problems.

Why should I use “retry”?

Using “retry” doesn’t mean that you don’t trust your code. Tests may fail, or be flaky, due to environmental reasons as well. The main purpose here is; to put another layer to monitor the whole testing process, in-depth. If your test code is successful already, you don’t need to worry about anything, right? Using retries also doesn’t mean that it will extend the time that tests take, it only retries when there is a failure.

As mentioned above, the main purpose should be to find out the root causes of flaky tests, resolve them and make historical analysis accordingly to prevent similar issues that may happen in the future.

What are the possible causes of flakiness?

Infrastructure: Tests may be flaky because of several external reasons. The most common ones are; network problems, database outages, CI related incidents or temporary testing framework issues. So, make sure that everything is infrastructurally stable before running your tests.
Dynamic Content/UI: I didn’t experience this kind of problem after started using Cypress since it manages asynchronous elements quite well. For instance; while testing a toast message I don’t need to use cy.wait() and wait for the related element to load or be visible. Cypress automatically understands the use of cy.get() and waits for some time for the server to respond and for the loading of the related element before moving on. However, if you’re using waits somehow, please pay special attention here. You may need to check and rewrite your test code to get rid of them, to shorten your test execution time and to be more stable at the end.
Data Shortage: Test Data Management is one of the most important processes of test planning. During the test cycle, you should have full control over the data, always. So, always prepare your test data beforehand, which serve directly and specifically to your needs. Do not try to run your tests with an “enough” amount of data. Try to avoid using the same data (users, accounts, roles, authorizations, etc.) over and over again for all the different scenarios and flows.
Date/Time Issues: If you’re working with a product which is available under different time zones or if your product is highly dependent on public holidays in several different regions, always bear in mind that you need to implement a logic where your tests should pass at any (t) time.
Poorly Configured Parallel Test Runs: If you have a huge number of test scenarios and flows, it can take a really long time for tests to complete while you’re running them serially. In this case, configuring parallel test runs across several virtual machines can save a lot of your time. Here, test order dependency plays the main role. Sometimes a test may pass while running serially and fail under different execution order. You need to split up your test suite and group your test cases carefully to get the most out of parallelization. Otherwise, parallelization could be your enemy and source of flaky tests while generating unstable test results and wasting your time at the same time. For instance; if test case X has a direct and continuous impact on test case Y respectively, then you should group them and make them run one after another [G1(X>Y)]-[G2(A>B)]. If you somehow poorly configured your test parallelization, and test case Y tries to run in the first group before test case X completes its cycle in the second group [G1(Y>A)]-[G2(B>X)], then this will disrupt the test execution process. Please pay special attention here as well.
Caching: Don’t forget to clear cache before starting your new test run, just as a precaution. I always clear cache before each test in the block, via using the cy.clearCookies() function. So, I know that I’m looking at the code I think I’m looking at.
Setup/Cleanup Environment: Always set up the expected environment depending on the related test case and after completing the scenario make sure that you tidy everything up for the upcoming tests. Otherwise, consecutive tests could get affected by the previous tests’ leftovers. For instance; when you add items to the cart for a specific test scenario, make sure that you clean the cart once you’re done and even log out if necessary. The next test case’s precondition could be an empty cart or a clean state. In other words, try to approach every test case as a whole.

Cypress-Plugin-Retries Installation

After a brief introduction of the term “flaky test”, mentioning the possible causes of flakiness and how to approach them, let’s move on with the plugin installation part to catch them.

1. Let’s start with adding the Cypress-Plugin-Retries to devDependencies. Navigate to your project and run the following command in your terminal:
npm install -D cypress-plugin-retries
You’ll see the installation confirmation message pretty quickly. If there are any vulnerabilities found, you may want to run npm audit fix command afterwards, as well.

2. Now, navigate to cypress/support/index.js file and add the following part:

// Retry failed tests:
require('cypress-plugin-retries')

3. Then, in order to set the retry value for all of your spec files, we’re going to use the cypress.json file and set the "env" key as follows:

{
  "env":
  {
    "RETRIES": 2
  }
}

4. Voila! Just commit your changes and run the test suite again. Now you’re ready to catch all the flaky tests. But let’s hope you don’t have any…

Conclusion

A flaky test is a common problem and it could cause severe problems when the codebases grow gradually. If you’re not going to invest some time and deal with flaky tests then do not retry them, do not try to hide them either. Let them fail, in peace.

Do not forget; reliability is the key value for us and our tests cannot be reliable when we have a set of flaky tests behind us.