Hunting flaky tests 5: Actions after delays
Included in our list of case studies involving flaky tests encountered at Doctolib are problems triggered by delays. Below you will find out why these instances can be troublesome, various examples of such issues and possible solutions to the problem.
Test speed is not a controllable variable. If you have a functionality that will execute something after a certain delay, you will not be able to test it reliably. Consider a situation where something is supposed to end after 5 seconds. What happens if your test is very slow and takes 6 seconds to assert for it?
Examples
RackAttack
We use the “rack-attack” gem, which allows you to define rules. These rules can throttle requests considered as offending. These rules are time based, for example, one will block the offenders if they perform more than 3 invalid requests in less than 5 seconds.
We test these rules, but sometimes a test will run so slowly that the number of requests within the specified time can not be reached and the test fails.
Javascript `setTimeout` usages
When a user saves an appointment, we display the flash message “Your appointment has been saved” which disappears after 5 seconds.
Some of our tests rely on it to check if the action was successful, by asserting the presence of the text “Your appointment has been saved”. Unfortunately, the test can take up to 6 seconds between the click on the save button and the message assertion.
Solutions
RackAttack
As a quick fix, we changed the rack-attack configuration in a test environment. We raised the 5 seconds period to 1 hour, so that the test can never fail. But there are still two problems with this approach. First, it makes the test work on a “different code” than that of the production. This is not ideal, the test should be as close as possible to the real case. Second of all, this fix does not give us the ability to test for the end of the throttling period, since we would have to make the test sleep for 1 hour to wait for the end of the ban.
A better solution is to use the Timecop ruby gem. At the start of the test, we slow the flow of time by 100 and when we want to test the end of the throttling, we travel a few minutes in the future.
# Slow time by 100
Timecop.scale(1 / 100.to_f)
...
# Travel to the future!
Timecop.travel(5.minutes)Javascript `setTimeout` usages
Timecop is very useful to test time dependant cases on ruby, unfortunately, it does not influence JavaScript time. Since our flash is controlled by a Javascript timer, we cannot use Timecop here.
Instead, we are forced to disable each Javascript timeout manually in the production code. We switch timeout values to 0 or infinite, depending on the behavior we need in each case. For our “appointment saved” flash confirmation, we prefer that it never disappear, as we base some of our tests on it. If we need to click on something behind it, we have to close it explicitly in the Capybara test.
const MESSAGE_TIMEOUT = (getEnvironment() === ‘test’) ? Number.MAX_SAFE_INTEGER : 5000
