Hunting flaky tests 3: Elements in Motion
Included in our list of case studies involving flaky tests encountered at Doctolib are problems related to elements in motion. Below you will find out why these instances can be troublesome, various examples of such issues and possible solutions to the problem.
Working on visual integration tests with Capybara involves interacting with a number of page elements. These interactions are made by calling various Capybara methods such as click_on. A call to click_on results in Capybara successively: finding the DOM element, calculating its coordinates, and then finally clicking on these coordinates. Because the movement of the element is not taken into account, at the time of the click, the coordinates may no longer reside within the element.
This has been the most frequent source of flakies at Doctolib, responsible for roughly 25–30% of our flakiness.
Examples
Moving element due to CSS animation
For instance, when you click on a dropdown with an slide down animation and, following that, click on a list item while the list is deploying, the item will be in motion during the click.
Mobile page transitions
For example, when you login onto the Doctolib mobile app, the home page will appear sliding up from the bottom. This page transition will not be taken into account by Capybara and all items will be in motion during the click as a result.
Moving element due to an Ajax call modifying the DOM
Sometimes, we dynamically show elements based on the result of an Ajax call. For example, when our appointment modal is opened, it checks the patient’s status using Ajax. Upon the return of this call, it will decide whether to insert a button or not. That means any DOM elements under the new button can suddenly be shifted down. If this movement takes place during an attempt to click on one of them, the click will fail.
Solutions
One possible solution would be to wait for the completion of all movement. In this scenario you would have to be aware of any applied CSS animations within the page you are testing, as well as the type of transitions between each mobile screen you are testing. Having to keep all of these additional variables in mind is far from ideal, so, in order to play it safe, you would have to add a wait between each line of your tests.
A more realistic solution is simply to remove movement as a variable in your tests. The drawback here is that you will not be simulating and testing within the context of a real user experience.
How to remove specific movements from the above examples:
Disable CSS and jQuery animations
We have a middleware to disable CSS and jQuery animations on all pages:
Disable mobile page transitions
We improved the Doctolib open source “react-router-transitions” library by simply adding a parameter to switch on or off all page transitions.
Wait for Ajax calls modifying DOM
In the case of the insertion of a button as a response to an Ajax call, any test interacting with the elements below the button will be flaky. It is extremely difficult to determine the link between this insertion and any flaky results. Keeping a visual record (we use Browserstack), allows you to keep track of each test run. Reviewing the video recording of flaky tests helps to identify the problem quickly. Once you identified the element that appears (or disappears), you must add an assertion on its final state before interacting with one of the moving elements in the test. See other post on this subject.
