Hunting flaky tests 2: Waiting for ajax

See table of contents here for more flaky test resolution examples.

If you are lazy, blindly wait for ajax

From experience, we know that Ajax calls create flaky situations. Most of the time, users are slow. They will not create concurrency issues between mouse interactions and Ajax calls. Imagine you need to click button 1, and then click button 2. If you are a human, the application will have the time to process the first click before receiving the second click. But when Capybara runs a test, it is extremely quick. It can click button 2 before the application finishes processing the first click. The second click will then interact with the application in an undetermined state, and will have undetermined result.

In response to this, we strategically placed calls to wait_for_ajax in areas that trigger Ajax calls. For instance: in our most used login helper, in various other helpers used to open modals, to select value in a dropdown that refreshes another part of the app, etc.

To implement this wait_for_ajax, we went through a few iterations. We started with a naive implementation found on the Internet:

def wait_for_ajax
  Timeout.timeout(Capybara.default_wait_time) do
    loop until page.evaluate_script('jQuery.active').zero?
  end
end

But there were two problems with this approach:

• We used the Ruby Timeout module, but encountered some inexplicable behaviour. At the time, we did not dig further into finding the source of this behaviour, however, it may have been caused by any one of the drawbacks listed here. To replace the buggy module, we now use a simple loop with a sleep and break the loop when the process clock reaches the maximum wait time.
• We only waited for one jQuery instance, but due to technical debt, we use two different versions. Plus, we have a Javascript helper (httpClient) that makes direct access to XMLHttpRequest. We had to wait for all of these possible Ajax query sources.

Below is our current implementation of this “wait for all possible Ajax queries”:

This was only a quick fix: you should not wait blindly, but should assert a precise state. In time, we replaced most of the wait_for_ajax by Capybara assert helpers that inherently include a wait.

If you are conscientious, use Capybara waiting assertions

In our codebase, many flaky occurrences were caused by a lack of proper asserts with native wait. You should ensure an assert with a wait is present after modifying the state of the application:

• When an Ajax call influences the rest of your test, wait for its return with a natively waiting Capybara assert helper. For example, if your test clicks on a ‘save’ button, add a call to assert_text('your stuff has been saved.'). If your test opens a form that will be dynamically filled, call assert_selector(:field, 'name', with: 'something'). If your test selects a value in a checkbox that will filter a result list, call assert_selector('.results .result'), etc.
• Most of the Capybara helpers natively wait, but you must provide them with the full state you are waiting for. To verify a field disabled state, call assert_selector(:field, 'name', with: 'something', disabled: true). To wait for a checkbox state, call assert_selector(:field, 'agreed', checked: true), etc.

Real world examples

• On one form, an Ajax call is triggered upon opening, enabling or disabling fields. Since this simple assert does not natively wait, tests failed when the Ajax call returned after the assert:

# Flaky (wait for visibility, but not for disabled state)
assert find('#LastName').disabled?
# Quick fix
wait_for_ajax
assert find('#LastName').disabled?
# Correct (capybara native with wait)
assert_selector(:field, 'last_name', disabled: true)

• Or if a field is dynamically filled with a value:

# Flaky
assert_equal('12 mn', find('.dc-appointment-duration input').value)
# Correct (Capybara native with wait)
assert_selector(:field, 'appointment_duration', with: '12 mn')

• To wait for a modal to open after a click that triggers a call:

click_at(x, y)
assert_selector('.dc-modal-dialog')

• To wait for a button to become enabled:

# Flaky (wait for visibility thanks to find, but not enabled state)
find('.merge-patient').click
# Correct (will wait for the button to be enabled)
click_button('Merge patients')

• You can find more examples here.

There are some instances where a blind wait must be used, which is something we will cover in the next post.