Detox: Writing Stable Test Suites

Two months ago I listened to episode 180 of React Native Radio. In that episode the panel had a discussion about Testing Strategies, Tools and Frameworks. Detox was mentioned right at the beginning, and over the course of the episode I heard a comment stating that Detox tests can “hang” and time out, and that they are flaky sometimes, I can relate to that... These are issues we also see internally at Wix quite often, and mostly there are quite straight forward solutions to most of them, there’s also a pretty sophisticated set of tools to help the tester understand where the issue is coming from. All of that made me think, maybe we’re not doing a good enough job in documenting, explaining and educating, after all, the whole premise of Detox is to have stable E2E tests, and if the testers don’t have that experience, we need to help them fix that.

So this post is aimed at helping you making your Detox tests as solid and as stable as possible. It will include tips on how to write a stable test, issue resolution, and some more advanced usage patterns that can help pinpointing issues and resolving them.

It is also an open invitation to have a discussion here about your experience, after applying all the methods mentioned below, are you still having an unknown flaky behaviour?

Ground Rules

Let’s begin with setting up a baseline, by having these few rules, everything will become much easier later on.

No Consistent Input === No Consistent Output 🤷‍♂️

An app might have multiple inputs that might change its behaviour when running: varying responses from servers, some applications might have experiments or A/B tests executing different code paths in the app, causing it to behave differently during different runs. You must make sure these are handled and configured to be identical throughout all iterations of a test, whether its locally or in CI. At Wix, we have developed an experiment override mechanism that configures a predefined experiment blob in the app per test, thus insuring we never get different behaviour while the app is under test.

Write Isolated Tests

This is probably the most important tip I can give you, make sure every test is starting fresh, make sure it is not dependant on execution of previous tests.

1. Restart your app before every test, use beforeAll to setup the suite, use beforeEach to setup each test.
2. Just like a best practice of every other type of testing, make sure you have setup, execution validation and cleanup phases in your test, make sure you finish with a validation phase (expect)

Here’s a simple example of a non-isolated test:

Notice this suite only launches the app once, it has a runs an expectation, and tries to go back to the previous screen at the end of every test. If First test fails at the expectation line, it will not clean itself up and Second test will start from a state it is not designed to start from, thus failing immediately.

Here’s the same test suite, but now isolated:

Simply restart the app before every test, and don’t worry about cleaning up or getting to a specific state at every test’s ending. As a rule of thumb, end every test with an expect line.

Now that we have a cleaner and more consistent testing environment to work with, let’s take a look at some common issues.

Issue: Tests hang and time out

There are a few reason tests can hang, most of which are related to the behaviour of the app-under-test. Now’s a good time to remind the concepts of Detox’s grey box mechanism. Detox manages sync between the app and test code, and waits for the app to go idle. If there’s a network request in flight, a looping animation or some kind of infinite loop, Detox will continue waiting, and the test runner will time out.

Detox (iOS) has a feature that prints synchronization status when an action/expectation takes a significant amount of time on the device. On previous versions you would need to add -d, --debug-synchronization to your detox test command for it to kick in (we didn't keep it on by default since it wasn't 100% stable). With Detox 18, we rewrote the entire sync mechanism, and this is now turned on by default.

Here’s an example of that output on Detox 18:

The system is busy with the following tasks:
​
Dispatch Queue
⏱ Queue: “Main Queue (<OS_dispatch_queue_main: com.apple.main-thread>)” with 1 work blocks
​
Run Loop
⏱ “Main Run Loop”
​
One Time Events
⏱ “Network Request” with object: “URL: “http://localhost:9001/delay/3000””

In this example we can see there an ongoing network request to http://localhost:9001/delay/3000. Such a request will block Detox from continuing with the test execution until it finishes.

And another one:

The system is busy with the following tasks:
​
Delayed Perform Selector
⏱ 2 pending selectors
​
Dispatch Queue
⏱ Queue: “Main Queue (<OS_dispatch_queue_main: com.apple.main-thread>)” with 1 work blocks
​
JS Timer
⏱ Timers: (
    9391
)
​
Run Loop
⏱ “Main Run Loop”
​
Timer
⏱ Timers: (
    "Timer with fire date: 2021-01-15 02:22:41 +0200 (time delta: 1.499997019767761) interval: 0 repeats: NO>",
    "Timer with fire date: 2021-01-15 02:22:41 +0200 (time delta: 1.499997019767761) interval: 0 repeats: NO>"
)
​

This example is of a test that is waiting for a (very long) animation to finish.

Or an ActivityIndicator that runs forever:

The system is busy with the following tasks:
​
Dispatch Queue
⏱ Queue: “Main Queue (<OS_dispatch_queue_main: com.apple.main-thread>)” with 1 work blocks
​
Run Loop
⏱ “Main Run Loop”
​
UI Elements
⏱ 0 layer animations pending

There are more cases of infinite animations that can interfere with the app-under-test getting idle, but these are a few representative examples. Make sure you don’t keep those in your view layout when reaching the final “loaded” state.

The main caveat of this tool is that it can’t pinpoint exactly to the user code responsible for that behaviour, as this is an impossible mission for Detox. The fact that React Native is a declarative framework means that the configuration code is separated from the actual code that executes these commands, thus it can only show info about the latter, which is usually not the user’s code, but the framework’s. The bottom line here is that you have to know your app, and might need to fiddle with it a bit to find the issue. We usually begin with trying to pinpoint what action in the app triggers that endless loop, and follow the code path.

Note: Debug sync only exists on iOS, we’ll get to the Android version, soon hopefully.

Issue: Tests on CI are slower, network issues, timeouts, slowness brings up a lot of issues we would not see when developing locally.

This is a very broad issue and there isn’t a one size fits all here. Again here, you have to know your app, but these tools will give you a lot of insight.

Detox offers a comprehensive artifact collection system: video recording, device log collection, on demand screenshots, timeline trace artifact, and a Detox Instruments recording.

This is how we set up our artifacts collection internally (we collect artifacts by default only when running in CI).

log

Log creates a file artifact with the entire output of the process, on iOS its pretty easy to find your console.logs by looking for com.facebook.react.log.javascript tag or com.facebook.react.log.native for all things native NSLog or os/log.

Example:

2020-12-13 21:11:55.692 I  WixApp[57321:c854a] [com.facebook.react.log:javascript] Selected locale is:en
2020-12-13 21:11:53.750 I  WixApp[57321:c8491] [com.facebook.react.log:native] Could not find branch.json in app bundle.

screenshot and video

These can give a quick overview on how the app looks like when the test fails, it makes it easy to find visual problems (overlaps in views, infinite loaders, etc).

instruments

This one is invaluable, especially for event collection and network data. This can help determining if there was a network issue during the test, timeout from server, server errors, etc.

In this example we can see a 401 unauthorised response from one of the servers, this can definitely be a cause for a failing/flaky test.

uiHierarchy
(Requires Xcode 12 to open) This will dump the view hierarchy, right after an action/expectation failure. It is especially helpful for cases of visibility assertion failures, if a screenshot is not helpful enough to determine the view's state.

...Failed: View "<RCTScrollView: 0x7f8d32296d70>" is not visible: view does not pass visibility threshold (0% visible of 75% required)

Still having issues? Retries

Retries are a great way to patch a flaky test which its culprit cannot be found and fixed. There are two approaches to retries:

1. Per Test
   Retry every failed test, assuming it is isolated from other test cases and that its initial state is not dependant on previous tests in that suite (follow the ground rules!). By using Jest (with Jest Circus) as your test runner, you can set a per-test retry strategy (https://jestjs.io/docs/en/jest-object#jestretrytimes).
   jest.retryTimes(2); //3 tries in total
2. Per File
   Use detox cli withdetox test --retries <N> [Jest Circus Only] Re-spawn the test runner for individual failing suite files until they pass, or <N> times at most. A less recommended approach, as it will re run the whole test file upon failure, but if there's absolutely no way to isolate the tests from one another, then this solution is also available.

To make each of these work, you’ll need to make sure you use Jest Circus, if you’re still not using it, here’s how to migrate.

Final Words

We would love to hear from you in the comments here about your experience, and if this managed to help you stabilizing your E2E test suite, or at least give a good enough insight as to what’s going on with your app while its being tested.
If you have any cool idea of improvement, let’s discuss it in an enhancement issue on Github.
And as I wrote at the beginning of the article, this is also an open invitation to have a discussion here about your experience, after applying all the methods mentioned below, are you still having an unknown flaky behaviour?

Update

I was having a conversation about this article with Jamon Holmgren and Harris Robin Kalash on episode 189 of React Native Radio

For more engineering updates and insights:

• Follow us on: Twitter/detoxE2E | Twitter/WixEng | Facebook | LinkedIn
• Read our blog
• Visit us on GitHub
• Subscribe to our monthly newsletter
• Subscribe to our YouTube channel
• Follow our Medium publication