Detox in 2020
Today we released a new major version of Detox, 17, which marks the first of a three stage rewrite of Detox’s iOS implementation. It is a process we’ve been working on for quite some time. For the past half a year, we’ve been doing some pretty radical changes in Detox, some for better user experience (which we understood as first party users, must be improved), some for much better performance, and stability. This new release is a great opportunity to share a bit about these changes, and also review changes in other areas of Detox.
Detox is being downloaded over 300K times every month, it is being used by React Native as its E2E testing tool, and by multiple other React Native related libraries and extensions. The following changes are our answer to conversations we’ve had with devs both internally and with users on our GitHub issues section, and what we believe will help Detox users, to be able to understand failures faster, pinpoint issues with a lot less effort than before, write tests easier, and have even greater test stability.
Here’s a rundown of the biggest things we’ve been working on lately, and a mission statement for the things to come.
Android Maturity
Android maturity has been for quite some time now, a very high priority goal for us. Although Detox supports Android E2E testing for over a year now, we can now safely say it is battle tested, in the most extreme conditions we could find (the Wix Android app, which is probably one of the biggest React Native apps in existence, with over 600 screens).
We’ve also closed most of the missing APIs gap with iOS (still a few things missing, but we’re getting there). We learned a lot about how to manage and handle Android Emulators own flaky behaviour, locally and in CI environments, we wrote a guide to setup a robust emulator environment, we worked a bit with the Emulator team at Google to try and pinpoint issues running multiple emulators in parallel, we deleted that guide (tl;dr, use Android Emulator binary v30 and up, and AOSP images, not Google Play Services images).
DetoxNext
DetoxNext is our reimplementation of Detox iOS. It is the biggest change we made in Detox’s history. We are building Detox from the ground up (while trying to maintain its API as much as possible), focusing on a few things:
Parting ways with EarlGrey
Detox has relied on EarlGrey since its inception, for test synchronization, view matching, actions, and expectations. EarlGray is a really powerful testing library, and by using it, Detox managed to be very fast and stable from a very early stage. With that said, EarlGrey posed a few challenges for us as both libs’ development progressed.
- EarlGray’s sync mechanism is polling based, running in an infinite loop, querying all IdlingResources, a thing which results in 100% CPU usage when a test is running. This has posed us with a problem when trying to performance benchmark apps, using Detox as the tool that drives the benchmark test scenario, by merely using Detox, we have altered the benchmark significantly.
- EarlGray 2 now uses XCUITest, running on two processes, communicating via XPC. Although this fits some of the changes we’re making in Detox, the polling mechanism still exists. Also, moving to this model, makes debugging much more complicated.
In DetoxNext we part ways with EarlGrey, bringing in a new sync mechanism called DetoxSync. This is an event based sync mechanism, with a very low CPU footprint on the app/test. DetoxSync is set to be introduced into Detox in the near future. We believe that DetoxNext’s performance will not only help us benchmark in a more accurate way, but will allow for better testing parallelism (more simulators running at the same time).
iOS Device support with XCUITest flavour
DetoxNext will feature two flavours, the first is in process sync/actions/matching, similar to the way Detox behaves today, and the other flavour will feature XCUITest as the action/matching mechanism on an external process, while running sync with DetoxSync inside the app process. This will give us much better flexibility to run Detox on real Devices, and also support out of process view interaction (system alerts, WKWebView (though only limited support), etc’).
DetoxNext also brings another (highly requested) feature:
element(by…).getAttributes()
element(by...).getAttributes() returns a matched view's attributes, which opens up new capabilities to run custom matchers in Node (jest expect for example) rather than inside the device. Here’s an example of getAttribute()‘s return value.
There are more features available in this release, read about them here
DetoxNext first phase has already been merged into master and released in Detox 17.0.0, we would love to hear your feedback!
User Experience and Visibility
Detox config
A new Detox config, adding almost infinite flexibility in configuring Detox in your project (separate config file, multiple configs, true config-file/cli configuration merge and override).
Record timeline trace artifact
Running Detox tests in parallel on multiple simulators/emulators can shortens the overall runtime of a test suite, but we found that this can be improved even further.
Timeline Recording (--record-timeline all) enables recording of tests and events on a timeline, for visual display on the chrome://tracing tool. By laying out this data on a timeline graph, we are able to see how Detox (and your app) are behaving across time, optimize and improve the test runtime as needed.
In this example I ran 3 files of Detox’s test suite on 3 workers:
detox test e2e/03.actions.test.js e2e/04.assertions.test.js e2e/05.waitfor.test.js -c ios.sim.release — record-timeline all -w 3In this graph we can see two interesting findings:
1. One long test suite, and two other very short suites, a way to improve the overall runtime would be splitting the long suite, or distributing some of the tests elsewhere.
2. The first two suites start a bit later from the first one, this is cause by Detox creating and spinning up two Simulators on the fly (I already had one Simulator running).
Crashes/ANRs reports
App crashes and ANRs are reoprted to the test log (instead of timing out a test), increasing crash visibility greatly. Up until that feature, a user could only find crash/ANR details in the device logs.
Custom logger
The new logger redirects all console.logs into Detox's internal bunyan logger, allowing for more metadata to be recorded on every line (pid, time, etc'), and also paves a way for bundling user logs into the timeline trace artifact.
More Notable Features
Choosing Jest Circus as a preferred test runner
I know we used to say that Detox is test runner agnostic, and that you can use whatever test runner you want, and that is kinda still the case, but our integration with jest Circus is by far the most advanced test runner integration we have. Jest Circus proves to be a very robust test runner, with few key benefits over Jasmine, mainly better and more predictable lifecycle hooks. We are investing our efforts in making the experience with Jest Circus as best as it can be, and recommend that use it as well. By using Detox with Jest Circus, not only will you get more accurate artifact collection timings, but you will be able to make changes in test environment before tests begin. A common problem we’ve been seeing quite a lot when we started running Android E2E after iOS E2E, is that our CI registered two runs of the same test, and could not distinguish between the two platforms. The new integration lets us easily add a prefix to a test name according to OS/configuration name for better readability, visibility and test history collection in CI.
Performance Benchmarking
We use Detox internally to drive use cases in our in house benchmarking framework, this is why you see commits like this one, and this one. We incorporate the recording capabilities of DetoxInstruments, and built a system that is able to take multiple recordings data, and compare them failing builds if a certain threshold is being crossed. We hope to open source our benchmarking tools soon. Stay tuned.
Final Words
Internally at Wix, we continue to improve our test coverage (which is already pretty high), and are aiming at “0 manual QA”. Detox is a strategic tool in our way to accomplish this goal.
P.S.
We might have an awesome logo soon…
