Detox or Appium — Which Test Tool is Top?

Photo credit to Jonas Leupe on Unsplash.

It would be a huge understatement to say that mobile devices are popular. As of the end of 2022, almost 5 billion people are online on mobile devices. Naturally, Apple’s iOS and Google’s Android operating systems lead the charge, with smartphones on either platform acting as the nucleus of our everyday lives.

This extends to our work within Digio (and the wider Mantel Group). Many of our clients seek cutting-edge, high-quality mobile experiences in industries such as finance and retail. To ensure the quality of our deliverables across Digio, we employ automation-heavy quality assurance strategies to ensure a sustainable approach that outlives the end of our project involvement and frees up our team members to focus on delivering great software without being bogged down by an over-reliance on manual testing. Mobile applications are no different, and while we have numerous solid options for mobile automation tooling, we took it upon ourselves to assess each tool for its viability in a fast-paced, fast-changing environment.

Comparing Mobile Test Automation Frameworks

We started off by identifying a list of tools to test mobile applications, including both open-source and commercial offerings. Our work with Digio often covers cross-platform mobile applications developed with frameworks such as React Native or Flutter. However, we would also include native iOS/Android apps with officially-supported frameworks for completeness. Our team set about brainstorming some options, and we identified the following:

Detox: iOS + Android, open-source

Appium: iOS + Android, open-source

Espresso: Android only, open-source

XCUITest: iOS only, proprietary (but free with iOS SDK)

EarlGrey: iOS only, open-source

Digital.ai: iOS + Android, proprietary

Perfecto Mobile: iOS + Android, proprietary

Calabash: iOS + Android, open-source (unmaintained)

Eventually, after whittling down the options, we settled on looking at two in more detail; Detox and Appium. As open-source offerings with support for iOS, Android and cross-platform apps, they would be far easier to roll out on client engagements than many proprietary offerings, and both were maintained actively with strong developer support. While Espresso and XCUITest are both more appropriate for native projects on their given platforms, the fact that many mobile engagements use cross-platform frameworks would justify having a similarly flexible automation framework in place, and the fact that both Espresso and XCUITest are maintained by the developers of their respective platforms makes their compatibility somewhat of a foregone conclusion.

Detox vs Appium — First Impressions

Detox and Appium feature different approaches to mobile automation. Detox touts itself as a “gray-box automation framework” targeted at React Native apps (though in our testing, it did work outside of an RN context) that observes the state of a running app to reduce test flakiness. Appium, on the other hand, is a more conventional black-box framework and shares more than a few similarities with web automation stalwart Selenium and relies on the test script to handle unusual conditions. This extends to its portability, where Appium can support scripting in any language Selenium supports (including JavaScript, Java, C#, Obj-C, Ruby and Python). In contrast, Detox’s approach locks it into JavaScript/TypeScript (fitting in with its tight focus on React Native).

Detox’s Gray-Box Approach

Detox and Appium’s different approaches to testing have repercussions on the types of applications that they can test. Detox works by installing a library within the application and communicating with the library inside the application’s memory. This can prove problematic for certain applications; on our assessment, we noted that the Android installation was more complex than the iOS one, and both require a non-trivial amount of mobile knowledge. It also limits the scope of the test to the running application and not to anything else; testing other apps, working with elements such as webviews (which technically sit outside of the application’s process space) and interacting with system constructs are largely unsupported, which can prove problematic in certain contexts. Detox needs the operating system to support communication between the internal library and an external Detox server; this is not an issue with Android, which happily uses the Android Debug Bridge protocol to talk to both emulators and real devices, but iOS’ architecture forbids this on device. As such, only iOS simulators are supported. This is especially problematic when running Detox tests on device farms, and any CI/CD integration to support both OSes would need to be strictly emulator/simulator-only.

Appium’s architecture lends itself well to testing system-level concerns and cross-application concerns, a current shortcoming of Detox.

Appium’s Black-Box Approach

Appium, on the other hand, leverages the native mobile automation framework capabilities. This is somewhat of a double-edged sword; while synchronisation of the app state is impossible (as it would require an understanding of the app’s internal workings), the separation does allow it to support out-of-process execution. This can even allow switching between apps as part of the test and is fully compatible with real devices (and by extension, device farms). The engineers on our assessments encountered some issues with installation that weren’t made obvious earlier. While tools like Appium-Doctor exist to troubleshoot configuration issues, it does require some legwork to get things running smoothly.

Detox vs Appium — Stability

Detox’s Synchronisation Mechanism

Thanks to its grey-box approach, Detox was written from the ground up for stability. By default, Detox will wait for actions like API calls and animations to finish before attempting to execute the next step in a test. It can sometimes get confused in certain cases; on a previous engagement that featured LaunchDarkly on a React Native application, Detox had paused a test and waited for LaunchDarkly to finish executing. However, as the implementation on mobile uses a WebSocket connection, Detox was left waiting until the test itself timed out. While certain URLs can be whitelisted, this approach will only work for certain scenarios, and the team was forced to rewrite the application such that LaunchDarkly would be disabled as a debugging step. Detox does offer an option to disable synchronisation altogether to mitigate this issue, but this would arguably remove one of Detox’s biggest benefits as an automation framework. While this would likely not prove an issue for less complex applications, heavier applications with real-time data or misbehaving SDKs may need to think carefully.

Appium and Test Flakiness

Appium borrows heavily from the Selenium way of thinking, treating the system under test as a completely isolated space. While this does lend a degree of separation, it can also mean that the test environment may be prone to flakiness due to the extra abstraction between the test, the test server, and the application. This is likely not an issue with simple test cases, but adding complexity would either result in tests prone to inconsistent failures or tests coded to have numerous safeguards against flakiness. A test suite can be written with stability as a core focus, but it isn’t quite as intuitive as Detox’s implementation.

Detox vs Appium — Ease of Use

Detox in the JS ecosystem

Detox features tight integration with JavaScript-based test runners, namely Jest and Mocha (though only Jest is supported out of the box as per version 20 of the library). For testers in the JavaScript ecosystem, it can integrate neatly within the package.json file and fits neatly into the same repository as the application under test.

…
 "jest": "^27.4.7",
  "jest-expo": "^44.0.1",
  "prettier": "^2.5.1",
  "typescript": "~4.3.5"
},
"jest": {
  "preset": "jest-expo"
},
"detox": {
  "test-runner": "jest",
  "configurations": {
    "ios.sim.e2e": {
      "binaryPath": "e2e/bin/ios/test-app.app",
      "build": "expo start --ios",
      "type": "ios.simulator",
      "name": "iPhone 13 Pro Max"
    }
  }
},
"private": true

It also follows a modern async-await JavaScript programming approach, which works well for those in the JavaScript ecosystem, though is less accessible for those moving from other languages. Writing tests is also relatively straightforward, with elements selectable by a number of attributes (including test IDs, text on the element, element types and traits) and with a number of common UI interactions already well-defined within the test tool. Finding the elements can be easily done with Xcode’s Accessibility Inspector for iOS, or Android Studio’s Layout Inspector, assuming some pre-existing experience with the tools.

Detox’s easy configuration means that communicating with a variety of simulated/emulated devices can be as easy as changing a configuration value.

Appium’s Architecture and WebDriver

Appium, as mentioned above, features bindings with multiple languages and can easily fit in with existing ecosystems. This is due to the Appium server existing independently of the library, meaning that all the library does is communicate with the server in the chosen language. This can complicate debugging; as one of the engineers on the assessment noted, “setup can take significant time if running into issues, mainly troubleshooting”. The similarity of the tool to Selenium, despite some of the shortcomings that entails (i.e. stability), means that the learning curve is relatively gradual for those with experience with WebDriver-based frameworks. Construction of test suites and execution is relatively quick, particularly when using some of the plugins, such as Appium Inspector (as an alternative to the platform-native tools). Like Selenium, it also plugs neatly into a number of different test runners across different languages, but with the understanding that many of these integrations are not built-in and require extra engineering effort.

Detox vs Appium — Performance

Detox as a Performant Framework

Detox lists performance as one of the major touchpoints for its framework. Its synchronisation and element matching is performant, with clever re-use resulting in UI test suites that can execute in a matter of minutes. Its integration with Jest means that parallel execution is supported, though some configuration is required. As the screenshot above shows, a typical test can be executed on the Countries SwiftUI sample in less than 20 seconds, from application start to test completion (though note this doesn’t include bootstrapping the simulator). For a problem domain typically plagued with slow performance, briefer testing times can help promote the value proposition of mobile automation quite well.

The Problem with Appium’s Performance

Appium, on the other hand, is somewhat hamstrung by its architecture. While integrating with XCUITest for iOS and Espresso for Android lends some benefits in terms of compatibility, it means that tests on those respective platforms can only ever be as fast as the original tools in terms of element matching, device bootstrapping and other test operations. Appium does technically support parallel testing, though with the important distinction that the different test executions must be on separate devices. A configuration with multiple iOS and Android devices plugged into a single device is supported (and can be leveraged on device farms) but lacks the simplicity of Detox’s approach. While optimisations in the framework have reduced the time, a simple text entry in the device’s built-in Chrome instance still took almost 30 seconds. The difference compared to Detox is not huge, but more complex tests and scaling up test suite complexity could easily grow this gap.

Detox vs Appium — Supportability

Detox — The Plucky Upstart

Detox is maintained by Wix, a website builder with a strong interest in React Native. It’s a relatively young framework, dating back to June 2016. Detox has also announced a gradual rewrite of their application, known as “Detox Next”, starting with removing an earlier dependency on EarlGrey and reworking configuration formats and progressing onto supporting XCUITest matching (which would help with the real iOS device and out-of-process limitations). Beyond the open-source nature of Detox itself, the tool itself is also designed for extensibility by third parties, with support for other test runners as well as for integrating web tests into the suite (using detox-puppeteer). However, Detox’s relative lack of community support compared to Appium may make it a riskier proposition for some projects; when interviewing experienced mobile engineers at Digio, they noted that Detox required a higher-than-usual investment to get running, and the leaner resources available for troubleshooting made integration tougher.

The amount of code contributed to Detox since 2016.

Appium — The Proven Performer

Appium is a far more mature framework, maintained by the JS Foundation but originally written by Dan Cuellar in 2011 as a C#-specific tool. It was then rewritten to incorporate a wider set of use cases. Its broader scope lends itself to wider use, as it can also support mobile browsers, Windows desktop applications, macOS applications and more, with its extensible driver model. Much as Detox has Detox Next as a future release, Appium has a beta version named Appium 2.0, which promises to reform Appium as a modern, extensible platform for UI automation. The current public release of Appium continues to see active development, and the architecture of even the current version of Appium, coupled with the wealth of compatible extensions, make it an attractive choice. The community focus on Appium is still high, with Detox sitting at 176k weekly downloads on npmjs.com at the time of writing, compared to Appium’s 324k, with this only accounting for installations in the Node ecosystem.

Which tool is right for me?

Our assessment started with the brief to identify one test automation tool in the mobile space that would fit the projects that Digio would tackle in the mobile space and to develop a body of knowledge around said tool. However, with the information discovered from the assessment and interviewing team members who have had hands-on experience, it becomes clear that no one good answer exists when asking, “which is the best mobile UI testing tool?”. Rather, the answer is specific to your team, preferences, and application.

Is Detox right for me?

Are you writing a React Native application or an app with fully native components? And do you want to leverage the power of the Node ecosystem to build a quicker, more stable test suite? Detox could be that framework that solves your problems.

Is Appium right for me?

Do you have a more complex cross-platform app with webviews or out-of-process interactions? Is your team already familiar with the Selenium way of doing things, or do they like to write their scripts in various languages? Appium could be the tool for you.

Is there anything else out there?

Naturally, there are many other modern and supported options; XCUITest is still a compelling option if you’re willing to put in the time to learn Swift (or Objective-C!), and Espresso’s support for both Kotlin and Java makes using it quite simple. These come with the caveat that you could be learning a new language just for automating in these languages, especially if you’re a QA new to the mobile space.

Last thoughts

Ultimately, while Detox and Appium provide us with enough power and flexibility to work with mobile applications, there’ll be engagements where we’d need to consider our options carefully. If all else fails, there’s still installing the app on a device and tapping away at the screen until a bug pops up.