Detox vs. Appium: automated UI tests in React Native

Developer’s edition

Jurijs Yuri Bormanis

Published in

Reactive Hub

7 min readDec 29, 2018

All new mobile environment

Probably just like you I came to React Native from different JavaScript background rather than as a native mobile developer. This is completely new environment with its bits and tricks to learn.

One of the important new fields you have to explore is testing. When it’s more or less clear in unit tests, how do we handle UI tests, end-to-end? There’s iOS, there’s Android. Market with lots of devices to choose from.

Despite its relatively new tech compared to native mobile, it’s still a mobile environment and many things we take and learn from native side.

Here are two frameworks I considered to use to make my life easier as React Native developer.

Appium

Using Selenium WebDriver behind the scenes Appium is a powerful testing framework with the large native mobile community behind it. Released even before React.js this is the number one in the industry.

Start is easy just by installing “appium” and “appium-doctor” packages globally or for your project using npm. Running “appium-doctor” command will tell you what you need to add and install on your system before writing and running the tests and if possible can help you fix the issues. When it’s all sorted, packages installed and jest configuration is in place, you can start Appium server and then run your tests.

I’m not going to detail too much test setup and writing process but this is how basic test looks like (with comments added):

/* selenium webdriver client for node
*/
import wd from 'wd'/* default timeout
*/
jasmine.DEFAULT_TIMEOUT_INTERVAL = 60000/* localhost because we run Appium server from our machine + port
*/
const URL = 'localhost'
const PORT = 4723/* creates webdriver object
*/
const driver = wd.promiseChainRemote(URL, PORT)/* Server capabilities.
* This is how you tell Appium Server,
* how to run the test, in other words config.
* You probably will want to move this out of here.
*/
const capabilities = {
  platformName: 'iOS', // or Android
  platformVersion: '12.1', // version of the OS
  deviceName: 'iPhone 8', // or Android Emulator or specific device
  automationName: 'XCUITest', // platform specific framework (UIAutomator2 for Android)
  app: '/path/to/.app' // path to app (in case of Android it's .apk)
}beforeAll(async () => {
  try { // before running our test
    await driver.init(capabilities) // initialise webdriver
    await driver.sleep(4000) // yes, we need to wait for launch screen finish his work before running test, hint of flaky stuff!
  } catch(err) {
    console.log(err) // just in case something goes wrong
  }
})afterAll(async () => {
  try {
    await driver.quit() // end testing session after test
  } catch(err) {
    console.error(err)
  }
});/* Jest, now do whatever you want!
* in this case we check if elements with testIDs 
* 'topLabel' and 'subLabel' contain the text we want
* Check Appium documentation
*/
describe("Home Screen landing", () => {
  test("render search screen", async () => {
    let topLabel = await driver.elementById('topLabel')
    let subLabel = await driver.elementById('subLabel')
    expect(await topLabel.text()).toBe("OK")
    expect(await subLabel.text()).toBe("Home Screen")
  })
})

The actual test part is the last few Jest lines that check if there’s text “OK” and “Home Screen” on the screen. As you can see it looks like typical Jest test so there are no problems to write it. Appium documentation page gives you overview of capabilities and examples.

The line to dislike is line `await driver.sleep(4000)`. We need it because tests doesn’t know what the hell is actually happening in the app. Black box. Imagine if would you write some Node code with http call and set timeout before it instead of using promise or callback. Flakiness.

In this basic setup we tell Appium (webdriver) to wait for 4 seconds before start the test since we need to launch the app and go past “LaunchScreen”, but as app becomes more complex, you will need to use it more often — http calls or animations, react-native itself — the bridge between JavaScript and native code will cause a delays.

There are alternatives such as Implicit Wait but it doesn’t change the Black Box model.

Blackbox testing, we have built of our app with no access to internal structure.

What I Like about Appium

7+ years in industry.
API capabilities, thumbs up.
Easy to find help & support online (also dislike, see below)
Various languages supported to write your tests including JavaScript.
Usual for JS developer environment for writing tests using Jest.
Used for end-to-end tests on MS AppCenter, BrowserStack and AWS Device Farm.
Tests with real devices.

What I Dislike about Appium

Searching for assistance online gives back results in all different languages, most of it in Java.
Black box testing (tests doesn’t know about internal process)
Works out of sync with the app. Flaky, even more flaky on react native.
testID prop ignored in Android case.

Notice three tabs in Terminal. Appium server logs, metro bundler and tests itself.

Detox

Wix’s Detox works similar way as Appium but important difference here is a Gray box testing. One of the reason why Detox was born is to solve “flakiness” problem — it will wait until action in app is finished and will continue only when app is idle. This is possible because of other framework called EarlGrey.

Same as Appium, we set configuration first.

/* Our Device config is in package file, see below
*/
const detox = require("detox");
const config = require("./package.json").detox;/* Jest adapter
*/
const adapter = require("detox/runners/jest/adapter");/* Set the default timeout
 * and use Jest adapter
 */
jest.setTimeout(120000);
jasmine.getEnv().addReporter(adapter);beforeAll(async () => {
  /* Initialise and start the server
  */
  await detox.init(config);
});/* beforeEach and afterEach detox test
 * we run beforeEach of the Jest and
 * do cleanup
 */
beforeEach(async function() {
  await adapter.beforeEach();
});afterAll(async () => {
  await adapter.afterAll();
  await detox.cleanup();
});

And additional configuration in package file.

"detox": {
  "configurations": {
    "ios.detox": { // configuration for iOS (run by detox test -c ios.detox)
      "binaryPath": "path/to/.app",
      "build": "xcodebuild -workspace ios/app.xcworkspace -scheme scheme -configuration Debug -sdk iphonesimulator -derivedDataPath ios/build", // workspace or project file. In this case we’re building debug package rather than production (release).
      "type": "ios.simulator",
      "name": "iPhone 8"
    },
    "android.detox": { // configuration for Android (run by detox test -c android.detox)
      "binaryPath": "path/to/.apk",
      "build": "cd android && ./gradlew assembleDebug assembleAndroidTest -DtestBuildType=debug && cd ..", // In this case we’re building debug package rather than production.
      "type": "android.emulator",
      "name": "Pixel_2_API_28" // name of the simulator. “adb devices” command will list available attached devices
    }
  },
  "test-runner": "jest",
  "runner-config": {
    "setupTestFrameworkScriptFile" : "./detox.init.js", // our configuration above
    "testEnvironment": "node",
    "testRegex": "(.*|\\.(ui))\\.(ts|tsx)$" // test regex for UI tests
  }
  "specs": "./__tests__/" // location of UI tests
}

Test are written the same way, instead of using Appium API, we use Detox capabilities and limitations. Check links in the end for detailed documentation on setup and reference.

Gray-box testing. Our application and access to internal structure.

What I Like about Detox

Made by Wix for React Native.
JavaScript focused.
Gray box testing (there’s connection between tests and internal processes).
Works in sync with the app. Not so flaky.

What I Dislike about Detox

Capabilities are not wide as Appium
Smaller community.

Flaky, flakey, flakiness

Despite Detox using Gray box tests, it’s still flaky. Test I wrote with text input and swipe gesture for some unknown reason 1 of 10 times failed. You can’t be 100% sure in UI tests.

Speed

Appium will be slower because of manual “sleep” commands, Detox in this case wins as it automatically continues action as soon as one is finished. Overall I would not make any conclusions yet until I run large amount of tests. In small 30 second tests and basic setup for this article Detox performed seconds faster. In terms of iOS vs Android, I tried several tests on both platforms on both frameworks it took +- same amount of time to run the tests. Just be ready that it takes much more time than usual unit tests.

What to choose

Myself I’m still exploring both of these frameworks and it will take time to fully understand both, but for now as a JavaScript developer I pick Detox.

Try both. Luckily it’s only two of them. It really depends on the app you are developing and structure of you team.

UI tests that will be used only in the team of developers, I would suggest Detox. In long and complicated end-to-end tests, because of wider API capabilities as well as support by automated platforms like BrowserStack, AWS DeviceFarm and MS AppCenter I would stick with Appium. It’s for now.