Detox — Gray Box Testing
Gray Box End-to-End Testing and Automation Framework for Mobile Apps
Detox does Gray box to allows the test framework to monitor the app from the inside and actually synchronize with it.
What is Gray Box Testing?
Gray box testing is the testing of a software application using an effective combination of both White box testing & Black box testing method. This is a nice & powerful idea to test the application.
The leading native gray box drivers are developed by Google — EarlGrey for iOS and Espresso for Android
EarlGrey is a native iOS UI automation test framework that enables you to write clear, concise tests.
Espresso — A key benefit of using Espresso is that it provides automatic synchronization of test actions with the UI of the app you are testing. Espresso detects when the main thread is idle, so it is able to run your test commands at the appropriate time, improving the reliability of your tests. This capability also relieves you from having to add any timing workarounds, such as Thread.sleep()
in your test code
Detox does not rely on WebDriver, since this is not the web. Detox communicates with its native driver (which extends EarlGrey and Espresso) using a JSON-based reflection mechanism, this allows a common JavaScript implementation to invoke native methods directly on the device.
Espresso idling resources
An idling resource represents an asynchronous operation whose results affect subsequent operations in a UI test. By registering idling resources with Espresso, you can validate these asynchronous operations more reliably when testing your app.
Architecture
The sequence diagram below shows the general communication scheme between the components in Detox.
Design Principles
Detox does not rely on WebDriver
Detox is built from the ground up to integrate with native layers of your mobile app directly. We try to avoid generic cross-platform interfaces that are often the lowest common denominator. We want to optimize per platform.
Detox does gray box, not black box
Theoretically, it sounds better to test exactly what you ship as a black box. In practice, switching to gray box allows the test framework to monitor the app from the inside and delivers critical wins like fighting flakiness at the core.
Detox relies on EarlGrey and Espresso
The leading native gray box drivers are developed by Google — EarlGrey for iOS and Espresso for Android. Detox relies on them using a JSON-based reflection mechanism which allows a common JavaScript implementation to invoke their native methods directly.
Friendly Protractor-like API for tests
Tests in Detox are implemented in human-readable JavaScript and can even be shared between platforms. This easy to use API completely abstracts the complex native driver invocations taking place under the hood.
Detox controls devices through low-level APIs
Let’s take iOS simulators for example, which are difficult to control efficiently since multiple concurrent instances aren’t supported. Detox uses AppleSimulatorUtils (another opensource library by Wix) to work around these issues and support test shading.
Built from the ground up for mobile and React Native
Detox is inspired by web testing methodologies but is not a direct translation of a solution designed for a different platform. Detox is built from the ground up for native mobile and has deep first-class support for React Native apps.
Detox relies on websockets for communication
Communication between the test script running on Node.js and the tested app running on device uses websockets. This provides true bi-directional communication and is much faster and resilient than REST-like protocols.
Both tester and testee are websocket clients
The test script running on Node.js and the tested app running on device are both clients. This allows one side to disconnect without affecting the other. A separate proxy websocket server is used to connect them.
Expectations run on the testee, not the tester
Traditionally, test frameworks evalutate expectations in the test script running on Node.js. Detox evaluates expectations natively directly in the tested app running on device. This enables operations that were impossible before due to performance reasons.
Test Life Cycle
Detox is test runner independent, we can use mocha/jest/chai etc.
Add Detox to React Native Project
Install Detox command line tools (detox-cli)
npm install -g detox-cli
Install Detox
In React Native project, go to its root folder (where package.json
is found) and type the following command.
npm install detox --save-dev
Install a test runner
Use any JavaScript test runner. Detox CLI supports Jest and Mocha out of the box
npm install mocha --save-dev
Add Detox config to package.json
"detox": {
"test-runner": "mocha",
"specs": "e2e",
"runner-config": "e2e/mocha.opts",
"configurations": {
"android.emu.debug": {
"binaryPath": "android/app/build/outputs/apk/debug/app-debug.apk",
"build": "pushd android && gradlew assembleDebug assembleAndroidTest -DtestBuildType=debug && popd",
"type": "android.attached",
"name": "yourdeviceID"
},
"android.emu.release": {
"binaryPath": "android/app/build/outputs/apk/release/app-release.apk",
"build": "pushd android && gradlew assembleRelease assembleAndroidTest -DtestBuildType=release && popd",
"type": "android.attached",
"name": "yourdeviceID"
}
}
}
android.attached for Android Device and android.emulator for Android Emulator
Create your first test (I have used mocha)
detox init -r mocha
- Create an
e2e
folder in your project root - Inside
e2e
folder createmocha.opts
(formocha
) - Inside
e2e
folder createinit.js
file - Inside
e2e
folder createapp.spec.js
Add Detox dependency to an Android project
In android/settings.gradle
add:
include ':detox'
project(':detox').projectDir = new File(rootProject.projectDir, '../node_modules/detox/android/detox')
In android/app/build.gradle
add this to defaultConfig
section:
defaultConfig {
...
testBuildType System.getProperty('testBuildType', 'debug') //this will later be used to control the test apk build type
missingDimensionStrategy "minReactNative", "minReactNative46" //read note
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
...
}
Please be aware that the minSdkVersion
needs to be at least 18.
In android/app/build.gradle
add this in dependencies
section:
dependencies {
...
androidTestImplementation(project(path: ":detox"))
androidTestImplementation 'junit:junit:4.12'
androidTestImplementation 'com.android.support.test:runner:1.0.1'
androidTestImplementation 'com.android.support.test:rules:1.0.1'
...
}
And in android/build.gradle
you need to add this under allprojects > repositories
:
buildscript {
repositories {
...
google()
...
}
}
Create Android Test class
You need to add the file android/app/src/androidTest/java/com/[your.package]/DetoxTest.java
Build your app and run Detox tests
Build your app
detox build
If you have more than one configuration(package.json) we need specify the configuration as below
detox build --configuration android.emu.debug
Run the tests
detox test
If you have more than one configuration(package.json) we need specify the configuration as below
detox test --configuration android.emu.debug
Troubleshooting for Android Devices
react: 16.6.1
react-native: 0.57.5
mocha: 5.2.0
detox: 9.1.2
a) I have used Android Mobile (Redmi Note 4) version 7.0 to test, while executing the detox test, got an error in application installation.
detox[69260] ERROR: [exec.js/EXEC_FAIL, #8] failed with code = 1, stdout and stderr:
The above error is due to installing the APK with -g (grant all runtime permissions), fixed by updating the below code in node_modules\detox\src\devices\android\ADB.js
let childProcess;
if (apiLvl >= 24) {
childProcess = await this.adbCmd(deviceId, `install -r -g ${apkPath}`);
} else {
childProcess = await this.adbCmd(deviceId, `install -rg ${apkPath}`);
}
to
let childProcess;
if (apiLvl >= 24) {
childProcess = await this.adbCmd(deviceId, `install -r ${apkPath}`);
} else {
childProcess = await this.adbCmd(deviceId, `install -rg ${apkPath}`);
}
b) Security Exception for INJECT_EVENTS permission
Reference:
[Android]:https://developer.android.com/training/testing/espresso/idling-resource#java
[Detox]:https://github.com/wix/Detox
[Detox]:https://github.com/wix/Detox/blob/master/docs/Introduction.Android.md
[Detox]:https://github.com/wix/detox/blob/master/docs/APIRef.Configuration.md#test-runner-configuration
[Blog]: https://hackernoon.com/detox-gray-box-end-to-end-testing-framework-for-mobile-apps-196ccd9564ce
[Youtube]: https://youtu.be/GgFFeI70PWw
Full example project in GitHub https://github.com/selvakumardreams/DetoxExample