At Doctolib, we have more than 25,000 automated tests of which a large part are end-to-end ones. Our mobile applications are mainly a webview but they are developed with React Native. However, the patient application also has some native screens for the video consultation and some features that require native code.
Since the redesign of mobile applications with React Native in February 2018, we have added Jest unit tests for some utilities, but nothing to actually test native features without manually testing the applications. We strongly believe in the benefits of end-to-end testing at Doctolib, so there has always been a desire to have end-to-end tests on mobile applications, but technically it is not as simple and well-equipped as it is for the Web.
Which testing framework?
A first attempt was made in April 2019 with the Appium but it was a failure. It was too complicated and too limited to set up locally as on our mobile Continuous Integration (CI). So we gave up as soon as the Proof-of-Concept was done.
A year went by and mobile end-to-end testing became a priority for the patient application. Detox seemed like a good candidate and was gaining popularity. It looked easy to write tests and to configure and implement it on a CI pipeline. Therefore I did some quick testing and was rapidly convinced. In a very short time I was able to write my first tests and have them working on the iOS simulator.
Another cool thing about Detox is we can easily launch our application from a push notification or configured with specific permissions (camera, location, etc.). This is a very useful feature in Doctolib cases, for example our telehealth diagnostic screen has different behavior according to camera or microphone permissions.
How to test the application?
The application is mainly a webview but it interacts with native features through events sent with Window.postMessage. As it is impossible to interact with its content, we created a React Native test screen to test the native features directly. This screen only exists in the staging and development builds.
This basic screen is directly available by a button from the home screen of the application. Each button on the test screen triggers a native feature. These are used by Detox to run automated tests but it is also a convenient way for developers and product managers to do manual tests on real devices. Let’s see some examples.
Testing native screens and OS permissions
The video consultation on the mobile application is totally implemented with native code. So with Detox, it is easy to navigate through different screens and check the behavior with different permissions given to the application by the OS. As you can see in the following video, we navigate through a flow of diagnostic screens to check whether your phone is capable of doing a video consultation. This flow is tested with different permissions for camera and microphone.
Let’s look at the code of the first test executed in the video. It is pretty simple to write and to understand. It is just a sequence of unit commands to launch the application with specific permissions, to click on buttons and to assert whether certain text is visible or not.
Detox is a cross-platform testing framework, so tests are executed on Android and iOS with the same code. But sometimes Detox does not have the same support on both platforms, or Android emulator and iOS simulator do not have the same capabilities. In that case you have to adapt your test with a condition on the OS.
The first test shows the difference between the capabilities of the simulators. The iOS Simulator does not have a battery state, so we check a message for low battery. On the Android emulator, we have the information so we just check that the message is not present.
The second test is not executed on Android due to a limitation of Detox on this platform. For the moment, it is not possible to change the battery permission on Android.
Testing native features called from the webview
Let’s look at the code of some tests executed in the video. Like the previous test code, it taps on buttons and checks if elements are present by testing their text or their id.
Here again we have some OS differences, so tests contain OS-specific code. Adding a calendar event on Android will open the Calendar application that is why we go back and check if we are returning to the test screen. On iOS it shows a modal over the screen, so we check if the test screen is not visible anymore. This is the only thing we can do because Detox can only pilot your application, not anything outside of it.
If you look at the last test, you will see we can assert text and tapping on buttons of the iOS sharing modal. This is a little hacky because we use the private UI class but it is possible.
Continuous Integration with Bitrise
As you can see in the previous videos, Detox tests are launched by simple shell commands. As a consequence, it was pretty easy to plug it to Bitrise, our mobile CI. For each commit in a pull request or on master and production branches, tests are launched and applications are built. Detox tests are executed for Android and iOS patient apps. They are very stable and fast on the iOS Simulator as shown by the following screenshot.
On the other hand, Detox tests are flaky on Android build due to the Bitrise Android emulator. They are stable when we run them on our respective machines but on Bitrise the emulator is not performing and reactive as well. Despite some adjustments on Bitrise, it happens regularly that a test fails because the emulator does not react to the tests.
This summer, I made an attempt with a Genymotion Cloud emulator instead of the Bitrise one. It was stable but some tests failed because of the physical back button used in some tests which were not understood by Genymotion. Genymotion confirmed to me very recently that there was no problem with the latest version of Detox. So we’ll have to follow up.
These Detox tests are really a first step for Doctolib and many things can be improved. As I mentioned before, the Android emulator of Bitrise is not stable so the test suite is flaky. It might be possible to use Genymotion cloud to fix this. We also can imagine executing all these end-to-end tests on real devices.
For the moment, we only have Detox tests on the patient mobile application. The doctor application has some features which use native code and it will continue to grow, so having Detox tests for this app should improve non-regression and simplify upgrades.
And you, what is your experience about end-to-end tests on mobile applications?