Android Test Orchestrator unmasked
Intro
Recently, we started using Android Test Orchestrator at Stepstone to help us with our instrumentation tests. In this article I would like to explain what it actually is, what issues it solves and how to use it. I’ll also uncover some of the magic that it uses under the hood to achieve what it does.
What is it and why use it?
Android Test Orchestrator (as explained in the doc referenced above) is a tool
which allows you to run each of your app’s tests within its own invocation of
Instrumentation.
This means that each test (method annotated with @Test ) will be run on a separate instance of AndroidJUnitRunner.
What issues does it solve?
When dealing with UI tests we identified 2 major problems which occur from time to time when run on CI or locally:
- Occasional crashes which stop the entire test suite.
- Test overlap.
Test crashes
Occasionally during UI tests our app would crash. Just so that there are no misunderstandings… By a crash I mean an unhandled error e.g. RuntimeException which would crash the entire app and not just an assertion which does not pass.
Sometimes it was a good thing as an error was detected. Sometimes it was due to some bug in the tests themselves (which was a bad thing which we had to fix). Regardless of the reason, the result would be that the rest of the tests in a test suite would not run.
If your test suite consists of a lot of tests and one of the first ones stops the entire process then you’re losing a lot of valuable information. Obviously, you know that something’s wrong as the app crashes (duh…), but there might be other undetected errors as well.
As Android Test Orchestrator runs each test in a separate instrumentation a crash would crash just that instrumentation and not affect the rest of the test suite. All tests would be executed.
Test overlap
Testing asynchronous calls is tough. The more asynchronous stuff you have in the app the harder it is to have stable UI tests. Sure, Espresso helps a lot with its idling resources, but sometimes it’s hard to account for every single asynchronous operation which might be happening in the app.
This in turn might sometimes lead to test overlap i.e. a situation where test A affects test B. This can be caused by a piece of code from test A that is still running even though test B already started. Imagine the following situation:
- some asynchronous operation starts in test A (e.g. a Service is started and does some IO operation),
- test A finishes with success, but the asynchronous operation it started does not end,
- test B starts,
- the asynchronous operation started in test A affects test B, e.g. it makes some database write operation which changes the app state.
The situation above could be avoided, however sometimes due to the complexity of solving the overlap it might take a lot of time.
Android Test Orchestrator starts a new instance of the app with each test and closes the one from the previous test. Therefore the overlap is solved automatically. The cleanup after each test is simplified.
NOTE: even though each test is executed in a separate process you still need to clean up after it e.g. clear shared preferences and database. This is actually something that Android Test Orchestrator might do in future releases, but for now you have to do it manually. See this thread for more info.
So how do I configure this thing?
When building with Gradle you need to complete these steps:
- Add the following statements to your project’s
build.gradlefile:
2. Run Android Test Orchestrator by executing the following command:
./gradlew connectedCheckUpdating build.gradle will also enable Android Test Orchestrator in Android Studio 3.0+.
If you’re using a different build system there’s also an instruction on how set this up from the command-line.
What happens when you run it?
As explained in the official documentation:
The Orchestrator service APK is stored in a process that’s separate from the test APK and the APK of the app under test, as shown in Figure 1:
Android Test Orchestrator collects JUnit tests at the beginning of your test suite run, but it then executes each test separately, in its own instance of
Instrumentation.
However, I’ve noticed that there are actually 4 APKs being installed (not 3 as shown in the diagram above):
- application under test
- test APK
- Orchestrator APK
- Test Service APK
E.g. when running our app from Android Studio we might see:
Test Services is a separate APK which Android Test Orchestrator uses to execute ADB shell commands which start AndroidJUnitRunner. You can find sources for both the Orchestrator and Test Services here (big thanks to Robert Zagórski for finding this).
Both Orchestrator APK and Test Services APK are being installed as result of adding:
androidTestUtil 'com.android.support.test:orchestrator:1.0.1'in our build.gradle thanks to Android Gradle Plugin 3.+ .
Now, let’s analyze the following statement:
Android Test Orchestrator collects JUnit tests at the beginning of your test suite run, but it then executes each test separately, in its own instance of
Instrumentation.
How does Android Test Orchestrator collect tests?
When you run your tests either with Android Studio or from the command line the following command is used:
The fun part here is that our instrumentation class is actually an instance of android.support.test.orchestrator.AndroidTestOrchestrator as explained here (sources for this runner can be found here). AndroidJUnitRunner (or whatever runner we’re using) is actually passed as an argument to this runner (targetInstrumentation key). So even if we have:
AndroidTestOrchestrator runner is used instead.
When Android Test Orchestrator runner is started it does some internal setup and then starts our AndroidJUnitRunner in a separate process. The first execution of AndroidJUnitRunner does not run any tests though. Instead, it is used to gather a list of tests to be executed, which is then passed back to the Android Test Orchestrator.
This is done in AndroidJUnitRunner’s onStart method like this:
where the tests are added with the help of OrchestratedInstrumentationListener like so (mOrchestratorListener in the Gist above):
Description class used in the Gist above contains the information about our tests.
So in the end AndroidJUnitRunner basically just looks for all tests annotated with @Test (or a subset of these tests if some filtering is applied).
What happens next?
Once the Orchestrator has a list of tests it starts them individually in new instrumentations. It is able to do so as these instrumentations receive individual test info under a class key which is added in the Bundle that gets passed to AndroidJUnitRunner’s onCreate(...) method.
In other words, for each test AndroidJUnitRunner’s onCreate(...) receives a different Bundle and this Bundle always contains a class key where the value of this mapped to this class key is the test name to run.
This is nicely presented on the diagram below:
To see more how the instrumentation arguments work you might want to check out this article:
I hope this sheds some light on how Android Test Orchestrator works 😃
Read more about the technologies we use or take an inside look at our organisation & processes. Interested in working at StepStone? Check out our careers page.
