Shard your Android Espresso tests for faster execution in parallel

Doug Stevenson
May 18 · 6 min read

Even after you’ve gone through great lengths to optimize the performance of your tests, and the hardware and software they run on, you still might be slowed down by the sheer volume of tests that you have to run. At this point, to get more significant performance gains, you’ll have to shard your tests to run on multiple devices concurrently. By “shard”, I mean splitting up the tests into groups of roughly equivalent size, to run on each device independently. This is probably the most difficult optimization to perform, but it’s definitely worthwhile if you have the hardware to support it.

By default, when the Android instrumentation test runner runs your test suite, it goes through each test one at a time, launching your app for each test, then tearing it down. If you’re using Test Orchestrator, that setup and teardown overhead can increase, depending on how far you want it to go to clean up the side-effects of each test.

To improve on this, the Android Grade build tools make it possible to run your tests in parallel on multiple virtual or physical devices attached to your machine. This is called “sharding”.

The test runner provides a way for you shard your tests using command line flags. Most of the time, you don’t deal with the test runner CLI directly (Android tools do that for you). But for the purpose of sharding, you will need to understand a bit about how the CLI works and the flags it accepts, so that you can configure things correctly. In this article, I’ll describe the arguments you can pass to the Gradle command line for various configuration options that get passed along to test runner.

First, how many devices can you run?

Since an Android device can only run one test at a time, you’ll need to have multiple devices to run sharded tests in parallel. If you have the physical devices, of course you should attach as many as possible. But running multiple emulators will actually work fine, if your machine has the resources to do so. (You can even run emulators in tandem with devices.) My MacBook Pro happens to have eight cores and 32 GB of memory, which means I could probably run at least 4 emulators on it without too much trouble, each churning through its own shard of tests. You might have to figure out (by experimentation) what your machine can handle. I assume that UI tests will make use of at least 2 cores. Bear in mind that the modern default Android emulator configuration emulates 4 cores — you might want to adjust accordingly if your app is very busy with background work.

In this post, for simplicity, I’ll just assume that the host machine can handle two emulator instances. If you have an emulator configured that you already like to use for testing, it’s easy to duplicate that configuration in Android Studio to create a second identical instance.

When you attach multiple Android devices or run multiple emulators to a host machine, adb should locate all of them. So, if you have two emulators, running adb devices should output like this:

You will need to know the unique names of all the devices attached to your machine, so you can shard your tests among them. What these names, you can run Gradle commands to kick off each shard. I’ll just get straight to a Unix shell script that does exactly that (if you run Windows, you’ll have to craft your own solution):

./gradlew assembleAndroidTestpids=env ANDROID_SERIAL=emulator-5554 ./gradlew \
connectedAndroidTest \
-Pandroid.testInstrumentationRunnerArguments.numShards=2 \
-Pandroid.testInstrumentationRunnerArguments.shardIndex=0 \
-PtestReportsDir=build/testReports/shard0 \
-PtestResultsDir=build/testResults/shard0 \
&
pids+=" $!"
env ANDROID_SERIAL=emulator-5556 ./gradlew \
connectedAndroidTest \
-Pandroid.testInstrumentationRunnerArguments.numShards=2 \
-Pandroid.testInstrumentationRunnerArguments.shardIndex=1 \
-PtestReportsDir=build/testReports/shard1 \
-PtestResultsDir=build/testResults/shard1 \
&
pids+=" $!"
wait $pids || { echo "there were errors" >&2; exit 1; }
exit 0

This script builds the app and its tests, runs the first shard on device “emulator-5554” in the background, then runs the second shard on device “emulator-5556” in the background. The script then waits for them to both finish, and checks the return code for errors.

Let’s take a look at some of the arguments to Gradle:

These two parameters are passing along the names and values of additional test runner command line arguments. The numShards and shardIndex parameters indicate to the Android test runner how many shards are being used overall, and which shard to use for this run. So, if you indicate numShards=2, then you should run two sets of tests, one with shardIndex=0 and the other with shardIndex=1.

The testReportsDir and testResultsDir properties are custom properties. They don’t have a specific meaning to Gradle or the Android test runner. But I use their values in build.gradle in order to tell the Android Gradle plugin where to put the test reports and results on a per-shard basis (the “0” here matches the provided shardIndex). I’m told by the AGP team that there is no way to set these file locations directly. So, in build.gradle, you’ll need to pass these property values along to the plugin like this:

Basically, what we’re doing here is overriding the default report and result locations on the command line, if present.

This strategy lets you scale up your tests on hardware that you have sitting in front of you. But for even more massive scalability, you’ll want to shard your tests across devices in the cloud. Turns out, there are cloud-hosted options for mobile app testing.

Firebase Test Lab scales for many target devices

Firebase Test Lab is Google’s answer for scalable mobile app testing. You can run your Android Espresso tests (and iOS XCTests) on both physical and virtual devices managed by Google. All you do is choose which devices you want to test with, provide the app and test APKs, and off it goes, running tests in parallel among each chosen device (as they become available from the shared pool). However, it still runs the entire test suite sequentially on each chosen device — it doesn’t shard your tests across devices.

If you want actual test sharding distributed across Test Lab devices, you can use another tool called Flank, created by Walmart Labs, to shard your tests across devices managed by FTL. I won’t describe how to use FTL and Flank here (the documentation is already pretty good). But I will encourage you to consider paying for FTL if you have a very large number of tests that need to be run frequently, and especially if you want broad device coverage.

Got lots of tests? Start sharding!

Sharding your tests can be a great way to boost the speed of your tests, using either your local hardware, or hardware in the cloud. It typically only makes sense for very large numbers of tests, as there is some overhead to get started with sharding, but that’s time you’ll get back in your future development cycles as you’re able to test more frequently and respond to new issues faster.

More articles on faster testing on Android

Mesmer

Mesmer, the leader in Robotics Process Automation for…

Doug Stevenson

Written by

Director of Developer Relations at Mesmer (mesmerhq.com)

Mesmer

Mesmer

Mesmer, the leader in Robotics Process Automation for Development (RPAD), is radically changing the way developers work. Mesmer’s AI-powered bots use patent-pending Deep Learning Automation (DLA™) to accelerate every function of customer experience testing. https://mesmerhq.com

Doug Stevenson

Written by

Director of Developer Relations at Mesmer (mesmerhq.com)

Mesmer

Mesmer

Mesmer, the leader in Robotics Process Automation for Development (RPAD), is radically changing the way developers work. Mesmer’s AI-powered bots use patent-pending Deep Learning Automation (DLA™) to accelerate every function of customer experience testing. https://mesmerhq.com

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store