How to use Gradle Managed Devices with your own devices
--
Google recently introduced a new feature for Android Gradle Plugin, Firebase Test Lab for Gradle Managed Devices. It uses Gradle Managed Devices API to launch tests not on the same machine where Gradle runs, but on a remote virtual or physical device inside Firebase Test Lab (paid feature). In this article, we’ll cover its functionality, how we can use our own device farm to launch tests remotely in the same manner as Firebase Test Lab does, and parallelize execution between multiple devices.
Gradle Managed Devices
Initially, Gradle Managed Devices was released to delegate a process of creation, launching, and closing emulators to Android Gradle plugin.
android {
testOptions {
managedDevices {
devices {
register("pixel2api30", com.android.build.api.dsl.ManagedVirtualDevice) {
device = "Pixel 2"
apiLevel = 30
systemImageSource = "aosp"
}
}
}
}
}By using the configuration above, we’re able to launch UI tests via pixel2api30Check task, and don’t need to connect a device or launch an emulator before running it. The environment of this test run is the same across different machines.
Firebase Test Lab
Firebase Test Lab for Gradle Managed Devices is a new feature, recently presented at Android Dev Summit 2022. This means that we can now run UI tests in Test Lab directly from Gradle, and don’t need to use command line tools or web UI.
plugins {
id 'com.google.firebase.testlab'
}
android {
testOptions {
managedDevices {
devices {
register("pixel2api30", com.google.firebase.testlab.gradle.ManagedDevice) {
device = "Pixel2"
apiLevel = 30
}
}
}
}
}
Is it also possible to do the same thing with our own devices? Let’s find out.
Custom Device implementation
As you can see, we’re using either com.android.build.api.dsl.ManagedVirtualDevice or com.google.firebase.testlab.gradle.ManagedDevice which both implement com.android.build.api.dsl.Device interface. So, what happens if we try to implement our custom one?
Let’s create a new module where we’ll add all required Gradle code, like plugins, etc. You can do that either by using buildSrc folder or creating a new included build. We won’t go into the process of creating a new Gradle plugin here, but you can find that in the official documentation or in my other article.
Declare MyDevice in the newly created Gradle plugin module and use it in an application module build.gradle.
interface MyDevice : Deviceandroid {
testOptions {
managedDevices {
devices {
register("myDevice", MyDevice) {}
}
}
}
}When we try to sync the project, we will face the following exception: Cannot create a MyDevice because this type is not known to this container. This means that android.testOptions.managedDevices.devices container does not know how to instantiate MyDevice because it’s an interface.
So how is this issue managed by Android Gradle Plugin? By searching for com.android.build.api.dsl.ManagedVirtualDevice we can find the following code:
dslServices.polymorphicDomainObjectContainer(Device::class.java).apply {
registerBinding(
com.android.build.api.dsl.ManagedVirtualDevice::class.java,
com.android.build.gradle.internal.dsl.ManagedVirtualDevice::class.java
)
}By using registerBinding the plugin is telling the container to use open internal class com.android.build.gradle.internal.dsl.ManagedVirtualDevice when API clients try to add anything of com.android.build.api.dsl.ManagedVirtualDevice type to the container. The container will create an instance of ManagedVirtualDevice class and provide it to the lambda of register method.
To do the same, we need an abstract class that implements MyDevice and a custom Gradle plugin and apply it to the project.
internal abstract class MyDeviceImpl(
private val name: String,
): MyDevice {
override fun getName(): String = name
}
class MyDevicePlugin : Plugin<Project> {
override fun apply(target: Project) {
target.plugins.withType(AndroidBasePlugin::class.java) {
target.extensions.configure(CommonExtension::class.java) {
it.testOptions.managedDevices.devices.registerBinding(
MyDevice::class.java,
MyDeviceImpl::class.java,
)
}
}
}
}plugins {
id `my-device-plugin`
}
android {
testOptions {
managedDevices {
devices {
register("myDevice", MyDevice) {}
}
}
}
}So, let’s try to sync again to see another exception:
Caused by: java.lang.IllegalStateException: Unsupported managed device type:
class com.bumble.devicefarm.plugin.device.farm.DeviceFarmImpl_Decorated
at com.android.build.gradle.internal.TaskManager.createTestDevicesForVariant(TaskManager.kt:1905)If we follow the stack trace, we’ll see that we need to add android.experimental.testOptions.managedDevices.customDevice=true to our gradle.properties and our MyDevice should implement ManagedDeviceTestRunnerFactory. So let’s investigate further.
internal abstract class MyDeviceImpl(
private val name: String,
) : MyDevice, ManagedDeviceTestRunnerFactory {
override fun getName(): String = name
override fun createTestRunner(
project: Project,
workerExecutor: WorkerExecutor,
useOrchestrator: Boolean,
enableEmulatorDisplay: Boolean
): ManagedDeviceTestRunner =
MyDeviceTestRunner()
}The factory itself has no significant value. The class it returns is more interesting — ManagedDeviceTestRunner.
interface ManagedDeviceTestRunner {
// returns true if and only if all test cases are passed. Otherwise, false
fun runTests(
managedDevice: Device,
runId: String,
outputDirectory: File,
coverageOutputDirectory: File,
additionalTestOutputDir: File?,
projectPath: String,
variantName: String,
testData: StaticTestData,
additionalInstallOptions: List<String>,
helperApks: Set<File>,
logger: Logger
): Boolean
}runTests method will be invoked for every Gradle module and contains a lot of data that we can use to run our tests. Here we can use testData to get APKs, install them and run our tests via instrumentation.
Instrumentation
We all know how to run tests via Android Studio by either clicking on the run button near a test name, or running connectedAndroidTest via Gradle. Now let’s explore how we can run them without these tools.
Both of these approaches use am instrument command on a device, which is run via ADB. You can read more in the official documentation.
adb shell am instrument -w <test_package_name>/<runner_class>To run tests, we should do the same from runTests method. To work with ADB, I’ll use dadb library from mobile.dev. It allows you to connect to a device via ADB protocol directly without invoking ADB executable. This really speeds things up and is convenient to use. You can read about it in their blog post.
Inside runTests method, let’s use Dadb to connect to a local emulator, install APKs and run a test.
override fun runTests(...): Boolean {
// Connect to a local emulator
Dadb.create("localhost", 5555).use { dadb ->
// Install application APKs
val apks = testData.testedApkFinder.invoke(DadbDeviceConfigProvider(dadb))
// Empty in case of library module
if (apks.isNotEmpty()) {
// Use multiple installation to support app bundles
dadb.installMultiple(apks)
}
// Install instrumentation APK
dadb.install(testData.testApk)
// Run tests
dadb.shell("am instrument -w ${testData.applicationId}/${testData.instrumentationRunner}")
}
return true
}runTests method and StaticTestData class have plenty of parameters. For simplicity, we’ll use only a small set of them — the required minimum to make everything work. We will use the following:
testData.testedApkFinderto get application APKs we need to install. In the case of a library module, it will return an empty list. It receivesDeviceConfigProviderto provide us with a proper list of APKs from App Bundle.testData.testApkis an instrumentation APK that contains code from androidTest folder.testData.applicationIdis an application id that we should run with an instrument command.testData.instrumentationRunneris a test runner likeandroidx.test.runner.AndroidJUnitRunnerthat we specify in ourbuild.gradlefile.
For now, I’ll skip implementation of custom DeviceConfigProvider because it simply invokes dadb.shell(“getptop name”).output for multiple properties like locale, screen density, language, region, and ABI. You can check the implementation details in the project repository.
Now, we’ve implemented the following structure:
Retrieving the results
runTests method should return true if all tests are passed, and false in any other cases, but now we always return true. To understand how to get instrumentation results, we need to dig a little deeper.
By default, if we try to run am instrument from a command line, we’ll see almost nothing except failures and the final result. To make it show more, it has two flags: -r and -m. The first one returns the results as a stream of text and the second as Protocol Buffers stream (supported only on API >= 26). We’ll just use the second one now for simplicity.
Android Gradle plugin uses RemoteAndroidTestRunner.StatusReporterMode.PROTO_STD enum to create an instance of IInstrumentationResultParser that can accept raw data from am instrument.
val mode = RemoteAndroidTestRunner.StatusReporterMode.PROTO_STD
val parser = mode.createInstrumentationResultParser(runId, emptyList())
Dadb.create(host, port).use { dadb ->
...
dadb
.openShell("am instrument -w ${mode.amInstrumentCommandArg} $arguments ${testData.applicationId}/${testData.instrumentationRunner}")
.use { stream ->
while (true) {
val packet: AdbShellPacket = stream.read()
if (packet is AdbShellPacket.Exit) break
parser.addOutput(packet.payload, 0, packet.payload.size)
}
parser.flush()
}
...
}This time, we’re using openShell method instead of shell method. It gives us access to a raw stream of data which is Protocol Buffers stream, and passing the data to IInstrumentationResultParser.
HTML & XML reports and implicit expectations
IInstrumentationResultParser will notify listeners about tests and their statuses. As you might notice, we passed emptyList() there. Let’s consider which parser we should use. There are a couple of ready-to-use implementations, but the answer is com.android.build.gradle.internal.testing.CustomTestRunListener which is used by Android Gradle plugin when it runs ManagedVirtualDevice.
val xmlWriterListener = CustomTestRunListener(
name,
projectPath,
variantName,
LoggerWrapper(logger),
)
xmlWriterListener.setReportDir(outputDirectory)
xmlWriterListener.setHostName("localhost:5555")
...
val parser = mode.createInstrumentationResultParser(runId, listOf(xmlWriterListener))
...
return !xmlWriterListener.runResult.hasFailedTests()CustomTestRunListener extends XmlTestRunListener and will write an XML report of tests that might be used by Android Gradle plugin, TeamCity or even you.
If we try to run myDeviceDebugAndroidTest without producing a report with CustomTestRunListener, com.android.build.gradle.internal.tasks.ManagedDeviceInstrumentationTestResultAggregationTask will fail with an exception. It expects that we’ll generate at least one XML report that starts with TEST-, and CustomTestRunListener enforces it. The task will generate combined XML and HTML reports, of which you can see a screenshot, here:
We can now also use CustomTestRunListener to return a proper value from runTests method, as we now have access to hasFailedTests method.
Remote execution
When using Dadb.create, we can pass not only localhost but any IP address. It means that we can use our code to run tests on a remote emulator or device. We can make our MyDevice configurable for this purpose.
interface MyDevice : Device {
@get:Input
val host: Property<String>
@get:Input
val port: Property<Int>
}
internal abstract class MyDeviceImpl(
private val name: String,
) : RemoteDevice, ManagedDeviceTestRunnerFactory {
init {
// Default parameters
host.convention("localhost")
port.convention(5555)
}
...
}register("remoteDevice", MyDevice) {
it.host = "192.168.3.4"
it.port = 43617
}And then use Dadb.create(managedDevice.host.get(), managedDevice.port.get()).
An attentive reader might correctly note that we can already do this without any code. It’s enough to call adb connect IP:PORT to make a remote device appear in adb devices list, and right inside Android Studio dropdown. We can even debug tests that we can’t do with Gradle Managed Devices. But, the fact that we can do these things is important for our next steps.
Parallel execution
By default, when we run tests, it only runs on one device at a time. It would be great to have the ability to run tests across multiple devices or emulators in parallel. The good news is, we found a way to do it.
AndroidJUnitRunner supports shard tests. It allows split tests into shards and runs one shard. For example, am instrument -w -e numShards 2 -e shardIndex 0 will run every first of every two tests and -e shardIndex 1 will run every second of every two tests.
But if you try to find a way to use it in practice, you’ll see something like this (source):
#!/usr/bin/env bash
./gradlew assembleAndroidTest
pids=
env ANDROID_SERIAL=emulator-5554 ./gradlew \
connectedAndroidTest \
-Pandroid.testInstrumentationRunnerArguments.numShards=2 \
-Pandroid.testInstrumentationRunnerArguments.shardIndex=0 \
-PtestReportsDir=build/testReports/shard0 \
-PtestResultsDir=build/testResults/shard0 \
&
pids+=" $!"
env ANDROID_SERIAL=emulator-5556 ./gradlew \
connectedAndroidTest \
-Pandroid.testInstrumentationRunnerArguments.numShards=2 \
-Pandroid.testInstrumentationRunnerArguments.shardIndex=1 \
-PtestReportsDir=build/testReports/shard1 \
-PtestResultsDir=build/testResults/shard1 \
&
pids+=" $!"
wait $pids || { echo "there were errors" >&2; exit 1; }
exit 0This is far from ‘convenient’, and means that we have to run two connectedAndroidTest tasks in parallel with lots of parameters and also manually merge XML results to be able to report them into TeamCity for example.
Gradle Managed Devices supports test sharding with android.experimental.androidTest.numManagedDeviceShards=<number_of_shards> as an option, but it only works with ManagedVirtualDevice. In our case, we want to use sharding with devices that we manage ourselves.
In Gradle Managed Devices, we’re working with Device abstraction. This abstraction can potentially implement many devices that are represented as a whole. Let’s introduce a new device type and register it as a device.
interface MultipleDevices : Device {
@get:Input
val devices: ListProperty<String>
}android {
testOptions {
managedDevices {
devices {
register("multipleDevices", com.example.MultipleDevices) {
it.devices.add("localhost:5555")
it.devices.add("localhost:5557")
}
}
}
}
}Everything related to ADB is extracted into AdbRunner class with a new parameter — ShardInfo(index, total). Parameters of ShardInfo will be simply added to dadb.openShell(“am instrument”) execution as is.
Now, we need to implement ManagedDeviceTestRunner so it runs tests in parallel.
override fun runTests(...): Boolean {
val devices = managedDevice.devices.get().map {
// Split "host:port" into Pair<String, Int>
val split = it.split(':')
split[0] to split[1].toInt()
}
val threadPool = Executors.newCachedThreadPool()
val futures = devices.mapIndexed { index, (host, port) ->
threadPool.submit(Callable {
val runner = AdbRunner(
host = host,
port = port,
shardInfo = AdbRunner.ShardInfo(
index = index,
total = devices.size,
),
)
val result = runner.run(
// Device name should be unique to produce a separate XML report for each device
name = "${managedDevice.name}-${host}-${port}",
runId = runId,
outputDirectory = outputDirectory,
projectPath = projectPath,
variantName = variantName,
testData = testData,
logger = logger,
)
result
})
}
val success = futures.all { it.get() }
threadPool.shutdown()
return success
}There are a couple of things worth paying attention to:
- I’m using
ThreadPoolwhich is a bad practice in Gradle,WorkerExecutorshould be used instead. You can get a usableWorkerExecutorinstance from parameters ofManagedDeviceTestRunnerFactory. The reason that I’m not currently using it is thatStaticTestDatais not serializable, and I don’t want to copy its properties to a separate serializable data holder to pass it toWorkerExecutor. nameshould be unique for every shard. We are passing name toCustomTestRunListenerwhich will generate an XML report with name in the filename. All XML reports will be collected later and merged together, so you can even see which test was executed on which device.
Sharding is very important and helps in the case of a lot of UI tests. In the sample repository I have prepared a library and an application module, both have 100 UI tests. It takes ~1m 10s to run tests in one module, so ~2m 33s for both modules. Running the same test suite on MultipleDevices will finish in ~1m 31s — almost half the time needed.
Emulators Test_time
----------- -----------
1 2m 33s
2 1m 31s
3 43s 
Parallel remote execution
Even with such a dramatic improvement, we’re not free from one problem: we need to run emulators locally, which consumes a lot of resources on Developers’ laptops.
Let’s solve this by creating a server that will host dozens of emulators and will have the following HTTP API:
GET /lease?devices=%number%
200 OK
[
{
host: "10.10.0.3",
port: 5555,
release_key: "ab34fd2d158f9"
},
...
]
POST /release
[ "ab34fd2d158f9", ... ]
200 OKThe server returns a pack of devices or emulators that we can ensure are only used by us, unless we release them with the corresponding API call. Let’s call it “Device Broker”.
Device Broker benefits not only Developers, but CI (Continuous Integration) too. The regular CI flow is building an app and running tests on emulators. Both these steps are very CPU- and memory-intensive tasks, which can be also done in parallel, leading to degraded performance. By outsourcing emulators to other servers, the build server can focus on building the app and verifying test results instead of sharing its resources with emulators.
The implementation on the Gradle side is pretty straightforward and looks similar to MultipleDevices.
interface DeviceFarm : Device {
@get:Input
val shards: Property<Int>
}
class DeviceBroker {
fun lease(amount: Int): Collection<Device> {
TODO("Make a network request to acquire devices, should wait if no available")
}
fun release(devices: Collection<Device>) {
TODO("Make a network request to release devices to make them available for others")
}
class Device(
val host: String,
val port: Int,
val releaseToken: String,
)
}
internal class DeviceFarmTestRunner : ManagedDeviceTestRunner {
override fun runTests(...): Boolean {
val broker = DeviceBroker()
val devices = broker.lease(managedDevice.shards.get())
devices.forEachIndexed { shardIndex, device ->
...
}
broker.release(devices)
return success
}
}
Results
We’ve investigated how to implement custom Device to integrate with Gradle Managed Devices. Device is an abstraction that can be backed by anything we want: a single device or emulator, hosted either locally or remotely in a web service like Firebase Test Lab, or our custom device farm with broker service. In the case of a single device, there are no real benefits because it can be connected directly to ABD via adb connect, without losing any functionality (such as the ability to attach a debugger). But, by implementing Device which is backed by multiple remotely hosted devices, we can release computational resources of Developers’ laptops and CI build servers. Also, by using the sharding feature, we can parallelize test execution and increase the speed test execution by 2X in the case of using 2 devices.
The code is available in the repository, but keep in mind that it’s a proof of concept and is not intended to be used in production. The code is a great starting point to adopting the approach in your organisation. Implementation details will heavily depend on your infrastructure. Finally, the code itself is not resilient to any errors and should be rewritten in a more reliable way.
If you have any questions, feel free to ask them in the comments section below.
