Stable functional tests with Genymotion Gradle Plugin and test-specific Product Flavors

We heart Genymotion (and its Gradle Plugin)

It’s probably safe to say that Genymobile’s Genymotion Android Emulator has become the emulator of choice of most developers out there. At least every developer that I know swears by it, and rightfully so as nothing out there including Google’s current, pre Android Studio 2.0 offering, delivers such simplicity, speed, depth of configuration, and stability. It’s also free for personal use in its basic but generous feature set. What’s not to love.

When a product increases my productivity by such degree that it becomes indispensible, I eagerly await every next update, and no other update got me as excited as version 2.5, released in June 2015, due to a brand new addition to its paid-for tier arsenal — the Gradle Plugin for Genymotion. The plugin lets you specify Android device definitions in your build.gradle file to a fine configuration detail, which will be started when connectedAndroidTest task is detected. The configuration is a breeze and can boil down to as little as this:

apply plugin: "genymotion"
genymotion {
devices {
nexus5 {
template "Google Nexus 5 - 4.4.4 - API 19 - 1080x1920"
}
}
}

The benefits of harnessing the potential power of this solution in a CI system are clear to see. Given the vast array of device definitions Genymotion offers out of the box, with Android APIs from version 10, you could have your code validated on all of them, or on a given subset, which you consider sufficient as part of your nightly CI runs. Furthermore, you could have smaller subsets for faster test completion to validate your team’s PRs, for every push to repo, etc.

genymotion {
devices {
nexus5 {
template "Google Nexus 5 - 4.4.4 - API 19 - 1080x1920"
}
nexus4 {
template "Google Nexus 4 - 4.4.4 - API 19 - 768x1280"
}
nexus 7 {
template "Google Nexus 7 - 4.2.2 - API 17 - 800x1280"
density "ldpi"
deleteWhenFinish false
}
}
}

The great news is, that the plugin does indeed offer that. On assigned Gradle launchTask , Genymotion starts the chosen devices, resumes the task letting the tests to run on the devices, and when they finish closes the devices allowing the task complete. There’s a caveat however — all those devices will be launched at the same time! Now, this could be manageable for the one, two, three devices that you’re happy to validate your code against, as they are launched simultaneously. Such number however can not cover a complete list of devices you want to test your app against. For example, if your app’s minSdkVersion is 16, and you’d like to cover popular phones and tablets that Genymotion comes pre-defined with, that could easily produce about 20 device definitions, and you probably don’t want your devs’ or CI machines to launch this many VMs for every test. Your machines wouldn’t cope, and if you’re making requests to your API’s in your tests, they might have problems as well, given the sheer speed and number of same requests sent to them. What you need here is the option to launch these devices in smaller batches, until the device list is exhausted, which would come useful in nightly CI runs. Having these sets of devices, would also open doors to being able to launch just that chosen subset at a time, without running the rest — pretty useful for your devs’ or CI sanity checks on PRs. So, does the plugin offer such ability out of the box? Sadly no, but with a few tweaks to your Gradle build file, you are able to achieve such solution.

But Product Flavors need our attention too

Gradle Product Flavors. Do you use them? It is possible that you don’t, because beyond the required parts of the buildVariant equation — buildTypes — you might have skipped the productFlavors part and be perfectly content with your dev/release Build Variant combo. The commonly found explanations of Product Flavors involve use cases where an app is offered in paid/free tiers. But what if you have never had a use for this in your live app? Well, it so happens that even that you might not, they are crucial to unlocking the device subset potential of your Genymotion Gradle Plugin and leverage this mechanism to divide the app into versions dedicated to running tests.

This approach is nicely explained in this recent post on Google Developers and even though this video by guys at Genymotion is where this solution has dawned on me, it would be great if it made its way as well to the source code examples as well… which I’m told that it will :-) In the meantime, this is how productFlavors can help you create as many device definitions as you need, and run them in batches, and in a stable fashion, to help you cover as exhaustive set of configurations as your testing needs require.

Configuring the Plugin for Dev, CI and batched test runs

As an example, say that you want to test your app on Nexus 4, 5 and 7 devices from API 18, where applicable. Your CI should run the tests on all configurations of those devices between APIs 18–21 as part of its nightly runs, which amounts to 8 device configurations. Because you don’t want your CI to launch all of them at the same time, you’d for example want to launch all per API level at a time. For your dev smoke tests, you’ll settle for just API 21 of those devices. The configuration of these is simple:

  1. Specify productFlavors in the android section of your build.gradle
android {
productFlavors {
dev; // dev / smoke tests
api18; api19; api21; // selected api levels tests
}
}

2. Configure your Genymotion devices and apply productFlavors properties

genymotion {
devices {
        // API 18
"Google Nexus 4 - 4.3 - API 18 - 768x1280" {
productFlavors "api18"
}
"Google Nexus 7 - 4.3 - API 18 - 800x1280" {
productFlavors "api18"
}
        // API 19
"Google Nexus 4 - 4.4.4 - API 19 - 768x1280" {
productFlavors "api19"
}
"Google Nexus 7 - 4.4.4 - API 19 - 800x1280" {
productFlavors "api19"
}
"Google Nexus 5 - 4.4.4 - API 19 - 1080x1920" {
productFlavors "api19"
}
        // API 21
"Google Nexus 4 - 5.0.0 - API 21 - 768x1280" {
productFlavors "api21", "dev"
}
"Google Nexus 7 - 5.0.0 - API 21 - 800x1280" {
productFlavors "api21", "dev"
}
"Google Nexus 5 - 5.0.0 - API 21 - 1080x1920" {
productFlavors "api21", "dev"
}
    }.each {
it.template String.valueOf(it) // which template to use
it.deleteWhenFinish false // keep the device after test
        // any other common device config items
}
}

3. The above configuration will create the following tasks:

connectedApi18DebugAndroidTest // launches two api18 devices
connectedApi19DebugAndroidTest // launches three api19 devices
connectedApi21DebugAndroidTest // launches three api21 devices
connectedDevDebugAndroidTest // same as connectedApi21Debug...

4. When executed, these tasks will be detected by the Genymotion plugin, which in turn will launch the assigned devices, and run the tests, eg:

./gradlew connectedApi18DebugAndroidTest
...
<...>
genymotionLaunchConnectedApi18DebugAndroidTest
<...>connectedApi18DebugAndroidTest
<...>
genymotionFinishConnectedApi18DebugAndroidTest
...
BUILD SUCCESSFUL

5. If you want to run the tests on all devices, you simply need to create your own Gradle task…


task allConnectedAndroidTests(type: GradleBuild) {
tasks = [
'connectedApi18DebugAndroidTest',
'connectedApi19DebugAndroidTest',
'connectedApi21DebugAndroidTest'
]
}

6. …which when executed will run all the tasks specified within, in succession

./gradlew allConnectedAndroidTests
...
<...>genymotionLaunchConnectedApi18DebugAndroidTest
<...>connectedApi18DebugAndroidTest
<...>
genymotionFinishConnectedApi18DebugAndroidTest
...
<...>genymotionLaunchConnectedApi19DebugAndroidTest
<...>connectedApi19DebugAndroidTest
<...>
genymotionFinishConnectedApi19DebugAndroidTest
...
<...>genymotionLaunchConnectedApi21DebugAndroidTest
<...>connectedApi21DebugAndroidTest
<...>
genymotionFinishConnectedApi21DebugAndroidTest
...
BUILD SUCCESSFUL

The benefit of having a separate task, like allConnectedAndroidTests to run all your api tests, as opposed to relying on connectedAndroidTest is that the latter would also run connectedDevDebugAndroidTest, which in our example is the same as connectedApi21DebugAndroidTest, introducing unnecessary repetition. The “dev” flavor in my example serves as a identifier for a subset of devices that can be tested on on every push to the repo, but as this will also create its dedicated *androidTest* task, we need to explicitly exclude it from our full, CI nightly task - allConnectedAndroidTests.

Bonus: that’s a lot of Build Variants you have there…

Yes, it’s easy to see how adding productFlavors just for smaller sets of test devices can create many buildVariants that a) won’t make sense, and b) will pollute the Build Variant drop-down menu in Android Studio. Here’s a snippet that will help you hide the unnecessary ones, grabbed from Android Tools Project Site.

android {
productFlavors {
dev; // dev / smoke tests
api18; api19; api21; // selected api levels tests
}
    variantFilter { variant ->
def names = variant.flavors*.name
if (variant.buildType.name == "release") {
names.each {
if (it.startsWith("api")) {
variant.ignore = true
return;
}
}
}
}
}

A complete sample demonstrating this solution can be found on GitHub.*

*an Indie or a Business license is required to compile and run the sample

Thank you for reading!