Test-Driven Development –Ready, Set, Go with Google Cloud Test Lab
Have you ever worked on an Android app, written tests for it configured with CI, and felt it was a really tiring setup process? If so, you are not the only one and this article is specifically for you!
At Google I/O 2015, Google announced a new testing service called Cloud Test Lab that would allow developers to submit an app and run automated tests on multiple Android devices from the Cloud. Currently, there are eleven physical and six virtual devices you can choose from — including some of the most popular Android models and brands.
How does The Cloud Test Lab promise to ease Android automated testing complication?
Competing Continuous Integration Options
I have experience with a couple continuous integration (CI) options for Android. On past projects I have used:
- Jenkins with Genymotion.
- Circle CI with Emulator (Before Android Studio 2.0, this was THE EMULATOR we’d known for so long).
- Jenkins with USB connected physical devices.
However, none of these options were really fulfilling my CI needs because of the following reasons:
Complex Configuration — All of these options require some level of scripting and cannot simply boot up and run. For instance, some require a script to boot up which means waiting for an online signal that may take up to five minutes to find.
Instability — Sometimes these options were unpredictable — failing for no actual reason and succeeding for no actual reason. One time, a build failed twice and on the third try, without changing any settings, it succeeded. In general, the builds were not consistent, did not provide any good reasons for failures, and hence no teachable moments about how to avoid them.
Slow Speed — This may sound silly, though it’s important. Not all of us can afford a multiple slotted CI service. Sometimes I would run multiple app’s job onto a single instance. One job’s delay would cause a bottleneck to another job which would affect the next job and then everything would be delayed.
Scalability — Due to slow speeds, I wouldn’t consider running tests for multiple devices and multiple orientations on the CI options listed above. Often, running a test on one device, with one orientation, would take about 15–20 minutes for a general-sized project. And when I did try to run tests for multiple devices, it was not always easy to configure on several devices.
Also, running acceptance tests with only one configuration does not create a lot of value; especially considering Android screens vary from small to x-large and resolution differs from high to xxx-high across major devices.
Experimenting with Cloud Test Lab
Given the complex configuration, instability, slow speed, and scalability issues with other continuous integration options for Android, would Cloud Test Lab be a game changer? Am I happy to move testing away from these options and onto Cloud Test Lab?
Here is a basic rundown of what it was like to setup Cloud Test Lab.
Cloud Test Lab is still in beta. You will need to signup for beta access at https://console.cloud.google.com to use it.
Once you log in, open the navigation drawer on left top and select “Test Lab”. If you are not signed up yet, click “Sign up for a free trial” and follow the instructions.
Next, create a Google project at https://console.developers.google.com/project.
Setup
There are a couple of options on how to use Cloud Test Lab. Since my goal here is to make testing work with CI, I choose a command line option.
For Android Studio option, please refer https://developers.google.com/cloud-test-lab/android-studio
1. Install Google Cloud SDK. Open terminal and
$curl https://sdk.cloud.google.com | bash2. Restart the terminal. Now you can use gcloud command3. On terminal, make sure to have sdk up-to-date
$gcloud components update beta4. Configure your project ID. You can find yours at https://console.developers.google.com/
$gcloud config set project your-project-id5. Authenticate to use Cloud Test Lab with gcloud
$gcloud auth login
Now Let’s Run, Commander !
*Assume you are using Gradle wrapper.
**I added a sample app’s GitHub link at bottom so you can try this out yourself. If you have your own app, I encourage you to setup with your app.
1. Generate app and app test’s apk.
$./gradlew assembleDebug assembleAndroidTest2. cd app/build/outputs/apk3. Run Run Run
$gcloud beta test android run \
--type instrumentation \
--app app-debug.apk \
--test app-debug-androidTest-unaligned.apk \
--device-ids Nexus6,mako \
--os-version-ids 21 \
—-locales en \
—-orientations portrait,landscape
To find available device IDs, type $gcloud beta test android devices list
Similarly, $gcloud beta test android versions list and $gcloud beta test android locales list, to find available versions and locales, accordingly.
For more options: https://cloud.google.com/sdk/gcloud/reference/beta/test/android/run
EASY ?
Easy.
No more scripts, no more prepping devices by yourself, using THE EMULATORS, or devices.
AND
You can configure to run with multiple devices / multiple orientations with just one simple command.
Result Output
You can view the console output like this, as well as more a detailed breakdown at: https://console.developers.google.com/testlab
Is This a Game Changer?
Before Cloud Test Lab was introduced, I mentioned four complaints about current CI solutions — complex configuration, instability, slow speed, and scalability. Let’s compare those issues with Cloud Test Lab.
Complex configuration:
The general configuration and setup experience are by far the best with Cloud Test Lab. Without the Cloud Test Lab, I had to use a few scripts and some commands just to prepare a device for testing. Now, I can use just four-line commands for that.
Instability:
It’s a little too early to judge this just yet. When experimenting with Cloud Test Lab, I ran a simple app with only three tests in it and it performed well. To really find out how stable the Cloud Test Lab is, I would need to try it with a few industrial apps containing a few more tests.
Slow Speed:
This issue remains mainly the same. A test project with three Espresso tests took 3–6 minutes / config plus a few minutes to upload app.apk and app-test.apk to the Cloud Test Lab.
Scalability:
On the one hand, Cloud Test Lab is easy to setup and run with multiple configurations. However, slow speeds still persist and running on multiple configurations costs quite a bit of time.
Conclusion
From my personal use, testing issues have either improved or remained consistent with Google Cloud Test Lab. With additional new features (such as an automatic screen recording) and improvement to issues like configuration complexity, and a lengthy list of available devices, Cloud Test Lab is definitely designed for maximum use.
However, Google Cloud Test Lab is still in beta and there’s still room to improve. For instance, general performance could be improved and while the log video is a nice feature to have, the video lag makes it hard to debug a test failure with.
I will definitely follow how Cloud Test Lab goes in official release, even if I’m keeping my options open in the meantime.
- * Sample app with tests: https://github.com/emmasuzuki/EspressoSpoonDemo
For more on APIs, open source, community events, and developer culture at Capital One, visit DevExchange, our one-stop developer portal. https://developer.capitalone.com/
