Android UI Testing with Firebase

In the past few months, the Android team has been adding a suite of UI tests in order to achieve our goal of automated test coverage of the most crucial parts of the app. As the number of tests started growing, we began to realize that our current setup of a single machine in SF running a single device was no longer cutting it — we needed an upgrade.

Instead of trying to scale our setup locally in San Francisco, we decided to hand off the work of maintaining multiple build machines and Android devices to a third-party: Firebase. Firebase is a tool built by Google that provides a lot of useful features, from analytics and push notifications to app indexing and crash reporting.

What we were interested in was Firebase Test Lab. Test Lab allows us to run our UI tests on multiple devices, in parallel, with full debug logs and video of every UI test.

Firebase is very easy to use when running tests through its website, but we needed a solution that would allow us to connect the Firebase platform with our existing TeamCity build servers. And since there was no existing documentation about how to connect Firebase and TeamCity, we had to figure it all out on our own.

After reading through a ton of Google Cloud documentation, we were able to create a bash script that would automatically initialize the Firebase library on any of our TeamCity machines. Then, we simply needed build the application in TeamCity and to upload it to Firebase.

The process looks something like this:

All that firebasing set off the sprinklers. By Uncalno Tekno, licensed under Creative Commons 2.0

No, not like that. Like this:

1. Pull Request is created
2. TeamCity builds the Android app apk and the test apk from that branch
3. Both apks are uploaded to Firebase
4. When Firebase finishes running our tests, the results are stored in TeamCity
5. Status is reported to GitHub

We can easily customize the device orientation, type of device, and the OS version through the command line when we upload our apks:

gcloud beta test android run
 — orientations portrait 
 — os-version-ids 19,23 
 — device-ids mako,herolte 
 — results-bucket test-lab-turoandroid-102 
 — app app/build/outputs/apk/app-legacy-debug.apk 
 — test app/build/outputs/apk/app-legacy-debug-androidTest.apk

Previously, UI tests could only run on one machine. Whenever a pull request was merged and the new set of merge tests were started, you might have to wait hours until your test even started. Now, there are 35 machines that can run our UI tests simultaneously. And as we all are well aware, parallel >>>>> sequential.

Firebase’s verbose logging is also a huge help for us when trying to find the root of a test failure. Before Firebase, the process of finding the cause of a test flake was pretty much trial-and-error. Now with full device logging and video it’s cut down our test debugging time dramatically. All in all, Firebase Test Lab has made our testing process much faster and easier, and sets us up nicely for the future as we continue to add to our suite of tests.