Using Firebase Test Lab in multi-module Android projects
It can be tricky to run your instrumentation tests in CircleCI when your project has several modules. Here we’ll explain a simple way to do it
If you’re a developer concerned with quality, you surely write tests in your app. When you have a large enough project, executing the tests locally becomes less and less of an option, and normally you rely on a CI system to execute them for you. At 21Buttons, we are using CircleCI to do it. In Android, executing Unit Tests in a CI environment is not difficult, you can just use ./gradlew testDebug. But if you want to execute instrumentation tests, it’s not that easy since CircleCI does not support it out of the box.
To solve this problem, I think that one of the best options is to use Firebase Test Lab. It’s also the one that CircleCI recommends. Let’s see what it is.
What’s it?
If you don’t know Firebase and you want to know what it is and what’s capable of, I recommend you this series of posts that talk about it. There’s even a specific one for Firebase Test Lab.
To put it in a short way, Firebase Test Lab allows you to execute your tests on a cloud automated environment using Google’s infrastructure, where you have a large array of real devices or emulators to use in your tests. In order to use the test lab, you need to create an account in Firebase, have a project on Firebase and set up a Google service account. This will allow Google to authenticate your app.
How much does it cost
At the time of this post’s writing, with the free tier you’re allowed to 10 suite tests per day (5 if using real devices). Each emulator you launch will consume one of these suite tests. If you want to split your tests in several sub-groups to speed up the execution (the so called shards) each shard will count for the quota too. Each module generates a different apk for testing, so it will launch a new test instance every time. In our specific case, we have a big module containing mostly old code, that has about ~400 instrumentation tests in it, and we usually split it in several shards (otherwise we would exceed the time limit), and then about ~20 small modules. You can clearly see that with our setup, we cannot run the tests in the free plan even if it was only once a day.
How does Firebase Test Lab work?
We won’t get very deep into the specifics on how to test our app using the Google Cloud SDK (gcloud) because, with our approach to the problem, we will not need it much. If you take a look at the documentation, you’ll see that the setup is not super straightforward. Fortunately, CircleCI team put up some documentation which makes it a bit easier.
If you have a multi-module project, the problem lies in this step:
Build the debug APK and test APK. Use Gradle to build two APKs.
Then you would execute a step like:
- run:
name: Test with Firebase Test Lab
command: >
sudo gcloud firebase test android run \
-— app <local_server_path>/<app_apk>.apk \
-- test <local_server_path>/<app_test_apk>.apk \
-— results-bucket cloud-test-${GOOGLE_PROJECT_ID}With this approach, you would have to write this piece of code for every module you wanted to test, which can be prone to errors, and not a very extensible solution. But don’t give up yet! We’ll present you with an easier solution to deal with this.
The easy way
After seeing this problem, we searched for solutions and we found this Firebase Test Lab Plugin Library. We saw it was very simple to use and it abstracted you about details on how to use gcloud. To use it we created a gradle script like:
And then we just add
apply from: "$rootProject.projectDir/gradle/with-firebase.gradle"in every module where we want the instrumentation tests to be executed, that way we’re not repeating a lot of code.
For this to work, previously you will need to have in your project a json file (here named gcloud-service-key.json) that you get when you create a Service Account in Google Cloud Platform. This is a sensitive file, so I recommend you not to store it in your repository and let your CI handle it as an environment variable when you need to execute the tests. Here’s an example on how it would look like if we wanted to add this step on CircleCI:
As you can see, we stored our service json file as an environment variable in CircleCI and then we pass it to the workspace (the root folder where your code has been checked out before). We also added a no_output_timeout: 2h parameter, this is because in our case, this task can take up a long time to finish and CircleCI will wait 10 minutes after it times out. Even while this task is being executed, you can see your test results at your Firebase Console, under the Test Lab section. Every time a new module has finished its tests, it will appear there.
And that’s all for us! We hope this post has been helpful to you and… happy testing! 🎊
