Bitrise — Run Android instrumented tests on different modules within the same project

I ran into this issue recently, and I found very little documentation (most of which was out of date) so I decided to write a pretty simple step by step tutorial.

First of all, let’s start with a small introduction on what is an Instrumented test. Quoting Google official docs:

Instrumented unit tests are tests that run on physical devices and emulators, and they can take advantage of the Android framework APIs and supporting APIs, such as AndroidX Test.

A great advantage of unit tests is that you can test real instances of your classes (without mocking them), or you can simply run UI tests (those kinds of tests are also referred as UI tests indeed), but on the other hand they are quite slow.

How to run an instrumented test

Disclaimer: here I will assume you already have (or know) how to set up a basic workflow on Bitrise.

Running Instrumented tests on Bitrise is quite simple, we have a dedicated step you can use for that purpose:

Let’s focus on the step called Android Build UI Test (I renamed it to Android Build for UI Testing — ui module, I always rename steps to better reflect what they are actually doing): it is pretty simple, you need to put the module for which you want to run the instrumented tests (in my case ui) and which variant (in my case debug).

After that step you can just add the step Virtual Device Testing for Android, chose instrumented as test type and it will run all the tests built on the previous step.

So what’s the problem?

The problem comes in when you need to run multiple instrumented tests for different modules.

In other words: you need 2 (or more) Build for UI Testing steps and (and 2 or more Virtual Device Testing for Android).

If you try to do that, you will get an error, something like this:

This is a known issue. Is not really an issue, is how Firebase Test Lab works (which is used by the Virtual Device Testing for Android step): that step sends a build slug to Firebase for every build, since the main apk (which is the app apk) is always the same, it will send the same build slug for both steps, throwing the mentioned error.

So having something like this:

where I run instrumented tests for 2 modules (ui and data) will make Bitrise build fail.

Solution

Luckily there is a solution to this problem: run the instrumented tests in different workflows.

In Bitrise we can trigger a workflow from a workflow, let’s take a look at the next one:

As you can see here, we have a parent workflow, and 3 are triggered* from the parent one:

• One running all our unit tests
• One running all the instrumented tests in our data module
• One running all the instrumented tests in our presentation module

*You can trigger another workflow from a workflow adding a Bitrise start build step

Let’s look into detail what this step requires in general:

• A bitrise access token (you can easily set up one within Bitrise)
• The name(s) of the workflow(s) that we want to trigger
• Eventual env we want to share across the builds
• Wait for build: is important to set this to true, otherwise triggering instrumented tests in parallel builds will lead us the same error we had previously

If any of those builds fail, the entire parent workflow will fail. That’s handy in some situations: we can, for example, add a step at the end of the parent workflow (like sending a slack message, or uploading a new Beta apk) and it will be executed only if all tests pass.

Considerations

This approach solves our initial problem, which was our main purpose. However, there are few downsides:

• You cannot share the whole project across the builds, that means you will have to clone the entire project for every workflow (=for every test)
• Build times will slow down a lot compared to if all the tests were run in the same workflow
• You will have to repeat the same steps across different workflows, which is not ideal

Nothing really big, but nice to keep into consideration.

There is another approach though, which is to set up manually Firebase Test Lab and send related commands through Bitrise, but I haven’t looked into it.