Running Instrumented Tests on Github Actions with Firebase Test Lab

Hello guys welcome back to another article, In the previous article we learnt about generating UI state in android. In today’s article we will learn how to run instrumentation tests on GitHub actions via Firebase Test Lab

Normally when we set up CI/CD pipelines we usually follow 3 general steps namely: Build,Test and Deploy. In android development we have different type of tests

• Unit Test
• Instrumented Test

Unit Test usually run on the JVM and only test a single portion of your code i.e a class or a function

Instrumentation tests usually run on an android device (Physical or Emulator).These tests usually need specify android components to run hence the need for an android device

On a CI/CD platform like github actions running unit tests is usually straight foward since you just setup the JDK and run the gradle command as shown below

on:
  push:
    branches: [ main ]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up JDK
        uses: actions/setup-java@v1
        with:
          java-version: 11
      - name: Cache gradle
        uses: actions/cache@v1
        with:
          path: ~/.gradle/caches
          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
          restore-keys: |
             ${{ runner.os }}-gradle-    
      - name: Run Unit Tests with Gradle
        run: ./gradlew test

But when it comes to instrumented test it is a bit tricky because we don’t have a android emulator on our CI server.While there are a few solution to this like using this action on a macOS runner

The solution I am about share makes it so easy

In comes Firebase Test Lab

As per the description , Firebase Test Lab is a cloud-based app testing infrastructure that lets you test your app on a range of devices and configurations, so you can get a better idea of how it’ll perform in the hands of live users.

Assuming you have an android project in which you have instrumented tests already written you can proceed

1. Setup a firebase project

For this you need to set up you android app with a firebase project For more info on how to get started check here

2. Set up your firebase Test Lab Service Account

For use to use the firebase test lab services we need a service account file to help the service authenticate us. To do that we need to go to the Google cloud console here and create a service account (Every firebase project is essentially a Google cloud project). To create a service account to be used in a CI system check here (In the example above they use Jenkins but the same can be used with github actions)

3. Build your androidTest Apk

Once you have defined a device configuration, you should now build and androidTest apk. The androidTest apk packages all of our tests defined in the androidTest source set into an apk that will be used to execute the tests. The androidTest apk is usually upload to firebase test lab and the apk is executed in the cloud.

4. Define a device configuration

For use to execute the apk we need a device. We need to specify some device configuration to test firebase test lab what kind of device we want our apk to run on. Note that you can run your tests on multiple different types of devices just to make sure you app runs correctly in different devices

To define a device configuration we create a YAML file in the root of our project e.g android-device.yml

In there we add the following

android-pixel-4:
  type: instrumentation
  app: app/build/outputs/apk/debug/app-debug.apk
  test: app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
  device:
    - model: redfin
      version: 30
      locale: 'en'
      orientation: portrait

In the above example we create a device configuration called android-pixel-4 with the following settings :

• The app- key are a path to the actual debug apk
• Test - Path to the android test apk
• device - Device specific settings
• Model — Device ID of a device supported on firebase test lab. For more device options check here
• Version — Version of the android OS
• Locale -
• Orientation — Orientation of the device when the tests run

5 : Run our tests

Now we are done with the setup, let’s run our tests To run our tests on our workflow add the following to your workflow file We use this action to access the firebase test lab APIs in our CI environment

- name: Build App with Gradle
        run: ./gradlew build

      - name: Build Android Test APK
        run: ./gradlew assembleAndroidTest
      - name: Run Android tests on Firebase Test Lab
        uses: asadmansr/Firebase-Test-Lab-Action@v1.0
        with:
          arg-spec: 'android-device.yml:android-pixel-4'
        env:
          SERVICE_ACCOUNT: ${{ secrets.FIREBASE_TEST_LAB_SERVICE_ACCOUNT }}

The arg-spec specifies the device we defined in the android-device.yml appended with the device settings. We then put the service account in our GitHub action secrets and reference it in the file

If all is good we should see our test results in the firebase console like this

Troubleshooting

If for some reason you workflow fails at the firebase test lab stage, check the log messages, they can help in deciphering the problem but here are a few possible causes :

• You wrongly configured the service account i.e. you gave the wrong permissions when creating the service account. Go over the instructions again
• You probably defined a wrong path to your apks
• The tests probably were uploaded but failed, Check your tests
• You passed a model id for your device configuration

I hope you liked this article and I hope it helped you run your instrumentation test more reliably on your CI workflow

For the example project I used here you can find it on GitHub here

You can also follow me on github and twitter

See you next time 👋👋