Setting up CI/CD for Android test infrastructure

Hey everyone! I am Alexey Pak, I am engaged in the manual and automatic testing of the Android application for the PREMIER online cinema.

After implementing Allure TestOps and completing autotests on Kaspresso, our team faced a new task. How to make our tests running in CI/CD? In this article, I explain to you how to set up the integration between Kaspresso, Allure TestOps, and GitLab CI. I’ll share my experience, tell you about the problems we encountered and the solutions we found, so that you can set up your test environment on your own without making the mistakes I made.

The article is useful for those who have thought of running autotests in CI/CD and receiving reports in Allure Test Ops and for those who are implementing their test infrastructure. To automate application testing, we write native autotests in Kotlin. In our Android application project, we use Kaspresso, Junit, Allure Test Ops, GitLab, and CI/CD.

Kaspresso is a flexible and convenient framework for UI automation. The framework provides a wrapper for Espresso called UI Automator. We chose this framework because of the following advantages:

· Good readability of the test code

· Solution for flaky tests

· Advanced logging

· Working with AdbServer

· Ability to record video and take screenshots

We also use a variety of additional Kaspresso features, such as disabling Wi-Fi and data networks to test the application offline, emulating calls and SMS, and much more. You can see the full list here.

Allure TestOps allows you to write test cases for manual and automated testing, as well as create defects, view reports, and manage runs.

Let’s start setting up the integration between Allure TestOps and GitLab CI. It consists of the following steps:

1. Downloading test results from GitLab

2. Authenticating GitLab build

3. Setting up a GitLab project to upload test data

4. Creating a build script

Downloading test results from GitLab

GitLab loads test data using allurectl. Allurectl is a command line utility for the Allure TestOps API, which loads test results from the pipeline and manages actions in Allure TestOps (test cases, runs, projects).

Authenticating GitLab build

At this step, set up authentication in TestOps and GitLab. To do this, generate a secret token and write it in the CI/CD settings.

1. Log in to TestOps using your personal account.

2. Go to Your profile.

Going to Your profile

3. In the API tokens section, click on Create.

Clicking Create

4. Enter the ALLURE_TOKEN name and click Submit.

Copying ALLURE_TOKEN

5. Save the secret token.

6. Next, move on to creating variables in GitLab to bind them to Allure TestOps.

Setting up a GitLab project to upload test data

Add information about the Allure server to the GitLab settings.

1. Go to the CI/CD project settings.

Selecting the CI/CD section

2. In the Variables section, create 3 variables:

§ ALLURE_ENDPOINT — Allure TestOps server address, for example, http://allure.company.com.

§ ALLURE_TOKEN — Allure TestOps token.

§ ALLURE_PROJECT_ID — ID of the Allure TestOps project.

More details here.

Variables section

3. Move forward to the final part and create scripts for running autotests.

Creating a job

Set up the pipeline in GitLab. Create a .gitlab-ci.yml file in the project root. This file contains scripts for CI/CD.

In the following example, let’s consider launching the smoke autotests manually using a button. This is how the final script for running autotests looks:

smokeTests:
  when: manual # launch only manually
  stage: test # application build stage
  before_script: # application build stage
   - adb start-server > /dev/null # launch adb-server 
   - ${ANDROID_HOME}/emulator/emulator -avd test -no-window -no-audio -no-boot-anim -logcat '*:s' & adb wait-for-device > /dev/null # launch the emu-lator
   - adb devices # display the emulator
   - adb shell settings put global window_animation_scale 0 & # disable animation
   - adb shell settings put global transition_animation_scale 0 &
   - adb shell settings put global animator_duration_scale 0 &
   - /android-wait-for-emulator
   - adb shell input keyevent 82
   - wget https://github.com/allure-framework/allurectl/releases/download/2.2.1/allurectl_linux_amd64 -O ./allurectl
   - chmod +x ./allurectl
  script:
  - ./gradlew :app:installGoogleDebug :app:installGoogleDebugAndroidTest  # install the build
  - adb shell input keyevent 82
  - adb shell am instrument -w -e package com.android.test.package com.android.test/com.kaspersky.kaspresso.runner.KaspressoRunner >  tests.log # run tests
  - adb pull sdcard/Documents .# save data about the run
  - adb shell rm -rf /sdcard/Documents/logcat # clear test data
  - adb shell rm -rf /sdcard/Documents/screenshots
  - adb shell rm -rf /sdcard/Documents/video
  - adb shell rm -rf /sdcard/Documents/view_hierarchy
  - ./allurectl upload Documents # upload test data to Allure TestOps
 artifacts:
  expire_in: 1 week # artifact lifetime, it should be stored only for one week
  paths:
   - /builds/com.android.test /android/tests.log # path to artifacts
 after_script:
    - >
   if grep -w 'FAILURES!!!' tests.log; then
     curl -X POST $TELE-GRAMM_CHAT"text=Project:+$CI_PROJECT_NAME+$CI_PROJECT_URL/pipelines/$CI_PIPELINE_ID/ Branch:+$CI_COMMIT_REF_SLUG - has FAILED!"
   else
     echo 'This will only run on success'
   fi

· before script. This block is used for preconditions — the commands executed before the main script.

• script. The script block is used for the commands executed on the runner; it contains the main commands, for example, running autotests.
• after script. The commands in this block are executed after the main commands.

Let’s look at some commands in more detail.

We use a custom docker with an emulator. You can also find a ready-made image and add it using the before_script block. Launch the emulator with the keys.

- ${ANDROID_HOME}/emulator/emulator -avd test -no-window -no-audio -no-boot-anim -logcat '*:s' & adb wait-for-device > /dev/null

You can find information about launching the emulator and commands here.

Note that allurectl can work in the following modes: Non-CI mode and CI mode. Here we use the second mode.

Download allurectl to upload the reports to Allure Test Ops.

- wget https://github.com/allure-framework/allurectl/releases/download/2.2.1/allurectl_linux_amd64 -O ./allurectl
    - chmod +x ./allurectl

Install the required build on the smartphone emulator. Pay attention to the second part, this is a build with our tests, which must be installed for the tests to run.

./gradlew :app:installGoogleDebug :app:installGoogleDebugAndroidTest

Here our colleagues share the details about the builds.

We had to struggle a bit with running the tests, and decided as follows: this example runs all the tests in the com.android.test.package folder. We also use KaspressoRunner to run the tests.

adb shell am instrument -w -e package com.android.test.package com.android.test/com.kaspersky.kaspresso.runner.KaspressoRunner >  tests.log

The output is sent to the tests.log file, as an error may occur if the log is very large.

Official documentation on command line testing can be found here.

Next, copy the test data from the folder. In Kaspresso 1.4.2, data was created in the /data/data/packagename/files folder, and administrator permissions were required to copy the data.

In Kaspresso 1.5.1 this behavior has been corrected and the data is saved to the sdcard/Documents folder.

Save the folder from the device.

adb pull sdcard/Documents .

Upload the test reports to the Allure TestOps server.

./allurectl upload Documents

After sending reports, delete the files from the emulator, as they take up a lot of space.

- adb shell rm -rf /sdcard/Documents/logcat
- adb shell rm -rf /sdcard/Documents/screenshots
- adb shell rm -rf /sdcard/Documents/video
- adb shell rm -rf /sdcard/Documents/view_hierarchy

The run is published in the following format:

Test run in the messenger channel

4. Launch the pipeline.

Launching the pipeline

5. In the logs, you can see a link to the run in Allure.

Pipeline data

6. After the run is completed, you can see a report in Allure TestOps.

Report in Allure

Official documentation is provided here.

In this article, we learned how to set up the integration of autotests on Kaspresso, Allure TestOps, and GitLab CI and how to run autotests in CI/CD. Now you can set up automatic running of tests according to a schedule or by a trigger and enjoy reports in Allure.

We don’t say goodbye to you. Next time, we can tell you in more detail about building the emulator or our test architecture.

I would like to express my deep gratitude to the authors and the Kaspresso community for their active help.

P.S. Bonus for those who read to the end

If you run your pipeline now, it will succeed, but the run may contain failed tests. The reason is that after launching, the job always passes successfully and the failed status doesn’t appear if the tests fail. As a workaround, we wrote a small script that sends the run results to Telegram. Save the log and look for the FAILURES keyword. In this messenger, we have a channel created and after the pipeline finishes, the bot publishes the results. The integration between GitLab and Telegram can be set up according to the official documentation.

after_script:
  - >
 if grep -w 'FAILURES!!!' tests.log; then
   curl -X POST $TELE-GRAMM_CHAT"text=Project:+$CI_PROJECT_NAME+$CI_PROJECT_URL/pipelines/$CI_PIPELINE_ID/ Branch:+$CI_COMMIT_REF_SLUG - has FAILED!"
 else
   echo 'This will only run on success'
 fi