Trendyol Fulfillment Android Instrumented Test CI

Published in

Trendyol Tech

6 min readJun 20, 2023

Hello! We’re Tuğçe and Yunus Emre from Trendyol Tech. We have prepared an article about our a long-term adventure of Instrumented Tests in GitLab pipeline.

guerrero incansable

When a new feature will be released, we definitely check its effect on other screens before distribution. In this context, UI Test automations have a critical role. However, it takes huge effort to run all tests manually on a local machine every release time. So, it would be great to run these tests on our CI/CD pipeline before the changes merged to release branch.

For this reason, we decided to move automations to our GitLab pipeline.

How did this adventure start ?

We had 2 options to realize this desire:

Creating a Docker image with SDK and emulator installed
Using a custom Linux runner

Option 1: Creating a Docker image with SDK and emulator installed

First of all, since our current pipeline is running on a Docker image, at the beginning of each stage in the pipeline, all the SDKs and libraries that we need to build an Android app are downloaded from scratch. When we added the download of the necessary libraries for the installation and operation of the emulator, our development speed would be slowed as the uptime of our pipeline would increase. This was something we never wanted as it would increase our lead time. The scripts we’re talking about are like here.

We built an image on Docker with the required tools about Android SDK and emulator installed and tried to run the gitlab pipeline through this docker.

You can see an example of one of our Docker file to get a view how we were keeping it. As you are about to see, we are embedding all the thing inside our image to shorten our pipeline.

Then we tried to run emulator from gitlab following commands:

Then, everything started with the below CI pipeline error message as follows:

KVM: entry failed, hardware error 0x8
ERROR | crashhandler_die: fatal: kvm_arch_handle_exit: hardware error
happened in KVM, aborting nowSo, what was the reason for this error ? Reason was simple but a bit tricky, android emulator was behaving just like running a virtual machine inside our runner. See gitlab runner linked url if you are not familiar with it.

Runner is tough but joy..

So, what was the reason for this error ? Reason was simple but a bit tricky, Android emulator was behaving just like running a virtual machine inside our runner. See gitlab runner linked url if you are not familiar with it.

After some research, it seems like we need to enable qemu-kvm intel based cores in our runner. So just a quick preview, our runners are running on another Kubernetes cluster which is more scalable and efficient because after your pipeline ends, you will not need to clean disk space or additional resources because runner pod will be gone.

Then, we tried to enable virtualization in our docker images with openjdk-11, java and all the tools the emulator needs but failed. See below to QEMU enabled Dockerfile and start.sh:

But nope! It did not work for us. So, it seems like we are making a mistake somewhere or as we researched with our valuable colleagues we need to run up a virtual machine itself and enable KVM acceleration there. Meanwhile, we discovered important information about VM acceleration restrictions on Android’s official developer site. Facing this truth caused a serious loss of motivation for us 😭 The first option had failed…

Despite all the difficulties, we continued on the way we started. Because our motto is “No pain No Gain”! 🏋‍♀

Option 2: Using a custom Linux runner

OK, we started research about the VM acceleration problem except for the Docker solution. Then, we tried to build an Ubuntu-22.04 VM (by Openstack (aka KVM) hypervisor infrastructure design) at first. At Trendyol, we are using different virtualization infrastructures such as VMWare or Openstack or even Hyper-V).

We’ve set-up a Gitlab-Runner and registered it to our project to run tests on Ubuntu VM.Yeah, it was working with the below config in our gitlab-ci.yml on self VM. (Register a Runner on Linux)

Anyway, we still get the error even though the QEMU tests and KVM acceleration tests results were OK. There must be something else on the infra side, so we switched to the VMware solution to see its behavior.

Not to forget, here our developer friends from the Trendyol Android Platform team let us be aware of this issue, thanks to them. So, as you can see below screenshot, VMWare gives us the opportunity to enable KVM acceleration on its hypervisor interface. Yeah, benefits of working and getting approaches to different technologies.🤩

Meanwhile, we observed the behavior of the emulator and seemed like it was running on a single core so we set up 2 core per each 16 socket (Default one per socket). Then we activated CPU acceleration by enabling Expose hardware-assisted CPU virtualization to guest OS.

Not long now! Fasten your belts!

You may estimate the rest. We expanded the disk on the VM because emulator images take up a lot of space.

Then, we got permissions to add global variables (such as ANDROID_COMPILE_SDK, EMULATOR_VERSION etc.) to .bashrc to abondonnate them from our gitlab-ci.yml in before-script sections. So we cut the long and one-shot parts from out gitlab-ci.yml file.

But we still get an error differently about root user permissions. Because the emulator desires to be working with some super-user privileges. Ok, we may edit the config.toml as the screenshot below to give root permissions to gitlab-runner user:

But this may cause some vulnerabilites so we decided another simple way, making a script file (such as android_sdk.sh), then copy it to the /root/ directory to run some #apt-get and similar commands to update and upgrade packages in each step we run the emulator and let it flow. 🔥

Let’s go!

If your Linux runner is ready for VM acceleration, the following operations must be performed on your VM.

Download the required libraries for Android SDK. (one-time, if an update needs you can run again)

Download the required image to create the emulator you want to test (one-time for related emulator)

Turning off animations for the emulator to run properly.

Create and launch the emulator via command line.

You can use these parameters to speed up your emulator.

Run the ui test automation with gradle command on GitLab pipeline.

TA DAA!!! Here connectedMockDebugAndroidTest task is our instrumentation test task in gradle.

Learnings..

While unit tests can be run on a shared runner on Gitlab, the instrumentation tests require hardware acceleration which is not available on shared runners. So, the second option above must be preferred. You need to configure your own machine as a runner that Gitlab can use to run your UI tests!

REFERENCES:

That’s all folks! Thank you for your patience. We hope this article inspired you create your own UI test automation structure on GitLab. We’re waiting your feedbacks. 🙏

Tuğçe AKTEPE & Yunus Emre GÖRMÜŞ

We would like to thank our Android Platform, Fulfillment SRE and Fulfillment Mobile teams who contributed to the completion of this work.

If you are interested in this world, you can join Trendyol’s rich technology world.

Trendyol jobs

Job openings at Trendyol

jobs.lever.co