Whitebox & Blackbox automation testing on Android using Genymotion Cloud on AWS (Espresso & Appium)

Published in

Genymobile

4 min readMar 3, 2021

Test automation is becoming more and more the norm in the Android development space, as it guarantees more robust apps and a faster development lifecycle with earlier bug detection.

When putting in place an automation testing routine, most of the work lies in the design of the test themselves — but crucial decisions are to be taken too in terms of tools involved; and they have a major impact on your operations and project economics:

- Choice of testing framework that will support your tests portfolio

- Choice of Android devices where the tests will run

Your choice of testing framework depends on your application, your use cases, your testing strategy, and much more; two main broad categories of tests can be mentioned:

- Blackbox testing focuses on testing the application functionality from the user perspective without the knowledge of the source code and its architecture. We usually test the user story. Popular blackbox testing frameworks are Appium, …

- Whitebox testing will go over the specifications of the application and will test each implementation and have access to the source code of the application. Popular whitebox testing frameworks are Espresso, …

Your choice of Android devices will have an impact on speed, scalability, and economics of the project. While real devices are needed to test elements that are specific to the manufacturer layer, in most cases, virtual devices will perfectly do the job — and more! Indeed, virtual devices have the following advantages:

- Faster: no need to queue to launch tests, virtual devices can be started and stopped in a second… at scale. Run as many devices as you want in parallel, with no waiting time.

- More manipulable: with virtual devices, it’s possible to inject values in sensors such as camera or GPS to test different scenarios.

- Cheaper: virtual devices are usually 10 times cheaper than real devices. Our solution Genymotion Cloud on AWS starts at $0.13 / hour + EC2 fees, with most commonly used devices costing $0.7 / hour all included; meanwhile, a typical real device will cost you around $5–10/ hour depending on the solution you choose.

Genymotion Cloud offers cloud-based Android Virtual Devices fully hosted on AWS. In this article, we will cover how to run Appium & Espresso tests on our Genymotion Cloud virtual devices hosted by AWS. This will allow you to test your applications using multiple device versions from Android 5.1 to Android 10.

There are different steps :

1/ Launch devices in Genymotion Cloud
2/ Launch your Appium or Espresso tests
3/ Stop devices

1/ Launch several Genymotion Cloud devices on AWS

To start devices, you can refer to AWS ec2 APIs

Below is an example using AWS CLI where we start an instance

#start instance
aws ec2 start-instances --instance-ids $instance_id#check its state
state=$(aws ec2 describe-instances --instance-ids $instance_id --query "Reservations[].Instances[].State.Code" --output text)#wait for the device to start, check every second device state
while [ $state -ne 16 ]; do
   state=$(aws ec2 describe-instances --instance-ids $instance_id --query "Reservations[].Instances[].State.Code" --output text)
   echo -n '.'
   sleep 1
done#get ip of the instance
ip=$(aws ec2 describe-instances --instance-ids $instance_id --query "Reservations[].Instances[].PublicIpAddress" --output text)

We check the state of the device to connect to it when the device is started

You can also use terraform to simplify the start of an instance and get the ip address

Connect devices through adb

You can enable ADB using this simple API call :

curl -X POST “https://[instance_ip]/api/v1/configuration/adb" -H “accept: application/json” -H “Content-Type: application/json” -H “Authorization: Basic [CREDENTIALS]” -d “{\”active\”:true,\”active_on_reboot\”:true}”

Note : CREDENTIALS is the value of `username:password` encoded in base64. By default, username is genymotion and password is the instance id. If these have been changed by APIs or from the user interface you need to use your own credentials.

When it’s done, you just have to connect your device using adb :

adb connect $instance_ip:5555

Of course, you have to repeat these actions as many times as the number of running devices needed!

2/ Launch your Appium or Espresso tests

a/ Appium

Requirements

- Python 3 environment
- Android SDK platform tools with adb installed
- A working Appium install

Let’s start an Appium server, simply using the basic command:

appium

Write your tests in Python

(Again, we’ve used Python throughout this tutorial but you can use your desired language.)

We’ve chosen to use Pytest as our test framework. It has several useful plugins like:

xdist: for running tests in parallel.
rerunfailures: for rerunning failed tests an arbitrary number of times to reduce build failures due to flakey tests

Here is a simple python script, please replace `[instanceX_ip]` values by the IPs of your instances:

How to run Python tests in parallel

Now it’s time to run tests on all devices!

If we run the python script as follows:

pytest test_example.py

pytest will execute tests on one device at a time. This is not what we want. In order to have results as soon as possible, we need to execute all the tests at the same time. To do that, we can take advantage of the pytest-xdist plugin and run the following command :

pytest -n 3 test_example.py

-n : number of worker processes you want to use. Here we start the suite with 3 processes

That’s it! You’ve been able to run parallel tests on several Genymotion Cloud PaaS virtual devices using appium and pytest.

b/ Espresso

When connected to the devices, simply go on the root of your project and run

./gradlew connectedAndroidTest

3/ Stop devices

Once your tests are finished, stop the devices using AWS APIs

- AWS CLI example

aws ec2 stop-instances $instance_id

- Terraform

Call terraform destroy and it will stop all the instances created.

That’s it! You’ve been able to run parallel tests on several Genymotion Cloud virtual devices using appium or espresso.
To learn more about Genymotion Cloud on AWS, you can refer to the user guide