Running Cucumber Features in Parallel on Mobile Devices

Test Pyramid

This is a test pyramid and the image says it all, UI functional test suite should be lean for two important reasons; Time to execute and the cost to maintain. Every framework solves the problem of execution time more or less in the same way and hardly there are any better solutions in mobile automation world.

With JUNIT/TestNG parallelising web regression suite is a walk in the park with little challenges to overcome however the challenges increase when we try to parallelise a mobile suite.

  • How to determine the number of devices available at the run time?
  • How to assign tests to devices for different test run modes?
  • How to maintain either multiple appium server instances or one server with multiple sessions?
  • How to communicate between different sessions in the case of Inter-APP testing?

Jonathan Lipps in his latest appium pro edition writes about parallelising mobile tests primarily using Maven and JUNIT style test cases. Here in this post we will delve into another important test stack, Cucumber and Gradle.

There are three key challengers to overcome in parallelising cucumber tests with Gradle build system

  1. Device Management — Detecting the number of devices available for a build (or run)
  2. Multiple Appium Sessions — Creating as many appium sessions for as many devices
  3. Feature File Distribution — Distributing feature files to all the threads for execution

OPTIMUS is an advanced mobile automation framework which solves the above challenges using a couple of handy libraries and keeps QA’s focus on the Functionality First approach.

Challenge 1 : Device Management

Device Management

Optimus MDB (Mobile Debug Bridge) is an intelligent device management tool that maps the devices connected to the test runner to a Device Matrix.

import com.testvagrant.mdb.android.Android;
import com.testvagrant.mdb.ios.IOS;
List<DeviceDetails> androidDevices = new Android().getDevices();
List<DeviceDetails> iosDevices = new IOS().getDevices();

We can also query for a specific devices connected to the runner using certain filters

List<DeviceDetails> androidDevices = new Android().getDevices(ofTypeEmulators());
List<DeviceDetails> androidDevices = new Android().getDevices(ofModel('Nexus'));

A DeviceDetails object keeps a track of all important parameters of the device attached to the runner as below

1. deviceName
2. platform (ANDROID | IOS)
3. platformVersion (eg: 6.x (for android) | 10.x (for iOS))
4. deviceType (DEVICE | EMULATOR | SIMULATOR)
5. udid
6. status (AVAILABLE | ENGAGED) [This property can be used when assigning a feature to a device]

Challenge 2: Multiple appium sessions

While creating multiple appium sessions for iOS is bit different than android. From Xcode 9 onwards it is possible to run tests in parallel on iOS and you can read more about it here.

Incase of ANDROID we need to pass a random BOOTSTRAP_PORT_NUMBER and for iOS we need to pass a random WDA_PORT which can be created as below


Challenge 3: Feature File Distribution

There are two typical modes we can run our feature files during a test execution.

  1. Fragmentation: Fragmentation mode is used to verify device coverage during regression. In this mode you can run all features on all devices.

2. Distribution: Distribution mode is used to verify functional coverage during regression. In this mode feature files are distributed almost equally on all devices available during a run. This mode can be used to get a fast feedback on a build for every commit.

To achieve the above run modes we will use GPARS, a concurrency and parallelism framework for Groovy and Java which offers a robust mechanism for distributing feature files.

Create a Fragmented Task

To run a feature in fragmentation mode we need to create a thread pool of number of devices connected and pass all feature files to each device.

A gradle task can be created as shown below

This definition will distribute all the feature files on all devices available. udidList can be created using MDB as we have seen in Challenge 1

Create Distributed Task

To run a feature in distribution mode we need to create a thread pool of number of devices connected and distribute feature files across devices.

A gradle task can be created as shown below

This definition will distribute a feature file to a device during execution.

Putting it all together

Running cucumber features in parallel involves overcoming the three key challenges which we have seen so far and can be summed up as follows

  1. Use MDB to find the list of devices attached to the runner during run time
  2. Pass the list of devices to the gradle tasks Fragmentationand Distribution
  3. Pass the feature files to the respective tasks.

Find out more about optimus on our optimus site


Handy Libraries

Following are some of the handy libraries which helps you in running cucumber feature files in parallel on mobile devices

  1. MDB — Mobile Debug Bridge to identify list of devices connected to the runner

Link: https://github.com/testvagrant/optimus/tree/master/mdb

2. Optimus Gradle — A Gradle plugin to run cucumber features in parallel and reporting

Link: https://github.com/testvagrant/optimusgradle