Grouping Appium tests with TestNG and Surefire

Adding more tests to an automation framework requires investing into flexibility in selecting tests for execution. We’ve faced this problem at N26, when the number of our mobile tests exceeded more than twice of what we defined as smoke suite. This blog post will explain how we handled this challenge.

Building the framework

To give you the background for our solution, I will first explain how our mobile automation framework is built. We use Appium, Java, Maven and TestNG, and both iOS and Android tests are in the same codebase. The code is structured using Page Object Model, meaning our screen (page) classes are separated from our test classes. Tests are executed against staging environment without any mocks. All existing test classes are added to two Suite XML files, one per platform, namely:

• tests-for-android.xml
• tests-for-ios.xml

Additionally, we have defined platform parameter in a Suite XML file:

Exploring TestNG possibilities

TestNG offers a feature called Test Groups. It allows the creation of simple or more complex groups of tests that later can be invoked when running the testng.xml file (see the very comprehensive TestNG documentation for details).

We’ve added the group parameters“smoke” and “functional” in the @Test annotation. The "smoke" group is meant for the happy path of high priority features. The functional group is meant for all tests. Every test tagged as smoke must be a part of the functional group as well.

Most of our tests are dependent on each other within one test class. If your setup looks similar, don’t forget to write your tests so that they can still run without interruptions when only a certain group is selected. Here is an example:

Important note: If you have any generic methods annotated with @BeforeClass, @BeforeMethod, @AfterClass, @AfterMethod etc., don’t forget to add the alwaysRun=true parameter. Otherwise, those methods will be excluded from the execution when you use the TestNG Groups feature.

Having group parameters in your tests allows you to configure the file testng.xml. However, as I mentioned before, we have two Suite XML files, one per platform. If we would like to add group configuration on top of that, we would need to split each XML file into at least two, one for the smoke group and one for the functional group. Adding more groups (e.g. by feature, like “login”) would force us to add more and more testng.xml files for each platform, and would make maintaining those files a bit harder. We’ve decided to take a different approach and use Surefire + TestNG together.

Combining the forces of TestNG and Surefire plugin

The Maven Surefire plugin is used by default whenever a test goal is executed. It also comes in handy when you want to add more configuration.

If you already have added Surefire plugin to your pom.xml file, it should look something like this:

Firstly, we want to make sure that the right Suite XML file runs for each platform, so we’ll use the suiteXmlFiles parameter and add the platform as an OS_TYPE environment variable.

Great! Now we can run iOS or Android tests from the command line:

OS_TYPE=ios mvn clean test

or

OS_TYPE=android mvn clean test

The next step would be to select which group should be executed, so let’s tweak the pom.xml file once more by adding the GROUP_TYPE variable in the <group> parameter.

Simple, right? Now you can run tests by specifying both platform and group:

OS_TYPE=ios GROUP_TYPE=smoke mvn clean test

Important note: Speaking of environment variables:

1. Make sure the variable is defined using uppercase letters to be platform-independent (so that you can run your project on Mac, Linux and Windows OS).
2. Calling our mvn clean test with an environment variable will set this variable in your system. Please make sure that the variable name is unique and you didn't overwrite anything important (e.g. like your JAVA_HOME path).

That’s it! This is our solution for a configurable test matrix that you can run from the command line but also set it up on Jenkins and Docker. We’re looking forward to hearing about your solutions for automated test suites!

Interested in joining one of our teams as we grow?

If you’d like to join us on the journey of building the mobile bank the world loves to use have a look at some of the Tech roles we’re looking for here.