TestNG. Life without XML: changing test discovery logic.

After working few years with Junit5, I recently started supporting a client with an existing TestNG-based project. It’s a very interesting process to return to the tool you used before after trying something new.

One of the first problems I faced is that test method discovery by multiple groups works according the ‘OR’ logic. It means that if have 3 tests:

@Test(groups = {"t1"}){...}
@Test(groups = {"t1", "t2"}){...}
@Test(groups = {"t2"}){...}

running TestNG with system property groups=t1,t2 will cause the execution of all 3 tests. And for this project I required to run the intersection of groups (i.e. only the second test annotated with both t1 and t2).

When you start googling about it you will probably find a few examples of using the beanshell syntax inside the testng.xml file. But as you see from the subject, I have 101 reason not to use the configuration file.

So my choice is to implement a custom method selector.

Step 1: Implementation:

In order to avoid a confusion with the standard testNG logic, I decided not to use the groups system property, so I will use tags instead.

public class ExpressionSelector implements IMethodSelector {
    
   @Override
   public boolean includeMethod(
     IMethodSelectorContext iMethodSelectorContext, 
     ITestNGMethod iTestNGMethod, 
     boolean b) {
        Optional<String> tagInput =  
            Optional.ofNullable(System.getProperty("tags"));
        return tagInput.isEmpty() ||
                asList(iTestNGMethod.getGroups())
                   .containsAll(asList(tagInput.get().split("&")));
    }
    @Override
    public void setTestMethods(List<ITestNGMethod> list) {
    }
}

So as you see if tags is empty it will run all tests, otherwise it will select only ones tagged by all tags (splitted by ‘&’).

Step 2. Configuration.

We already have a selector, but how to make it work? There’s no much documentation about it. The only thing we can found is:

-methodselectors: A comma-separated list of Java classes and method priorities that define method selectors.Lets you specify method selectors on the command line. For example: com.example.Selector1:3,com.example.Selector2:2

But… priority? What value should we get? Ok, let’s see.

I prepared a simple test class:

public class ApiTests {
    @Test(groups = {"t1"})
    void passedTest() {
        System.out.println("IT'S T1 TEST");
        Assert.assertTrue(true);
    }

    @Test(groups = {"t1", "t2"})
    void passedTest2() {
        System.out.println("IT'S T1 and T2 TEST");
        Assert.assertTrue(true);
    }

    @Test(groups = {"t2"})
    void failedTest3() {
        System.out.println("IT'S T2 TEST");
        Assert.assertTrue(true);
    }

Now running the testNG (I will use Intellij plugin) with following params:

-methodselectors com.skippersoft.testng.ExpressionSelector:1

-Dtags=t1&t2,

I expected to run only one test, annotated with both t1 and t2 groups, but the selector was ignored and all tests were executed.

Ok, it’s a debug time… After putting a breakepoint onto org.testng.internal.RunInfo#addMethodSelector, we can realize that priority of the standard XmlMethodSelector is 10. Ok, we’ll bit it:

• methodselectors com.skippersoft.testng.ExpressionSelector:11

After executing I got the desired output:

IT'S T1 and T2 TEST
===============================================
Default Suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================

Step 3. Execution.

Nice, but what if I want to run it using a Maven Surefire plugin? That’s the way I used to run this project on a CI server. Remember, I said that there’s no much documentation about using a command line key? Ok, so for Surefire case it’s not documented at all ( or I don’t know how to google).

The first thing that came to mind was to inject it using Surefire properties as I do it for listeners:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.0.0-M5</version>
    <configuration>
        <properties>
            <property>
               <name>methodselectors</name>                 
               <value>com.skippersoft.testng.ExpressionSelector:11</value>
            </property>
        </properties>
    </configuration>
</plugin>

And… it worked!

Probably I achieved the behavior I wanted to implement. Running:

mvn test -Dtags=t1&t2

executes only tests annotated by both t1 and t2 groups. ( So in the real life I can run sanity&UI and it will execute only the required subset ).

What’s next?

Probably the selector can be improved to get a real logical expression. Nothing should block us from running (Service2|ApiGateway)&Sanity&!Flaky :)

PS. You can find the sample code on this github repository.

Other articles of this cycle:

TestNG. Life without XML: handling flaky tests.