TestNG. Life without XML: handling flaky tests.
I’m continuing my return journey to the testNG framework. I will try to keep it covered with short blog posts.
Some theory.
Let’s talk a little about automation in general. What do you think is the main condition for making it valuable? I answered this question for myself a long time ago. It’s a trust. No, it’s “THE TRUST”. Every person in your organization should know: “if there’s a failure, we are all in trouble”.
If automation is unstable, if there’re failures not related to the product defects, no one will take it seriously. In this case there’s a risk to deliver a bug to production even if it was cought by an automation test. So why do we need the automation at all?
But what about tests that are still in development? Or the ones that failed due to a known bug that was scheduled to be fixed in 2050? For this purpose I used to have 2 types of execution. One for the quality process ( you rememder: “if there’s a failure, we are all in trouble”) and one more for automation developers, who want to see the unstable part as well. Now let’s talk on how to implement it in testNG.
Implementation.
The technical task: I want to skip unstable tests when system property skipUnstable is set to true.
- We need an annotation to mark flaky tests:
2. We need to implement the IAnnotationTransformer interface:
As you see, I’m checking here if the property is injected (either -DskipUnstable or -DskipUnstable=true) and a method is annotated by @Unstable. If so I’m changing enabled property of @Test annotation to false. It will cause skipping the test.
Configuration.
In testNG documentation we can see the following:
The @Listeners annotation can contain any class that extends org.testng.ITestNGListener except IAnnotationTransformer and IAnnotationTransformer2. The reason is that these listeners need to be known very early in the process so that TestNG can use them to rewrite your annotations, therefore you need to specify these listeners in your testng.xml file.
But we (I) declared a war on testng.xml :) So… for local runs I add to the runner options (I use intellij plugin) the following:
-listener com.skippersoft.testng.listener.FlakyTransformer
Execution.
When running the project using the maven on CI pipeline, we can append the transformer as a Surefire plugin property:
Now in my configuration when the job is triggered by developer’s pull request it runs mvn test -DskipUnstable … and when it’s caused by manual aor scheduled trigger, it runs mvn test … .
PS. You can find the sample code on this github repository.
PPS. Pay your attention that you cannot specify more than one annotationTransformer. I will show how to pass this restriction soon. Follow me on medium so you don’t miss it out.
Other articles of this cycle: