Allure Reporting in Selenium using TestNG and Maven
Selenium is a widely used tool for automated testing of web applications. It is open-source, supports many programming languages, such as Java, Python, Ruby, C#, and Javascript, and works across multiple operating systems. However one of the missing features in Selenium is that it does not have any reporting mechanism.
Test automation reporting is a crucial thing, and without any reporting tool it becomes difficult to get statistics from the automation results like how many test cases got executed, how many Pass/Fail, and so on.
Be it manual or automation, reports should always be clear, and concise but comprehensive.
Considering the above points, in this blog I will help you understand how to integrate Allure reports with Selenium using Maven and TestNG.
Below are the main topics that I would be covering:
- What is Allure?
- Annotations in Allure
- Download Allure and set path in the system environment variable
- Adding Allure dependencies in pom.xml
- Running tests with Maven
- Generate Allure reports through the command line
- Viewing and understanding Allure reports
What is Allure:
· Allure is an open-source framework designed to create interactive and comprehensive test reports by the Yandex QA Team
· Integrating Allure with Selenium gives beautiful and presentable reports which have detailed information about the automation run. Basically, it generates reports based on maven surefire plugins.
· It supports popular CI/CD platforms like Jenkins, TeamCity, Bamboo, Gradle, and Maven.
· Allure Reporting framework works fine with any test framework like TestNG, JUnit, etc.
· There are several annotations in Allure reporting which give the following information in reports
The severity of the TC, description of TC, steps involved in that, and screenshot for the failure scenario.
Annotations in Allure:
First, let us get familiar with the annotations.
@Epic
@Features
@Stories/@Story
@Severity
@Description
@Step
@Attachment
Though this list is long, I would be explaining only a few Allure annotations that are used widely.
If you are familiar with TestNG annotations, understanding Allure annotations would be like a cakewalk😊
· @Severity: We can define any @Test with @Severity annotation with any of these values like BLOCKER, CRITICAL, NORMAL, MINOR, and TRIVIAL. By looking at this, we can understand the severity of the test if failed
· @Description: We can add a detailed description for each test method. Remember that this is different from the TestNG description attribute of @test annotation. Will see the difference in Allure reports later.
· @Step: In order to define steps in Java code, you need to annotate the respective methods with @Step annotation. When not specified, the step name is equal to the annotated method name.
· @Attachments: An attachment in Java code is simply a method annotated with @Attachment that returns either a String or byte[], which should be added to the report
Download Allure and set path in the system environment variable:
1. Go to the Releases section of Allure on its GitHub page https://github.com/allure-framework/allure2/releases
2. The latest release would be at the top. Scroll down to the Assets section and download allure-2.17.3.zip (The current release is 2.17.3, you may take the latest release from GitHub when this gets older). Make sure you download the correct zip. Don’t download the source code.
3. After downloading, paste the Allure zip in any folder say for example D:/, and extract its contents.
4. After extracting, there should be folders as pictured below:
5. Open the system environment variables by clicking on the Windows icon on your system and searching for env. Open the system properties and click on Environment Variables.
6. Copy the allure bin path where you have placed the allure unzipped file. In my case, it’s “D:\Allure\allure-2.17.3\allure-2.17.3\bin”. Select the “PATH” variable from system variables and click on Edit. Click on New in edit environment variables and paste the copied allure bin path. Save the settings.
Adding Allure dependencies in maven’s pom.xml:
Now let us add the dependencies in the pom.xml file.
Assuming that Selenium and TestNG dependencies are already available in the pom.xml (You can refer to my medium page for adding the selenium and TestNG dependencies if you wish to), let's add Allure dependencies in pom.xml.
Open the pom.xml file in eclipse/ any editor that you are using and add the following dependencies related to the Allure:
- aspectj property:
Add aspectj property under <properties>
<properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <aspectj.version>1.9.9.1</aspectj.version></properties>
2. allure-testng dependency:
Add allure-testng dependency under <dependencies>
<dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-testng</artifactId> <version>2.17.3</version></dependency>
3. Maven compiler plugin and maven surefire plugin:
Add maven compiler plugin under <plugins>
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.8.1</version></plugin>
Add maven surefire plugin under <plugins>. Inside maven surefire plugin add java agent argument line under <configuration> and add aspectjweaver dependency under <dependencies>
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.22.2</version> <configuration> <argLine> -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar" </argLine> <suiteXmlFiles> <suiteXmlFile>testng_jenkins.xml</suiteXmlFile> </suiteXmlFiles> </configuration>
<dependencies> <dependency> <groupId>org.aspectj</groupId> <artifactId>aspectjweaver</artifactId> <version>${aspectj.version}</version> </dependency></dependencies></plugin>
If this is too confusing to understand do not worry. You can check the entire pom.xml in this bitbucket repo.
Running tests with maven:
Now all the configuration in pom.xml is done. You need to run the testcases and generate the Allure report post-execution. So, for that make sure you don’t run your test cases from the testng.xml file and rather execute it from pom.xml. Because all the plugins related to the allure are been configured in pom.xml and not testing.xml
Before running mvn command (mvn clean test)to execute the testcases, make sure there are no errors in your project otherwise mvn will not build the project and hence your actual script execution won't happen.
Browse till your maven project root directory in your local system(where pom.xml file is present) for which you need to run the testcases and generate Allure reports.
For me its “C:\Users\lenovo\eclipse-workspace\JenkinsDemo”
If you don't have any maven project with you at the moment and you are too eager to see how the Allure report work, you can take a pull from this repo and try your hands on it.
Open the command prompt in that path and run the following command:
mvn clean test
The scripts should run and give some result either fail/pass
To generate Allure reports, on the same terminal, run the following command:
allure serve allure-results
Hit Enter, wait for some time, and Voila…..your default browser will open with the Allure results.
The above command generates a report in the temporary folder from the data found in target/surefire-reports/ and then creates a local Jetty server instance, serves generated report, and opens the HTML report in the default browser.
There are different sections in the Allure report such as Categories, Suites, Graphs, Timeline, etc. Access each link to get detailed information about the run.
As I mentioned earlier that @Description annotation in Allure is different from the description attribute of @Test in TestNG, the following picture explains that.
