Integrating Allure Report as Automation Test Report Framework in TestNG

Introduction

Test report is a document to measures the quality of a product in its development cycle. A test report contains all of the test activities and test results according to the project being developed. It can be used to determine whether a product can go to the next development stage or not.

Automation Test Report

There are so many testing frameworks especially in Java programming language such as JUnit, JBehave, Serenity, TestNG, etc. Most of them have the same functionality and test reporting as JUnit. Currently, for my project in DANA, it’s used TestNG for the testing framework. For TestNG, the default generated report will look like this

source: www.guru99.com

It formed as an HTML report showed the test execution result including the number of tests marked as passed, skipped, and failed with the time taken for the test. The test report only provides minimal information about the test activities and test results. With this kind of report, it relatively difficult to understand except for the person who makes the automation test because we don’t know the exact information what the test is about, also it quite difficult to measure the quality of the product for a project with many test cases because it didn’t show enough information as the summary report.

A Good Automation Test Report?

A good automation test report is subjective, but obviously, we must agree that it should be easy to understand by all project members and give clear information about the test activities and its result so we can decide the quality of the product. To achieve that, a good automation test report must contain at least:

Test Detail

Including what we test, test description, test step to conduct the test, and maybe attachment containing a log or screenshot for each test step

Metrics

Including test execution status (number of test case executed, test marked as failed, test marked as success) and defect distribution. This will help to measures the quality of the product.

Trend

Trend of the test execution test between each test will help to track the quality of the product.

Allure as Automation Test Reporting Framework

Allure is an open-source test reporting framework designed by Yandex QA Team software, providing a bunch of cool features regarding test reporting. Allure can give a good visual representation of test output, including metrics, trend, defects distribution, and also test time execution. During test execution, all information about the test will be “recorded” by listener adapter and saved as XML files, after test execution completed the XML files can be converted as HTML report through command line.

Source: demo.qameta.io

Using Allure Report in TestNG

Since my current project using TestNG as automation testing framework, here is the tutorial to get started integrating allure report with TestNG. First we need to modify pom.xml in our codes:

After that, we can use all of Allure Report features by assigning annotation in our codes. For the full annotation list, you can check the documentation here, but basically, for day to day API testing, there are several annotations that we mostly used :

Display test name method

This annotation useful to give a human-readable test name method by assigning @Description(Test=”example”) or @Title. We used to give the method name and display test name method with the test case name so we can easily recognize what the test is about and the time taken to track the code or do test code maintenance will be more efficient. Here is the example use of the annotation:

And for the test report it will be shown as below :

Step

Any action that takes during the test can be stated with @Step annotation. This will be useful to give more chronological detail about the test activities. We use this annotation to mark activities such as manipulating or getting the database data, making HTTP requests, and also when performing test assertions. Here is the example to use @Step annotation :

Here is what the test report looks when we add @Step annotation:

Attachment

I think it is the coolest feature, you can add an attachment in your tests like a screenshot, log, or anything depends on your need. To use the feature you can add @Attachment annotation in your codes. We currently add this annotation to attach the log of a query statement, request and response from HTTP, and also test assertion. For this attachment, we need to return the data that will be composed as an attachment, for log purposes we can return the string data of the log but for media like screenshot it needs their byte data. Here is the example use of the annotation :

For the test report it will be shown like this :

Running and Generating Allure Test Report

Run your test by executing the command :

mvn clean test

After that, you will see the allure-results directory inside the target project directory which contains XML files that saved test execution information. To generate Allure report simply type the following command :

mvn allure:report

The HTML report then will be generated at target/site/allure-maven-plugin. If you open the report, you will see almost all the graphs inside the HTML page shown, but there is an empty state showed for the trend graph both in Overview and Graph tab.

This caused by empty history data for the previous test, so the trend graph can’t be created. To showing the trend graph, at least we have been running the test two times, after the next test run complete, then the history directory inside the previous allure-report directory must be copied to the current allure-results directory before generating the report. Here is the result for second run test report

Final Words

A test report is an important document to measure the quality of the product. Most of the automation test framework especially in Java provides mechanisms to generate their report but it quite difficult to understand and have minimal information about the test activities and the test result. However, Allure framework can be the perfect tool to generate automation test report, since Allure provides cool features to give more detail in our automation test report and it’s easy to use.