As part of Dailymotion’s DevOps initiatives, we decided to move from Jenkins to Jenkins X to be faster in our release cycle and to further empower our teams. This article talks about how we migrated our Allure test reports using Jenkins X.
To accelerate our release process we started adopting CI/CD best practices, one of which was to implement Jenkins X and Kubernetes in our CI pipeline. We believe another key value is sharing information across different teams. To share and communicate about the result and coverage of our automated tests, we decided to use Allure, as it is a tool that produces great reports from your automation codebase. The challenge was to plug Allure with Jenkins X.
Integrating Allure in a testing project to generate great reports
One of our UI component integration tests is using Java, Maven, Selenium and TestNG. We decided to add an extra layer with Allure to produce nice-looking reports.
There’s no reason to reinvent the wheel when it comes to integrating Allure into your project. Just follow the instructions from the documentation. Allure has a plugin that integrates well with Maven and TestNG, so that when you execute your Maven command
mvn test, Allure will automatically produce some files to build a report.
As we were using Jenkins, we wanted the report to be directly integrated into the Jenkins build. The Allure plugin also makes this part very easy — simply follow the guide to install it on Jenkins, let the magic happen, and appreciate your first nice-looking report!
OK, now it’s time to take a look at the serious stuff and move to Jenkins X. Our current setup is using “Static Jenkins” in Jenkins X and our pipelines are defined using the Jenkins Declarative Pipeline DSL. A pipeline defines what stages and steps will be executed during the build. One step of our pipeline is to execute our test in a Maven container. We execute the Allure plugin in a
post step of the
stage. Here is an example of a Jenkinsfile:
This is the theory, but in practice, some complications arose and made it more complex.
Issues we faced (plugin installation, security context…)
The first issue we faced was installing the Allure plugin on our Jenkins X server. Installing the plugin manually through Jenkins UI is easy, but as mentioned in our previous article in the section: the Gitops applied to Jenkins X, the Helm chart used to install Jenkins has a startup script that resets the Jenkins configuration with values coming from a Kubernetes
ConfigMap. You’ll need to update this
ConfigMap by adding the Allure plugin command-line installation XML. This way, when the startup script runs, it will always reinstall the Allure plugin.
If you manually install the Allure plugin on your Jenkins pod, you can then rsh in your pod (
jx rsh <jenkins-pod-name> -n <jenkinsx-namespace>) and find the XML file for the Allure plugin in
/var/jenkins_config/. You’ll need to add the content of this XML file in your
ConfigMap used by Helm chart to install Jenkins.
It will look like this:
We now need to override the configuration for the plugin in the
apply_config.sh part of the
rm -rf /var/jenkins_home/ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation.xml
ln -s /var/jenkins_config/ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation.xml /var/jenkins_home/ru.yandex.qatools.allure.jenkins.tools.AllureCommandlineInstallation.xml
This means that the plugin will always be installed when the startup script will be executed. The second issue we faced was the file system permissions on the Allure files. When the step in the Integration tests stage is executed in the Maven container, the default user is root. It means that all the result files generated by Allure were created by the root user.
The Allure script to generate the report is executed in the default container jnlp. The default user for this container is jenkins. As the containers in the pod share the same volumes, when the Allure script tries to write the Allure report files, it gets an access denied error. To solve this, we had to change the default user of the Maven container to jenkins user. To achieve this, we need to add the following parameters to the container description:
Where 1000 is the UID of the jenkins user in the container.
Here is how the final Jenkinsfile looks like:
The test report can be found in the Jenkins build (in classic view, Blue Ocean is not yet supported):
Allure will take care of marking the Jenkins build as unstable if there are test failures.
Here is where you can find a demo report to see how an Allure report looks like: https://demo.qameta.io/allure/
Extending to all projects
Now that we have integrated this reporting tool in a Java project built in Jenkins X, the goal is to extend it to all our projects. They use different technologies and languages. Some languages are supported by Allure, like Python or JS, but some, such as Golang, are not. We started to develop an open-source library that allows us to integrate Allure in our code and generate Allure results for Go projects. Here is the link: https://github.com/dailymotion/allure-go
Having a nice-looking report is not the only benefit we get from Allure. It allows us to establish better communication between the testers, developers and Product Owners. In a TDD approach, it allows the Product Owner to directly review the test script logic without having to know how to read code.
Our testers are involved in the design and development phases in an earlier stage, the developers have better visibility on the tests and are much more involved in Quality. It also helps to investigate quicker, as we don’t need to search for an error in hundreds of lines of logs. The steps are directly highlighting the errors and pasting the error stack trace in the report. We also use its features and stats to get rid of our test documentation and to estimate the coverage of our functional tests.