Showing Quality Metrics in one location
In this blog post, I will describe how to show test coverage metrics in one location using HostedGraphite with Grafana.
At relayr, we build a cloud platform based on microservice architecture. That means we have many independent software projects.
Our customers or investors often ask about the quality of our platform. Surely, they want to know if it’s reliable and maintainable. Although we have quality metrics artefacts available per project, we wanted to have them in one location to see and show a bigger picture.
We started from test coverage for all our microservices. How do we show the test coverage in one place? The standard solution (relayr team likes standard solutions for standard problems) would be to integrate SonarQube or Coveralls.
But. When I was suggesting different tools and gathering requirement, I also collected some insights on why these tools do not fit our needs:
- Focus on something besides test coverage (e.g., static code analysis).
- Lack of support for a given technology (SonarQube doesn’t have an official Scala plugin).
- Requires access to the codebase (e.g. Coveralls).
- The maintenance that comes with self-hosting such a tool.
- Limited budget.
And we though: Send unit test coverage to Graphite and display it in Grafana might work.
Graphite is a data source for Grafana. Any numbers you will send to Graphite, Grafana will show that on a dashboard in form of a graphs or table.
At that moment we already had been using HostedGraphite with Grafana for other things.
Requirements
- Jenkins
- Cobertura test coverage report (standard Cobertura report can be generated for Java, Scala and JS based projects)
- HostedGraphite
Implementation
We use Jenkins Pipeline for continuous integration. Every microservice we build, has a Jenkinsfile with stages that create artifacts and perform different actions. These stages of a pipeline are:
- check out source code
- run unit tests and generate a coverage report
- run static code analysis
- build application
- create Docker image
- ship the image to a Docker Hub
To fulfil our goal, we added a pipeline stage:
- send the test coverage to Graphite
# Jenkinsfilestage('Send Test Coverage To Graphite'){
sh """python coverage_to_graphite.py pretty-service target/scala-2.11/coverage-report/cobertura.xml"""
}
coverage_to_graphite.py requires two arguments, service name and a path to Cobertura report, to build a Graphite metric. The Graphite metric is a sort of an end-point where the data are available for Grafana. Create a cobertura report before this step.
The steps implemented in the Python script are:
- Parse the coverage report — XML file
# coverage_to_graphite.py
from lxml import etree
from io import StringIOtree = etree.parse(StringIO(data)) # data is a unicode string with a content form an xml file
- Get the coverage value
# coverage_to_graphite.pycoverage_value = tree.xpath("number(/coverage/@line-rate)"
- Build a Graphite metric string with a coverage value
# coverage_to_graphite.pymetric = "test.coverage.{}.{} {}".format(pretty-service, master, coverage_value)>> metric
"test.coverage.pretty-service.master 75"
- Send the metric to Graphite
# coverage_to_graphite.py
import requestsrequests.post(url=self.graphite_url,
auth=(self.api_key, ""),
data=metric)
When you are all set and the metric is sent to Graphite it’s time to create a Grafana dashboard. Look here for instructions: Grafana Dashboards.
Every time any project is built, the test coverage value will be sent to Graphite. Grafana will show the test coverage for all the projects in one location.
Conclusions
It was not the most obvious, but the simplest solution that worked for us.
As a team, we became aware by comparison that few of our services have very high test coverage (mostly JS projects), and some of the Scala projects need to be taken care of.
It is possible to create a link to the dashboard for sharing it with team members or any other stakeholders.
