Automated Testing and Monitoring in CI/CD: Implementing Grafana for Enhanced DevOps Data-Driven

In the DevOps lifecycle, the CI/CD is fundamental and represents a culture and process around constantly integrating new code. It can help decrease complexity, increase efficiency, and streamline workflows.

During the ‘test’ stage, it runs unit tests and functionality tests that are created by Test Engineers such as UI automation testing, API automation testing, or both. But, during the ‘monitor’ stage, Test Engineers only check the ‘test’ stage every time the pipeline runs on many microservices one by one. So, how can improve this ‘monitor’ stage?

CI/CD lifecycle

We want to create a dashboard for monitoring all ‘test’ stage runs on all microservices to analyze and make the correct decision.

In this case, we will use the implementation of API automation testing as an example. We used Ruby, Capybara, and Cucumber for this project here.
If you want to test it, make sure you’ve already implemented automation testing. For language programming and the framework, you can use whatever you want or develop legacy code in your company.

To run this successfully, you also need:

• Success rate threshold
• Database in Google Cloud Platform
• CI job
• Grafana

Success Rate Threshold

Every time we run API automation testing, we can generate a report in JSON or HTML format. We use a JSON report to fetch and parse it into a lot of data that can be used in Grafana and calculate success scenario data into the success rate threshold. If the success scenario is less than the threshold we initiated, the job will fail.

Database

Create a Database in the Google Cloud Platform with the specification you want.

Database info in Google Cloud Platform

The next step is to collect the data from the JSON report and then create a table structure.

Table structure

Insert data that become variable from success_rate_threshold.rb into the table.

CI Job

In this step, we will handle running time (running time = end time — start time) and also run file grafana_report.rb.

From the file grafana_report.rb line 156, if the success rate is less than the success rate threshold, the job is set to fail (exit code 1).

Job failed (exit 1) because of success rate less than threshold

Grafana

Why do we prefer Grafana to other monitoring tools? Simply because Grafana has a simple and intuitive user interface. Grafana makes it easy to create visualization, dashboard, and alert. It’s also a powerful query editor to visualize data.

Grafana also supports various data sources, including InfluxDB, AWS, and Elasticsearch. It allows users to customize and tailor visualization while supporting encryption and secure data transfer protocols to ensure users’ data remains secure.

Create dashboard
Go to dashboard menu > create new folder > create dashboard

Create variable
Setting > Variable
The variable will be used to change the microservice name and environment.

Add panel
Add panel > Add new panel > [type of panel]
You can add new panels based on your data query to show in the dashboard.

Grafana dashboard: API Automation for microservice-name

Conclusion

From this Grafana dashboard API automation testing reports, Engineers can:

1. Do reporting and alerting
   Generate periodic reports about the status of the network or individual endpoints to track network patterns over time and use that data to support capacity planning.
2. Do behavioural analysis
   Detecting anomalies and outliers deviates from the general pattern in data sets to distort predicted outcomes and create vulnerabilities.
3. Have network transparency
   To better understand what is happening throughout their network.
4. Maintainable
   View errors even though from the automation script, can be fixed quickly.