Visualising and Monitoring in Test Automation

In this article, I’ll be sharing how to integrate Prometheus and Grafana for visualising & monitoring the Selenium WeDriver tests.

We will see step-by-step “how to set up monitoring for selenium tests”.

1. Prometheus: Prometheus is an open-source monitoring and alerting tool. It collects data about applications and systems and allows you to visualise the data and issue alerts based on the data. Prometheus collects and stores its metrics as time series data. It works on the Pull Model: Prometheus server pulls metric data from exporters. Agents do not push data to the Prometheus server. Exporter is one of the basic components of the Prometheus server that exposes the data about systems and applications, which is collected by the Prometheus server. A Prometheus server gathers the metrics from different exporters and makes them available. There are a lot of exporters available e.g. Node Exporter, Apache Exporter, JSON exporters etc.

1.1 Configuring Prometheus

There are ways you can configure the Prometheus server. You can use Docker to run the Prometheus server or use pre-compiled binaries. More info can be found here . I will be using binaries to install the Prometheus server and give you a step-by-step installation process. I will be using 3 instances for the complete setup for our monitoring.

1. On the first machine, we will be configuring the Prometheus server
2. On the second machine, we will configure PushGateway (I will talk about this separately)
3. On the third machine, we will have Grafana running (will be talking separately on this)

However, if you want you can use a single machine for the installation. I will be using 3 Ubuntu instances for my configuration. Configuration for each OS is different. I will be using Ubuntu for my entire configuration.

Let’s start configuring our Prometheus server.

1. Create a user and group for Prometheus.

sudo useradd -M -r -s /bin/false prometheus

2. Create directories for Prometheus

sudo mkdir /etc/prometheus /var/lib/prometheus

3. Download and extract the binaries

sudo mkdir /etc/prometheus /var/lib/prometheus
wget https://github.com/prometheus/prometheus/releases/download/ v2.16.0/prometheus-2.16.0.linux-amd64.tar.gz
tar xzf prometheus-2.16.0.linux-amd64.tar.gz prometheus-2.16.0.linux- amd64/

After extracting the binaries you will see something like this

4. Now move the binaries to appropriate locations

4.1 Move prometheus, promtool to /usr/local/bin

sudo cp prometheus-2.16.0.linux-amd64/{prometheus,promtool} /usr/ local/bin/

Now change the ownership of the directories to prometheus user that we have created in previous steps

sudo chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}

4.2 Now copy consoles,console_libraries and prometheus.yml to /etc/prometheus/

sudo cp -r prometheus-2.16.0.linux-amd64/{consoles,console_libraries} /etc/prometheus/
sudo cp prometheus-2.16.0.linux-amd64/prometheus.yml /etc/prometheus/prometheus.yml

4.3 Now change the ownership of the directories to prometheus user

sudo chown -R prometheus:prometheus/etc/prometheus
sudo chown prometheus:prometheus/var/lib/prometheus

5. Now let’s start our prometheus service by running the command

prometheus - config.file=/etc/prometheus/prometheus.yml

You will see something like this in your terminal.

If you get output like this, it means your prometheus server is ready.

6. Now lets create a systemd unit file for Prometheus. Run the below command

sudo vi /etc/systemd/system/prometheus.service

7. Now Copy paste the content to the file

[Unit]
Description=Prometheus Server
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus -- config.file /etc/prometheus/prometheus.yml -- storage.tsdb.path /var/lib/prometheus/ -- web.console.templates=/etc/prometheus/consoles -- web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target

Save changes in vi editor by pressing escape and typing wq

8. Now run the below commands

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

You will see something like this in output.

If yes, Congrats!!! 🥳 You have configured the Prometheus server. You can also test the connection by running.

curl localhost:9090

To access the server in browser you need to hit below url in your browser

http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

In case you are running server locally you can access it by accessing

http://localhost:9090

2. PushGateway

Prometheus works on a pull model. But there are instances where jobs are not running for long or there are batch jobs that run for shorter duration, in those cases Pushgateway can be used to expose their metrics to Prometheus server.

Configuring PushGateway

I will be using a separate machine for Pushgateway but the same configuration can be done on the machine where the Prometheus server is already running. Let’s start configuring our Pushgateway.

1. Create a user and group for Pushgateway:

sudo useradd -M -r -s /bin/false pushgateway

2. Download and install the Pushgateway binary:

wget https://github.com/prometheus/pushgateway/releases/download/ v1.2.0/pushgateway-1.2.0.linux-amd64.tar.gz
tar xvfz pushgateway-1.2.0.linux-amd64.tar.gz

3. Move the binary to appropriate folder and provide the ownership to pushgateway user

sudo cp pushgateway-1.2.0.linux-amd64/pushgateway /usr/local/bin/
sudo chown pushgateway:pushgateway /usr/local/bin/pushgateway

4. Now let’s create the sytemd unit file for Pushgateway

sudo vi /etc/systemd/system/pushgateway.service

Copy paste the below contents to unit file

[Unit]
Description=Pushgateway
Wants=network-online.target
After=network-online.target
[Service]
User=pushgateway
Group=pushgateway
Type=simple
ExecStart=/usr/local/bin/pushgateway
[Install]
WantedBy=multi-user.target

Save changes in vi editor by pressing escape and typing wq

5. Now enable the pushgateway service by running the commands

sudo systemctl enable pushgateway
sudo systemctl start pushgateway
sudo systemctl status pushgateway

6. You can test the connection by running

curl localhost:9091/metrics

7. Now we need to configure the Pushgateway configuration file for Prometheus so that the Prometheus server can collect the data for display.

Edit the Prometheus config:

sudo vi /etc/prometheus/prometheus.yml

Under the scrape_configs section, add a scrape configuration for Pushgateway.

- job_name: 'Pushgateway'
  honor_labels: true
  static_configs:
  - targets: ['<PrivateIPOfPushGatewayServer>:9091']

If you are running Pushgateway on the same server where Prometheus is running you need to provide targets as ‘localhost:9091’. Save changes in vi editor by pressing escape and typing wq

8. Restart the Prometheus service by running the commands

Now open the prometheus server in browser and run the query

pushgateway_build_info

You will see the output like this.

We have configured the Pushgateway successfully. Now let’s push the data to Pushgateway so that Prometheus can display this.

Go to the machine where Pushgatway is installed and run the below command.

echo "total_tests 20" | curl - data-binary @- http://<PublicIPOfPushGatewayServer>:9091/metrics/job/test_job

If you are running Pushgateway service locally on the server where Prometheus is running you need to give IP as localhost

echo "total_tests 20" | curl - data-binary @- http://localhost:9091/metrics/job/test_job

After running the curl command, go to the query browser for prometheus and type total_tests in the query text box. You can open the query browser by navigating to

http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

Or http://localhost:9090 if you are running locally.

Scraping will happen in every 15s as this is defined globally in prometheus.yml which we have configured previously.

You can send multiple metrics at once as well. You can add multiple labels as well.

cat << EOF | curl - data-binary @- http://172.31.127.248:9091/metrics/job/test_job 
failedTests{author="jeeshan"} 1
passedTests{author="jeeshan"} 10
EOF

Now you can run query failedTests and passedTests in the query browser. You will see the output like this.

Setup for the Pushgateway is completed.

3. Grafana

Grafana is a great tool for visualising your Prometheus data. We will cover all the steps required for configuring Grafana with our prometheus server.

Again, here, I am going to use a separate instance for running Grafana. You can use the same machine where your Prometheus server and Pushgateway are running.

To install required packages and download the Grafana repository signing key, run the following commands:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

To add a repository for stable releases, run the following command:

echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

After you add the repository, run the following commands to install the OSS release:

sudo apt-get update
sudo apt-get install grafana

Enable and start the grafana-server service:

sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

You will see the output like this

You can test the connection by running curl localhost:3000

You can also verify Grafana is working by accessing it in a web browser at

http:// <GRAFANA_SERVER_PUBLIC_IP>:3000

Or if you are running locally, you can access it in web browser at

http://localhost:3000

Log in to Grafana with the username admin and password admin. Reset the password when prompted. After that you will land on home page for Grafana

Click Add data source.
Select Prometheus. For the URL, enter

http://<PROMETHEUS_SERVER_PRIVATE_IP>:9090

Click Save & Test. You should see a banner that says Data source is working.

Now it’s time to set up our test by running some PromQL query.

Go to home and click on Create your first Dashboard panel. Click on Add visualisation.

Let’s enter some queries e.g. failedTests, passedTests, totalTests which we have created when we have configured our Pushgateway.

Change the Time Series graph to Stat from the top right corner in Grafana. After this, start adding the query in the query section. You can add multiple queries.

Hurray! We have configured the Grafana successfully. We have configured our Prometheus server, Pushgateway and Grafana successfully. Now we should configure our tests suite to report the execution summary on Grafana. For this, I will be using Selenium Webdriver and TestNG as the testing framework. Our tests will be running with GitHub Actions and the status of test execution will get reported back to Grafana.

Configuring the GitHub Actions

Source code for the tests can be found on GitHub. I have created a reusable workflow for running the tests with Github actions. The source code can be found here.

Understanding the Reusable workflow

Link for workflow:

https://github.com/jeeshan12/resusable_workflows/blob/main/.github/workflows/selenium.yml

In this workflow, I am configuring an Ubuntu instance with Java and Maven. I am also installing the Chrome browser to run the tests. After the execution is done, I am uploading the execution results. Here I am leveraging the testng-results.xml file which is generated by the surefire plugin for us. It will look something like this.

From this file, with the help of shell script, I will read the total tests summary and push it to the PushGateway so that Prometheus can collect the data and Grafana will show these results. Sounds interesting ??? Let’s see how this will happen with Github actions.

Let’s understand our ci.yml file for triggering our tests from the repository

https://github.com/jeeshan12/SeleniumPrometheusIntegration/blob/main/.github/workflows/ci.yaml

Our Github actions will trigger on every push to the branch and whenever a PR is created from the main branch. We have two jobs, the first is test and the other is monitoring.

In test job we are calling our reusable workflow

jeeshan12/resusable_workflows/.github/workflows/selenium.yml@main

More information on how to use reusable workflow can be found here. Our workflow accepts a fileName as a mandatory parameter which we are providing as testng.xml in our case. We will be needing Pushgatway server IP which we have configured in our previous steps. We will set this as secret in our GitHub actions as PUSHGATEWAY_IP.

PUSHGATEWAY_IP: ${{ secrets.PUSHGATEWAY_IP }}

Our tests will be executed in this reusable workflow and tests will be archived as test-results.

In the monitoring job, we are downloading the archived results from the previous step. This job will run after the completion of the test job irrespective of whether it passes or fails. We will read testng-results.xml and send the necessary information about the test execution count to the Pushgateway server which Prometheus uses to scrape and Grafana shows this in Dashboard. There are a wide variety of Dashboards that Grafana provides. You can use whatever fits your requirement. I will be using Stats to show the total test execution count. I am providing the Pushgateway IP as a secret to my actions and exporting it as an environment variable, as this has been used as a variable in the shell script. Our shell script accepts one argument which is the name of the results file i.e. testng-results.xml

You can see the shell script in the repository. This shell script will be pushing selenium_ignored_tests, selenium_total_tests, selenium_total_tests_passed, selenium_total_tests_failed, selenium_total_tests_skipped, selenium_suite_execution_duration metrics to pushgateway server.

Now let’s test our step by pushing any changes to the repo that will trigger execution using GitHub actions and report the results on Grafana.

As you can see both jobs ran successfully. It’s time to switch to the Prometheus server and Grafana to see the results.

I have configured selenium_ignored_tests, selenium_total_tests, selenium_total_tests_passed, selenium_total_tests_failed, selenium_total_tests_skipped as a Stat Dashboard.

I have configured another dashboard for the total suite duration.

There are different varieties of dashboards available for Grafana. You can configure them according to your requirements.

That’s all for this article. Hope this was helpful.

Connect with me on LinkedIn.

Further Reading:

Kafka monitoring: Message brokers observability and troubleshooting

Kubernetes Monitoring with OpenTelemetry

API observability: Leveraging OTel to improve developer experience

Serverless observability, monitoring, and debugging explained

API monitoring vs. observability in microservices

Combining OTel and Prometheus metrics for alerting machine

Lambda monitoring: Combining the three pillars of observability to reduce MTTR

Fargate custom application metrics with Prometheus in CloudWatch