How to Enhance Your Test Automation Reports with Slack Integration and Variable Extraction ?

Before delving into the technical details of extracting values from reporting tools and sending them to Slack via webhooks, it’s important to provide some context about Dolap and the challenges we faced with our test automation reporting.

At Dolap, we had been struggling to streamline our reporting process, as we were using multiple tools and systems to track and report on test results. This led to inefficiencies and inconsistencies in our reporting, making it difficult to get a clear and accurate picture of our test automation progress.

To address these challenges, we explored different options and eventually settled on a solution that involved extracting values from our reporting tools and sending them to Slack using webhooks. In this article, we’ll share our approach and provide step-by-step instructions on how you can implement it in your own company.

We need to use GitLab Pages instead of other methods for several reasons, which are listed below:

• Firstly, we predominantly use GitLab, and GitLab Pages is a familiar for us.
• Additionally, GitLab creates an internal URL that can only be accessed internally, which is beneficial for our security.
• While AWS S3 Bucket is another viable option, creating a bucket and setting credentials for every team can be an unnecessary effort.
• Comparing GitLab Pages to AWS, GitLab is easier to set up and use for multiple teams.

To make reports with minimal effort, we decided to start using GitLab Pages Structure.

• Comparing GitLab Pages to AWS, GitLab is easier to set up and use for multiple teams.

To make reports private with minimal effort, we decided to start using GitLab Pages Structure.

At this point, we are using 2 libraries in our projects. They are JQ and XmlLint.
Mention these shortly, JQ is a command-line json processor and XmlLint is generally used for XML formatting. The purpose of selecting these 2 libraries is that we are using Cucumber Reporting and Allure Reporting.

Allure is creating a report by exporting JSON files so we are using JQ on projects which we use Allure Reporting. Cucumber Reporting is generating HTML files so in this case, we are using XmlLint on that.

Example Of Cucumber Reporting

In this section, we are going to use XmlLint because of Cucumber Reporting. As we said before Cucumber is generating Html files. When we look at that HTML file we are able to see the picture which is added below. When we inspect the code, we are able to see there is a tfoot with the class named total and below it here is a table row (tr) and table data (td). As we can see and know from Cucumber Reporting, total parts style is not changing at all, so we are freely getting values on that part. When we look up from Xpath, we can easily create an Xpath equal to “//tfoot[@class=’total’]/tr[1]/td[10]”.

At this step we have an Xpath, so we can discuss how we are going to use it.

TOTAL=$(echo "cat //tfoot[@class='total']/tr[1]/td[10]" | 
xmllint - html - shell public/overview-features.html | 
sed '/^\/ >/d' | sed 's/<[^>]*.//g')

To review more in detail there are few parameters in XmlLint which is able to convert our HTML file to XML format. With the sed command, we are able to delete unnecessary parts and with the ‘cat’ command we are able to get the related value of Xpath. At the end of it we are using echo to get and set the TOTAL variable. We can simulate the same steps for other variables like PASSED and FAILED.

Example Of Allure Reporting

In this section, we are going to give an example of our other library which is JQ.As we mentioned before Allure is generating json files for reporting.

{
  "reportName": "Allure Report",
  "testRuns": [],
  "statistic": {
    "failed": 16,
    "broken": 2,
    "skipped": 3,
    "passed": 171,
    "unknown": 0,
    "total": 192
  },
  "time": {
    "start": 1666079755805,
    "stop": 1671180009843,
    "duration": 5100254038,
    "minDuration": 1,
    "maxDuration": 61009,
    "sumDuration": 760162
  }
}

Let’s get deeper into JSON files that are generated from Allure Reporting. In this case, our variables are total, passed, failed, broken, and skipped. At this point, we are going to use the line below this paragraph.

TOTAL=$(cat report/widgets/summary.json | jq -r '.statistic.total')

Related command is easy to understand. With the ‘cat’ command we are able to read our file and with the ‘jq -r’ part we are able to get related data from the JSON file. For example, we can see that the passed value is inside of an element named statistic. So at this point, we are telling our data to JQ with typing ‘.statistic.total’ and after it, we are using echo to get and set the related value to TOTAL variable.

We can simulate this step to other variables for PASSED, FAILED, BROKEN, and SKIPPED.

Sending Message To Slack Channel

At the beginning of this part we have multiple options to send a message to slack but we preferred Slack Webhook because this option was easier for us. After creating a webhook we can inspect the json which Slack wants from us.

{ 
  "text": "text",
  "channel": "#channel-name",
  "username": "User name of message sender"
}

In this part Slack wants 3 variables from us. Text part is for the message we want to send, channel is for channel name which we are going to post and username variable.

Here we gained most of the variables to send our message. In our projects we used the curl library to send Slack message. Curl is waiting method type, a data to send and a header if there is any.

First of all let’s begin with the text variable. For projects CI/CD phase we are using GitLab, so we added a variable called RUN_TYPE there. This variable tells us when test runs occurred. For example if there is a scheduled run, this variable will be equal to ‘Scheduled’, if it is run by triggering from another project variable will be equal to ‘After API Deployment’. With this variable we can understand why our tests are run. Because of using GitLab we are able to use the pages feature of it. With pages feature we are able to publish our reports on the web page. When we take all the information that we mentioned in this paragraph, we are using text equal to the value under this part.

"$RUN_TYPE Project Automation test report is generated at 
http://pages.domainName.com/-/projectPath/-/jobs/$CI_JOB_ID/artifacts/
report/index.html \n
*Total : $TOTAL * \n
*Passed : $PASSED * \n
*Failed : $FAILED * \n
*Broken : $BROKEN * \n
Skipped : $SKIP * \n
 *Duration: $MINS minutes and $((SECS%60)) seconds\"

After all, we need to set channel and username value. After setting those values Slack wants data from us so we are setting all to a variable named DATA.

DATA="{\"text\":\"$RUN_TYPE Project Automation test report is generated at 
http://pages.domainName.com/-/projectPath/-/jobs/$CI_JOB_ID/artifacts/report
/index.html \n 
*Total : $TOTAL * \n 
*Passed : $PASSED * \n 
*Failed : $FAILED * \n 
*Broken: $BROKEN * \n 
*Skipped: $SKIP * \n 
*Duration: $MINS minutes and $((SECS%60)) seconds * \",
\"channel\":\"#channel_name\",\"username\":\"GitLab-CI\"}"

After creating the DATA variable we have the last step to create our curl. At this point we are going to use the line here below;

curl -X POST $SLACK_WEBHOOK -H $CONTENT_TYPE - data "$DATA"

In this command line we are telling to curl that we are going to use the POST method. We are using the variables from GitLab which we already set with name SLACK_WEBHOOK, and header with -H parameter with the value is again from GitLab with name CONTENT_TYPE. At the end of it, we are adding our data with parameters –data part.

With all of this information, we are able to send messages to Slack.

Creating Sh File for Projects

At the end of all, with the data we gathered we are creating a sh file. With this sh file we are able to run it at CI/CD stages, so we are able to get message after our test runs. We can find all values that come from this article below of this paragraph.

#!/usr/bin/env sh

TOTAL=$(cat public/widgets/summary.json | jq -r '.statistic.total')
FAILED=$(cat public/widgets/summary.json | jq -r '.statistic.failed')
PASSED=$(cat public/widgets/summary.json | jq -r '.statistic.passed')
BROKEN=$(cat public/widgets/summary.json | jq -r '.statistic.broken')
SKIP=$(cat public/widgets/summary.json | jq -r '.statistic.skipped')
TEMP_DURATION=$(cat report/widgets/summary.json | jq -r '.time.duration')
SECS=$(($TEMP_DURATION/1000))
DATA="{\"text\":\"$RUN_TYPE Project Automation test report is 
generated at http://pages.domainName.com/-/projectPath/-/jobs/$CI_JOB_ID/
artifacts/report/index.html \n 
*Total : $TOTAL * \n 
*Passed : $PASSED * \n 
*Failed : $FAILED * \n 
*Broken : $BROKEN * \n 
*Skipped : $SKIP * 
*Duration: $MINS minutes and $((SECS%60)) seconds * \",
\"channel\":\"#channel\",\"username\":\"username_value\"}"
curl -X POST $SLACK_WEBHOOK -H $CONTENT_TYPE --data "$DATA"

functional-tests:
  stage: functional-tests
  allow_failure: true
  image: "${IMAGE_PATH}"
  variables:
    CONTENT_TYPE: 'Content-Type:application/json'
    SLACK_WEBHOOK: "${SLACK_WEBHOOK_URL}"
  before_script:
    - export GRADLE_USER_HOME=`pwd`/.gradle
  script:
    - sh gradlew :x-test:clean :x:build
    - sh gradlew :x-test:test --continue
  after_script:
    - mkdir report
    - sh gradlew :x-test:allureReport
    - mv x-test/build/reports/allure-report/* report
    - sh x-test/.gitlab/ci/scripts/announceSlack.sh
  artifacts:
    when: always
    paths:
      - build
    expire_in: 1 week

After creating the sh file we can use this file in CI/CD stage like the job we write.

If you are looking for another way to publish reports or making reports public using AWS S3, you can read our next article which has written by my colleague Kerim Can Yücel.

If you want to be part of a team that tries new technologies and want to experience a new challenge every day, come to us.

Trendyol Tech

Dolap Tech