Test Automation Framework · Reporting & Observability (Part 3/5)
Reporting is a reactive activity and information must be presented with complete details, solid error evidence, and full traceability of the resolution of each test activity.
Alerting is also a reactive activity that is powerful to notify stakeholders and hold them accountable for any test situation or action, such as test results, deterioration in test coverage, increase in testing times, recommended optimization tasks based on historical information, and more.
Why are reporting and alerting important to Observability? Both approaches improve the way test results are communicated to the teams and reduce the time required to analyze results and troubleshoot system errors by quickly diagnosing the root causes of failures. Teams are able to access the information immediately after test execution is complete, with no manual activities required other than clicking on the message and reviewing the details.
Reporting and alerting are fundamental aspects for informing and sharing test execution progress and results, increasing the observability of test automation processes.
Both functionalities use the previously collected test coverage information, but the outputs are different. Reporting provides an HTML test report with general test information and test details. Alerting notifies you of relevant test information via a dedicated messaging system for immediate awareness and resolution.
As described in the diagram above, the HTML test report is generated and attached to the test results notification. However, it can also be shared via email, integrated into the test management tool, or published in the CI tool. All solutions are valid and complement each other.
Test Report · Real-life example
The HTML test report is generated to provide both a comprehensive overview and detailed information about each test, such as the test steps, API call errors, error messages and full stack traces, screenshots or other additional information to support the analysis.
There are two approaches for creating reports:
- Create a custom report with a specific model and CSS styling.
- Use any open source tool that provides an extensible and flexible API to configure report creation, such as extentreports or allure-report.
ExtentReports provides support for creating multiple report types and a custom API that lets you implement extensions. ExtentReports is a logger-style report library for automated testing. A logger is simply an object for logging messages or events for a particular system or application. ExtentReports uses the logger style to add information about test sessions, such as creating tests, adding screenshots, assigning tags, and adding events or sequences of steps to sequentially display the flow of test steps.
Below you can see how to create a test with 1 running log:
Below are some examples of test reports for Web and API tests.
Alerting · Real-life example
Integrating with a messaging tool is nothing more than sending a specific notification or alert to a channel or group of people using the desired platform. So what are the messaging tools? It depends on the tools that are available to the project and also the testing infrastructure that supports it. However, messaging tools can be divided into at least two categories:
- Email: An email is sent to a specific distribution list and contains information about the test execution, its details and the HTML report as an attachment.
- By alert via chatbots: a chatbot sends a custom notification with a specific structure and relevant information. Reports can also be attached as part of the notification.
In both categories, there is an entity responsible for sending the message and the message itself. The message must be as useful as possible to increase the visibility of the tests and reduce the time invested in the analysis.
- The entity responsible for sending a message supports a specific communication protocol and contract, as well as specific permission on how data is manipulated, transmitted, and shared.
- The message must contain as much detail and audit evidence as possible to provide a clear and accurate view to those responsible for analyzing and evaluating the results, and ultimately to perform error correction.
Slack is one of the most used messaging tools on the market. It is a perfect communication tool and is also great for sending alerts and custom notifications using a bot.
1. Create custom messages
One or more notifications can be created, depending on the visibility and details provided. In this case, two messages are created, one as the main message to open the conversation, and another to continue the conversation in the same message but with full details.
The main message opens the conversation by providing high-level information such as a representative name for the execution, environment, tags, version, and overall success rate with details about the number of tests passed, tests failed, and other test statuses.
The secondary message or message in a thread as part of the main message continues the conversation by providing additional details, attaching the HTML test report and grouping the test results by the failure cause.
2. Send notifications
First, a BOT is useful for centralizing message configuration and permissions (what the BOT should do and for whom) and avoiding the use of personal accounts. Using a BOT also looks nicer and more elegant.
What is a bot? A bot is a type of Slack app designed to interact with channels and users who open a conversation. A bot is the same as a regular app: it can access the same APIs and do all the magical things a Slack app can do. But when you create a bot for your Slack app, you give that app a face, a name, and a personality, and encourage users to talk to it.
Once the BOT is ready, it’s time to send the messages through the Slack API.
Slack provides a robust API to perform various operations over channels, over a particular conversation, and most importantly, over the configuration of the message and its parts.
Importantly, one of the mandatory fields for API calls is the token that you can access through the Slack bot app. Each app has tokens and permissions. Once the app is configured and the notification is ready, it can be published, and also through an already opened conversation.
Below is an example of sending a custom notification, as test results, using a Slack bot for alerting.
References
ExtentReport
Slack API
