Test Automation Framework · Reporting & Observability (Part 4/5)
Integration with a Test Management Tool is essential to link the test plan, test execution and test results with the automated testing execution. This integration not only improves visibility, agility and transparency throughout the testing process but also opens the door for better analysis and data-driven decisions making, increasing the whole Observability process.
Normally, as part of the development and testing cycle, test plans are created per sprint that link the test cases to be executed to the user stories to be validated. The team executes both manual and automated tests and updates the result. So using a test management tool is always better and more productive than any other approach.
There are several test management tools on the market. Some are open source with only limited features, others are licensed tools with extensive support and integration with the agile project management tool like Jira.
As one of the recommended tools, Xray is perfect for handling test management activities and integrates with JIRA as a plugin. It has various features like test plans, test sets, test cases, test execution and test reports with various metrics and dashboards. Xray also has its own API through which you can perform various operations to reflect the results of the automated tests.
As a precondition for having the test case properly identified in the context of an execution process, the following elements are required to be created and set in the test management tool: Test Plan, Test Execution and Test Status.
Each automated test must contain the Xray test case ID as metadata (same as using any other test management tool). This test case ID is always represented as the ID of the ticket.
Below is an example of the test case with the corresponding ID TEST-1234 and the relationship to the automated tests, with the reference to the ID added as metadata as @ TEST-1234
- Test Plan
Creating a test plan per sprint ensures multiple test executions related to it, rather than multiple test executions in isolation and unrelated.
The creation can be manual or automated, but always consider the reuse of the test plan ID per team and per sprint.
2. Test Execution
Once a test plan is created, one or more test executions can be assigned to it at various stages of the testing cycle. However, the test execution can be created on demand without any relationship to the test plan, so the ID may not be required.
The test execution has relevant information such as environment, tags, owner, and any other representation of the test execution itself.
The test execution contains all executed tests with their details and, more importantly, their status.
3. Test Status
Each test case has relevant information and both mandatory and optional fields, but its status is the one that is updated when the test is executed.
The status can contain default values or custom values (depending on the JIRA project configuration). In some cases updating the status is a direct operation, in other cases, some transitions are required.
As part of test execution, each test can be updated from Unexecuted to Passed, Failed, Canceled, Executed, etc.
Additional operations can be performed through specific endpoints, such as adding comments to tickets, linking test results to open defects, and providing additional information as attachments in the test execution details.
4. Creating Defects
When a test fails, the automatic defect creation reduces manual effort and complements the integration with the test management tool. Having the test case ID and all the necessary information about the defect, the defect is created using the same API for creating test plans, test executions and updating the results, but using a different endpoint. This way, not only the test results are updated per execution, but also the defects:
- Update an existing defect by retrieving (via the API) the defect associated with the test case and adding a comment (the defect still happens) or closing it with a comment as well.
- Create new defects with information about the test failure.
Finally, these are some of the most important or relevant operations for integrating test automation with test management tools, but as testing evolves and observability increases, more operations can be included.
References
XRAY API:
