Test Management Process with XRAY Transformation at TAV Technologies
The importance of using test management tools has become an essential part of the software development process for many years. It is an indisputable fact that test management tools facilitate testing process in many companies. There are a number of tools for test management whose pros and cons differ from each other, therefore it is very important to select the most appropriate one for your testing requirements.
We, as TAV Technologies, have started to implement Xray test management tool for our testing activities recently. In this article, we will explain the fundamentals of Xray, and how we design our test processes using Xray in our organization.
What is XRAY?
Xray is a Quality Assurance and Test Management Jira plug-in which manages testing activities and therefore ensures high-quality testing of software applications. Xray helps users plan, design, execute and monitor test cases in a simple and efficient way for regardless of complexity of the project.
How did we start XRAY transformation?
Before using Xray, another Jira plug-in test management tool called Qlity was being used in our company, however Qlity hasn’t fulfilled our expectations in test management entirely. After Xray is integrated with our projects in Jira, it was implemented within a few weeks after the following steps:
1. Creating test scenarios for each product
First we, as QA Team, determined our test scenarios for each product which was also the design process of test management. Defining Smoke and Regression Test Scenarios was the first thing we needed to focus on, so we started to prepare test cases on module and page basis.
2. Holding a workshop for managing tests in XRAY
We have organized a one-day workshop in TAV Headquarter as QA Team in order to deep dive on Xray tool features. In the first half of the workshop, we discussed about the ideas of how tests should be planned, designed and monitored in accordance with our daily testing activities. In the second half of the workshop, we have started to create our Smoke test scenarios ,which we have determined before, on XRAY tool.
3. Planning and managing test scenarios in XRAY
Since Xray is a Jira plug-in, All Xray issues use Jira issue types which facilitates the usage of Xray for users that are familiar with Jira. An Xray issue can be simply created just like a story or a task is created in Jira. Various types of Xray issues exist which are used to manage test proccesses.
We started to create test scenarios related with the modules of product. Each scenario can be created with “Xray Test” type issue which is very similar to create any issue in Jira like a story or task.
We have written all the test steps related with the test scenarios which would be followed during test execution. Action to be taken, necessary data in order to run the test and the expected result of the steps are entered in steps in order to clarify the scenario.
Test steps can be easily created, cloned, moved or deleted which are remarkable features of the Xray.
Test scenarios already written in the project can be called in a scenario so that, user does not have to re-enter all the steps in order to run the test. “Call test” option is used for this purpose. We did use this method frequently when creation our test scenarios.
One of the most significant properties that we have mostly used in Xray was that any step can be imported among json, test, csv etc. which dramatically simplifies test case writing and saves a huge amount of time.
We have defined preconditions in a test scenario which are the conditions need to be performed in order to execute the test.
4. Organizing tests by using Test Sets
The first priority of the test management was to plan which type of tests will be executed for the quality assurance of the product. First, we planned the test types under which the test scenarios to be executed. First, we have created different test sets starting from Smoke Test Sets which include the most critical scenarios in our products and Regression Tests which would be run to ensure that the new developed features don’t impact existing functions of the product. In Xray, Test Sets are created in order to organize test scenarios for this purpose.
We determined our test sets which simplifies test execution process that instead of adding scenarios one-by-one, we just call test sets in order to run the tests.
5. Creating Test Repository
Test Repository enables users to organize test scenarios as folder and sub-folder structure. All tests created are displayed in test repository by default. We created our test repository for each product including all tests related with modules, pages and flows. Organizing tests as test sets and test repository facilitates finding the related tests that should be executed in projects or sprints.
6. Test Planning
Test planning provides us a better view about what should be tested within a period of time or scope. Test progress and results can be easily monitored with the help of test planning processes.
We have created a test plan for each and every sprint in our projects in order to plan our test activities in an efficient way.
Test plans consist of tests and test executions that are associated with the requirements. Test executions and/or test cases can be added into the test plan simply with “Add Test Executions” and “Add Test” functions. Test sets can be also added so that all tests defined under a specific test set are added in a bulk action.
When a test execution is added into the test plan, the related tests under the test execution is automatically added to the test plan without any need to add tests one-by-one.
Test plans also enable to track and access the linked issues (stories, bugs, storybugs, subtasks etc.) associated with the tests as a whole.
7. Running the tests with Test Executions
Test Execution is the process of running the tests by verifying that each step in the test scenario works as expected in the scope of the requirement.
After the tests and test steps regarding to each test are completed; test executions are created and linked to the relevant tests. One or multiple tests can be run within a single test execution, and a collection of test executions forms the test plan.
When a step becomes failed, defects regarding to a step can be linked to the step during an execution by either creating a sub-defect or adding an existing defect created before.
Creating a sub-defect provides all the steps to be automatically inserted into sub-defect description so that test scenario can be easily reproduced by the development team in order to fix the bug.
If the bug has already been created for the related step, it would be enough to choose the associated defect or multiple defects with “Add Defect” function.
Sub-test executions are considered as a sub task and created in the scope of a specific story/task issue. We have created a new sub-test execution for each and every story & task issues in the sprint so that test runs and results can be monitored related with the requirement on the Agile board.
Creating a “Test Execution” (not a sub-test execution) seems to be a better way for us to organize other issues except planned stories or subtasks in the sprint which are also needed to be run in the context of a bug or task. “Test Execution” is represented as an independent issue on Agile board.
Here is a very useful function that Xray provides: While executing tests or right after the execution, a test scenario may be needed to be updated with a changing requirement. When a test step is added, deleted or updated in a test and test execution is reopened, Xray offers merging or resetting the existing test definition.
· If reset option is selected, all the execution is reset and the current execution data is deleted.
· If merge option is selected, all the updates in the test scenario are reflected automatically in the existing execution.
8. Tracking and managing the test progress
Test analysis is an essential part of testing processes in order to identify that the tests are conducted in an efficient way. We can keep the track of test results with the built-in report function of Xray.
Test Coverage Report
Test coverage is a measurement technique that defines whether test cases cover the issue(story, task etc.) as expected in context of requirements. It facilitates to observe the status of tests at a glance for a given period of time and scope.
In Xray, Test Coverage Report can be accessible through “Testing Board” menu in Jira for the related project. A graphical representation shows status of tests for a given criteria.
Different coverage status exist displayed as below:
OK: All tests related with the issue are successfully PASSED.
NOK: There is at least one unpassed test, so the test related with the issue is FAILED.
NOTRUN: There is at least one test having TO-DO or ABORTED status, also having no FAILED status or even the test has not been executed yet.
UNKNOWN: There is at least one test having UNKNOWN status, also having no FAILED status
UNCOVERED: The issue has not been covered yet, no tests are prepared associated with the issue.
In XRAY, we observe our test coverage for each test plan that we have created in the sprint by simply selecting a specific test plan and sprint from the criteria.
When clicked on a box in the graph, the related issues of a selected status are displayed as a list so that we can easily track which issues are covered at which status, and which are uncovered with coverage percentage values and number of issues.
Traceability
Traceability report enables to monitor the progress through requirement(issue), tests, test runs and defects within a given project or a set of criteria. Traceability helps to monitor the related executions regarding to a test associated with the issue and analyze the related defects of each test with their Jira status(closed, validated etc.)
In Xray, Traceability Report can be accessible through “Testing Board” menu in Jira for the related project. We generally use Traceability Report with various filters that “Project”, “Test plan” and “Sprint” are the most commonly used filters by our team.
The relevant tests and test runs of each requirement are displayed according to the filters applied. The remarkable property of traceability report is that, the requirements having multiple test runs can be monitored. So that, the root cause of multiple runs may lead to a requirement of writing test automation for such issues.
9. Continuous improvement of test processes
We are aware of that it is crucial to keep our test repository up-to-date since new features or changes are developed in every sprint throughout the project. New test plans, test sets and test scenarios will be created as well as existing tests need to be updated constantly. Furthermore, we have been doing brainstorming sessions within our QA team periodically about how we can improve our test management processes day-by-day in order to increase efficiency because;
“Quality is never an accident; it is always the result of intelligent efforts.”
John Ruskin
Aycan Aydın
