Integrating Robot Framework with Xray

One more step towards the automation of a daily routine

— by Artur Rybak

One of the primary goals of the QA team at Alteos is to minimize the amount of manual work within daily activities in our projects.

Before rolling out a new product version to production, we test our ‘release candidate build’ in a sandbox environment.

In order to define the scope of testing, provide the visibility of testing progress and share testing results with interested parties, we used to create testing artifacts (e.g. test plan) in Jira.

We soon realized how time-consuming it was, to execute test cases manually in Jira. Especially in a fast-paced environment, testing needs to be conducted quickly —

but without neglecting quality.

Source: https://docs.eazybi.com/eazybijira/data-import/data-from-jira-and-apps/xray-test-management

Having achieved a solid automated-test coverage…

…we switched to automated tests execution during the ‘release candidate’ testing phase.

Robot Framework, which we use for test automation, generates well-formatted XML test reports. We thought it would be more efficient to import this report directly into Jira, instead of executing every test manually.

And this is how Xray got into our sights.

What’s Xray anyway?

Essentially, it is a Test Management tool for Jira. It supports every phase of the testing life cycle and allows to manage both manual and automated test cases.

What matters to us, is that Xray provides REST API for easy integration with Continuous Integration tools and automated test frameworks. Also worth mentioning: the detailed documentation and responsive support.

Source: https://www.getxray.app/blog/how-to-integrate-robot-framework-with-xray/

How to import Robot tests report into Xray

There are just a few easy steps — we’re excited to share them with your team and help you save time as well!

This guide applies to the Xray Cloud version, currently in use within our project.

• The REST API base URL for all endpoints is https://xray.cloud.xpand-it.com/atlassian-connect.json.
• Requests to Xray’s REST API require authentication based on an API Key which can be generated in the Xray Global Settings > API Keys page.
• We execute automated tests from GitLab pipelines. The scope of every pipeline is defined by an automated test case tag used to specify which report we would like to import into Xray Jira. Any other environment variables can be used for this purpose, too.

The final bash command in our case looks like this:

if [[ "$ROBOT_TAG" == "Test" ]] && [[ "$environment" == "sandbox"  ]]; then
    XRAY_TOKEN=$(curl -s -H "Content-Type: application/json" -X POST --data '{ "client_id": "'$XRAY_CLIENT_ID'","client_secret": "'$XRAY_CLIENT_SECRET'" }' https://xray.cloud.xpand-it.com/api/v1/authenticate | tr -d '"')
    curl -H "Content-Type: text/xml" -X POST -H "Authorization: Bearer $XRAY_TOKEN" --data @"output.xml" "https://xray.cloud.xpand-it.com/api/v1/import/execution/robot?projectKey=QA"
fi

• For every single report being imported to Xray, a new Jira ticket of type Xray Execution is created.
• Each automated test is imported as the Xray Test issue of type Generic (unstructured).
• Jira Test issue is mapped to Robot test with the Definition automatically generated based on the test name and suite. If you maintain a test and a suite (update names etc.), the same Test will be used on next results import.
• Execution details are available on the Test Run details page.

Data such as test name, labels (Robot tags), execution status (passed or failed), detailed execution logs (with duration and status for every Robot keyword) and failure reason (if test failed) are imported as well.

A pretty good result, considering that not long ago, we had to provide most of those details manually!

And the bottom line is?

Imported automation test results can be added to a Test Plan. Along with manual test executions for visual checks, we can combine everything into the consolidated Test Plan.

As a result, we eliminated the necessity to execute tests and provide execution results manually in Jira. It saves us time and speeds up the release process in general.

Every team member has access to a Test Report which provides better visibility and tracking of the process.

About the author:

Artur is a QA Engineer at Alteos. He started as a cartographer and finally found his passion in quality assurance. As part of the Alteos team, he implements best practices and supports QA processes within the tech department.