How to publish Automated Test Results in real-time to Azure DevOps Test Plan

5 min readMar 19, 2023

What is Azure DevOps?

Azure DevOps is a cloud-based platform that provides a set of tools for software development, deployment, and testing. It includes a range of services such as version control, build and release management, project tracking, and collaboration.

“Test Plans” is a feature of Azure DevOps that enables teams to manage their testing efforts more efficiently. It provides tools for creating and managing test cases, suites, and plans. Teams can use Test Plans to design and execute manual tests, as well as to automate their testing using frameworks like Selenium.

With Test Plans, teams can track the progress of their testing efforts, manage test results, and generate reports to gain insights into the quality of their software. It also integrates with other Azure DevOps services, allowing teams to link test results with work items and source code changes, and to trigger automated builds and releases based on the outcome of their testing.

Pre-requisites:

Azure DevOps Account: You will need an Azure DevOps account to create and manage test plans. You can create a free account or use an existing one.
Project and Team: You will need to create a project within your Azure DevOps account and set up a team that will be responsible for testing.
Test Plan License: To use Test Plans, you will need a valid license. If you have a Basic or higher level of Azure DevOps subscription, you already have access to Test Plans(There is a 30-day trial version for this as well)
Test Cases: You will need to create test cases that will be executed as part of your test plan. These test cases should be designed to cover all aspects of your application’s functionality.

About Test Automation Framework

For this example, the test automation framework being used for automated testing is built using Selenium WebDriver, Maven, and Java. The framework is designed to use the Behavior Driven Development (BDD) approach, which helps to ensure that the tests are focused on the application’s behavior and the desired outcomes.

ADO Test Point API

To publish our test results we can use the “Test Point — Update” API provided by Azure(documentation)

To use this API you will need to create a Personal Access Token(PAT — refer)

PATCH https://dev.azure.com/{{organization}}/{{project}}/_apis/testplan/Plans/{{planId}}/Suites/{{suiteId}}/TestPoint?api-version=6.0-preview.2URI Parameters:

URI Parameters:

Request Body:

[
    {
        "id": 6,
        "results": {
            "outcome": "passed"
        }
    }
]

The “id” in the request body will be mapped to the test case id of the ADO Test Plan and the “outcome” will be mapped to the test result. There are the possible values for the outcome:

Now we will try to test this API using postman. But first, we need to add the authentication type to postman. This will require “Basic Auth” type and the password should be set with the “Personal Access Token(PAT)” which was generated earlier.

I have added the values for organization, project, planId and suiteId as environment variables:

If everything was done correctly we should get a successful(200 OK) response like this:

We can also confirm that the API is working fine via the ADO Test Plan:

To retrieve all the “id” values(or Test Cases/Test Points) in our Test Plan we can use the “Get Test Point List” API provided by Azure(documentation)

GET https://dev.azure.com/{{organization}}/{{project}}/_apis/testplan/Plans/{{planId}}/Suites/{{suiteId}}/TestPoint?api-version=6.0-preview.2

Now let's check how we can implement this in our Test Automation Framework…

How can we get the “Test Point Id” values for each test?

Since in our framework we are using Cucumber, using cucumber annotations we can simply add the test point id to each test scenario as below:

  @ado_tc_id=6
  Scenario: This test should always pass
    When I execute this test
    Then It should pass

  @ado_tc_id=7
  Scenario: This test should always fail
    When I execute this test
    Then It should fail

In this way, each of our Gherkin test scenarios will have a one-to-one mapping with ADO Test Plan.

How can we get the “Outcome” values for each test?

In Cucumber the After hook is executed after each scenario, and we can use it to capture the outcome of the test scenario. Here is an example:

How can we implement the PATCH API Request?

The “updateTestResultInAzureDevOps” method will have two parameters, the test scenario and the outcome of that scenario. We can use the Rest Assured Library to authenticate and execute the “Test Point — Update” API as below:

NOTE: Please make sure not to commit framework secretes like PAT or other API Tokens to the project repo. Instead either you can store them in an encrypted file using git-crypt or you can pass it as a system property as a maven command at runtime.

After executing our test suite we can verify if the ADO Test Plan was updated with the latest test results of execution.