An introduction to Azure Devops Test environment

9 min readOct 26, 2023

Azure Test Plans

Azure Test Plans provide the ability to quickly and thoroughly test and publish manual and exploratory test scripts for Azure.

Test Plans is an Azure DevOps tool that allows customers to integrate a cloud-based testing platform to manage all their testing needs, including:

Scheduling manual tests.
User Acceptance Testing (UAT).
Observational research.
Stakeholder input.

Users can use Azure Test Plans to develop test plans and test cases within a pipeline.

This can be integrated with Azure Boards to build a test run from the Kanban boards and jointly plan and develop tests.

Test Plans help create user acceptance testing (UAT) plans and assign people from DevOps systems.

It also supports the Test and Feedback browser extension, which allows interested parties to perform exploratory testing without external technologies.

Additionally, Test Plans allow users to test on any platform while maintaining full traceability and robust data collection to diagnose and resolve any issues discovered.

It is the only Azure DevOps service with a free tier due to its extensive toolkit that is only available to commercial users.

Azure Artifacts

Azure Artifacts makes it possible to exchange code between teams and organizations. Packages can be shared and added to pipelines.

This is the Azure DevOps Artifact Library service, which allows building, storing, and sharing packages (development artifacts).

Azure Artifacts allows users to integrate full-featured package management capabilities into CI/CD pipelines.

Additionally, Azure Artifacts allows users to manage different package types, such as npm, Maven, and others, and keep them organized in a single library relevant only to the project they are working on.

#06 Extensions Marketplace Finally, Azure DevOps has an Extensions Marketplace with over a thousand extensions, including Slack, Jenkins, Docker, and Kubernetes, that you can add to your Azure DevOps setup.

Azure DevOps also has a best-in-class cloud-based DevOps service that provides a powerful and comprehensive toolkit for building and managing a comprehensive DevOps process. It offers users the ability to:

Adaptable to any DevOps requirement, regardless of programming language, technology or platform.
Deploy anywhere, from containers to external clouds.

All this is now possible with Azure DevOps, which provides unparalleled scalability and availability without the burden of maintaining separate software for different DevOps operations.

Benefits of Azure DevOps Azure DevOps allows users to write, deploy, and monitor code without accessing many different interfaces. You can manage all this from one view and increase customer convenience.

Automated Testing with Azure DevOps

Using automated tests, such as security and compliance tests, helps identify issues during the testing phase. This allows resources to be provisioned quickly and the entire production environment to be configured quickly.

Continuous Integration & Continuous Delivery (CI/CD)

When code is written specifically for a particular task, it is automatically built and checked for errors, detecting defects early. This allows companies to deploy quickly and consistently into the production environment at any time.

Any Platform, Any Language It supports different platforms and runs on different frameworks. Developers using Java, Node, PHP, .NET and Python can easily work with it.

App Insights

Azure DevOps Azure Application Insights delivers insights through application performance management and rapid analysis. Additionally, you can monitor infrastructure health with Azure Log Analytics and Azure Monitor.

How to use Azure DevOps (Automated Testing with Azure DevOps) Test cases in your test plans can be automated and executed directly from Azure Test Plans. Here are some benefits of automated testing:

A simple method for testers who are unfamiliar with running tests in Build or Release procedures.
The ability to run specific tests on demand instead of scheduled tests in Build or Release workflows that run all tests that meet the filter criteria.
The ability to rerun some tests that failed due to testing infrastructure issues, or when you have a new build with patches for failed tests.

Requirements

Here are some of the prerequisites you need to have before you can get started with automated testing with Azure DevOps:

You have to participate in a project. Create a project if you don’t have one yet.
You must have a project added to you, or you must be a member of a project or team.
You must have at least Basic access or higher to view or run manual or automated tests.
A test plan is a collection of automated tests that you have associated with automated test methods in Visual Studio 2017, Visual Studio 2015, or earlier.
A build process that generates test binaries in the builds.
A program to evaluate. You can use the app for on-demand testing and deployment as part of the build and release process.
Create and manage releases, modify a release environment, and manage deployment with the necessary permissions.

Setting up the Azure Test Environment

Here are all the steps you need to follow to accurately set up the test environment for automated testing using Azure DevOps:

Choose your test plan on the Test Plans page, open the shortcut menu and select Test plan settings.
Select the build pipeline that generates builds with the test binaries in the Test plan settings window. You can then test a specific build number or have the system automatically use the most recent build when tests are run.
To run test plans in Azure Test Plans, you need a release pipeline built using the Run automated tests from the Test Manager template. If you have already built a release pipeline with this template, select it and choose the stage in the pipeline where the tests will run. Otherwise, select the ‘Create a new link’ option to create a new release pipeline with only one stage and the Visual Studio Test task already included in the dialog.
If necessary, give the release pipeline and stage meaningful names.
You must install the Visual Studio Test Platform on the agent machine. You can skip this step if you already have Visual Studio installed on the agent PC. If not, you’ll need to add the Visual Studio Test Platform Installer task to your pipeline design.
Add and configure the Visual Studio Test job in the release pipeline.

Here are the steps you can follow:

The version number of the task settings panel is displayed in the drop-down list at the top left.
Make sure you select the ‘Test run’ option in the ‘Select tests using’ section.
Choose ‘Installed by Tools Installer’ to select the version of Testplatform.
When running UI tests with physical browsers, make sure the agent is set to run as an interactive process with auto-login enabled. Before you queue a build or release, you must first set up an interactive agent.
The interactive process configuration is not necessary when running UI tests on a headless browser.
Choose how the test platform is delivered, the version of Visual Studio, and the placement of the test platform on the test machines.
Select the appropriate settings to extract the file from the build artifacts if your tests require input parameters such as app URLs or database connection strings. If you don’t include the settings file in the artifacts, you can use the Publish Build Artifacts task in your build process to publish it to a drop. The application URL is available in the run settings file in the example below. You can override this by applying the ‘Override test run parameters’ setting to set it to a staging URL.
Select the Agent job item and verify that the deployment queue is located on the one containing the machines running the tests.
On the release pipeline ‘Pipeline’ page, verify that the build pipeline containing the test binaries is linked to the release pipeline as an artifact source.
You need to save the release pipeline.
Return to the browser page with your test plan settings if you created a new one in the Test Plan Settings dialog box in step 2 of this example. Select the release pipeline and stage you just saved in the Test plan settings window.

Running Automated Tests

To run the automated tests with Azure DevOps, follow the steps below:

Open the test plan on the Test Plans website and choose a test suite that contains the automated tests.
After selecting the test(s) you want to run, choose ‘Run Test’ from the ‘Run’ menu.
Select ‘OK’ to start the testing process. The system checks the stage to ensure that the Visual Studio Test job is present and has valid parameters, checks the user’s permission to create a release for the chosen release pipeline, creates a test run, and finally initiates the creation of a release to the selected stage.
Choose ‘View Test Runs’ to view progress and investigate failed tests. The error message, console logs, stack trace, and attachments are all shown in the test results, which you can use to debug failed tests.
Once the test run is complete, the test results will appear in the ‘Runs’ tab of the Azure Test Plans. The Run summary page provides an overview of the run. Additionally, there is a link to the release used to run the tests, making it easy to find the release that ran the tests if you want to analyze the findings later. You can also use this link to open the release and view the release logs.
The results for each test in the test run appear on the ‘Test Results’ page. For failing tests, you can select a test to see debug information such as the error message, stack trace, console logs, and attachments.
If your tests change after test execution, go to the ‘Test Plans’ tab and select the test plan to view the status of your tests. Select a test to get the most recent test results.

Frequently Asked Questions

Q1: Why do you run tests in release stages?

A1: Azure Pipelines provides an effective orchestration methodology to receive test binaries as artifacts and execute tests. This workflow uses the same concepts as the planned testing workflow, making it easy to customize for users already running tests in the expected workflow. For example, by cloning an existing planned test release pipeline.

Another important advantage is the extensive number of tasks in the task catalog, which allows various operations to be performed before and after running tests. Examples include preparing and cleaning up test data, generating and cleaning up configuration files, and other tasks.

Q2: How does the “Test run” option of Visual Studio Test version 2 task work?

A2: The test run object is used by the Test Management subsystem to pass the list of tests selected for execution.

The test task searches for the test run identifier, retrieves test execution information such as container and test method names, runs the tests, updates the test run results, and assigns test points to the test results in the test run.

The Visual Studio job provides a trace from previous releases and test run identifiers to the tests submitted for on-demand test execution from an audit standpoint.

Q3: Is it better to run the agent in interactive mode or as a service?

A3: To enable the agent to launch a web browser when running UI tests such as coded UI or Selenium tests, the agent must be running on the test machines in interactive mode with auto-login enabled, not as a service.

The agent can be used as a service or in interactive mode if you use a headless browser such as PhantomJS. See Agent pools, Deploy an agent on Windows, and Build and release agents.

Q4: I already have a test release pipeline in place. Is it possible to use the same pipeline for on-demand testing, or do I need to design a new pipeline as described above?

A4: For on-demand automated testing, we recommend using a different release pipeline and stage than Azure Test Plans because:

You may not want to deploy the app every time you want to run some on-demand tests. Instead, the default configuration is for scheduled testing stages to install the product and run tests.
New releases are triggered for each on-demand run. If a large number of testers perform a few on-demand test runs every day, your planned test release pipeline can become overloaded. It can also be difficult to find releases that have been triggered for the pipeline that includes planned testing and deployment to production.
You may want to use a Test run identifier as input to the Visual Studio Test job so you can determine what caused the release.

Q5: My team has several testers. Can they use the same release pipeline to run tests from different test suites or test plans in parallel?

A5: They can use the same release pipeline to run multiple tests in parallel if the following conditions are met:

The stage’s agent pool contains enough agents to handle multiple requests simultaneously. You can still start the runs if there are not enough agents available. However, releases will be queued for processing until more agents are available.
You have enough jobs to allow simultaneous work. For more details about parallel jobs, see Azure Pipelines or TFS.
Depending on the execution order, you cannot run the same tests at the same time. This may overwrite the current results.
You need to set the Pipelines stage trigger option for the behavior when there are multiple releases waiting to be deployed, as described above, to allow multiple different test runs to run in parallel.

Conclusion

Overall, Azure DevOps is one of Azure’s newer offerings that allows development teams to collaborate and manage their assignments and releases.

And now you have undoubtedly learned a lot about Azure DevOps in a detailed way; Now you can explain it to everyone and start using Azure DevOps services yourself!

Azure DevOps and the cloud ecosystem are a dynamic and evolving area. Make sure you stay up to date with the latest developments in Azure DevOps coming soon.