Azure Test Plans: Test Plans

This is my second article on Azure Test Plans, the first one concentrated on best practices for Test Cases. In this article I will show you how I add those test cases to a test plan for execution.

In todays DevOps/Agile development world hopefully most of your tests are automated and running with your Build and Release pipeline. If not you may be doing a fair bit of manual testing. However even if you have a lot of automated tests in the pipeline there are still going to be some test cases that need to be run manually. So lets talk about best practices for Test Plan Creation in Azure test plans.

First lets get on the same page. Many people I have talked to have different definitions of what a Test Plan is and consists of. For the purpose of this discussion a test plan is:

• a collection of test cases that validate requirements being delivered
• for each configuration we need to test for
• to be executed against each build as they are delivered until all tests are passing

That’s it. There is no need for all kinds of extra stuff people never read anyway.

The life of a Test Plan is best when short. If you are developing in sprints expect the test plan to have the lifespan of a sprint. If you are not agile it should have the lifespan of a release. If you are still waterfall and taking a year or more to deliver something to the business, you have more important things to be reading than this article.

For the purpose of this writing lets assume we are a scrum team and running in 2 week sprints. If you read my last article you know that during grooming you would have added test cases to the stories as they were groomed. Now during sprint planning we commit to completing those groomed stories, therefore your test cases are attached and ready to be executed.

As the team member responsible for manual testing of the stories this sprint. You will go into Azure Test Plans and create a new test plan for the sprint. I like to name the test plan after the sprint.

There are 4 ways to organize an Azure Test Plans test plan.

1. Directly add test cases to the root of the test plan
2. Add a Static test suite and add your test cases to that (this is like adding folders)
3. Create a Query based test suite that pulls test cases into the test suite via a query
4. Create a Requirement based test suite which will pull the test cases into the test plan that we attached to the Stories during grooming.

Number 4 is the best practice.

Lets try it out, create a test plan named after your sprint and add a requirement based test suite.

You will be presented with a query window. Use this query to find all the stories committed to in Sprint 2.

The above query returns all the stories and bugs assigned to my team and my current sprint.

NOTE: If you are attaching your test cases to Features, change the first claus to “Work Item Type In Group Microsoft.FeatureCategory”

Run the query, and select all the work items returned by the query result, then click Create suites. Your test plan will now have a test suite for each story in the sprint. As you select each test suite you can see that your test cases have been brought into the test plan along with the stories.

That's it, you have a test plan to test the stories in your sprint. As each story is delivered execute the tests for that story and set their outcome accordingly. (Passed, Failed, Blocked)

To show everyone the current status of the test plan, you can create a chart for the test plan and add it to your dashboard.

As you mark tests passed or failed this will of course be updated.

If you are wondering how this chart is created here is view of the editor. To get test results. Group by the Outcome field.

What about Regression Tests. Some teams want to rerun regression tests each sprint for some of the more critical features. Query based suites are handy for this. First add a tag to all the test cases that you want to run as a regression test.

Then add a Query based suite with a query that brings back test cases with the “Regression” tag.

This test suite will pull all the test cases with the regression tag into the test plan for you.

Now you have a self authoring test plan. If a test case is tagged as regression it will show up under the regression test suite and if a test case is added to a story it will show up in the test plan under the story test suite.

Another best practice is to document the build you are testing. At some point during the sprint a build will be ready for manual testing. When that happens you should set/change the build in the test plan settings so that any test runs have the build number documented in them.

Not a bad idea to also select the stage you are running the test on. The documentation in the test run the better.

There are two more things you may want to take advantage of.

When you look at the test cases under a test suite, notice they have two properties Configuration and Tester.

Lets talk about that.

Configurations:

Configurations are a way for you to show that a test case has been executed for more than one configuration of the application. For example you may need to validate your app works on more than one browser.

Under Azure Test Plans you can create all the configurations you are required to test for.

Then in your test plan you can associate configurations to test suites.

Lets say we have been asked to test on Chrome and Edge for Story 1892.

Once we select the configurations for this test suite the test cases are duplicated. To see this, switch to the execute tab.

Now you will need to execute each test case twice, once for each configuration. And the reporting will reflect that. This test plan really is a living document. :)

Assign Testers

You may also want to assign specific Test Cases or Stories to be executed by specific people on the team. This can be done at the test suite level or the test case level.

To assign an individual to execute all the tests for one story use the Assign testers to run all tests menu item under the test suite context menu.

To assign individual test cases to team members use the Assign Tester option on the test case context menu.

If you want to know how things are broken down create a chart as we did above but group by configuration or tester.

My goal is always to get the most transparency with the least amount of work. It breaks my heart when I hear someone creates a QA report manually, just so something can be signed off.

Instead of creating that report to tell people the code is good to go, let them look at your dashboard and see it throughout the whole software development lifecycle.