Test Categories and Running a Subset of Tests in Team Foundation Server 2010

Disclaimer: I’m writing this at a time when only Beta 1 is available for Visual Studio Team System 2010 so the information may have changed by the time it has been released. I have included links to the relevant MSDN articles which should remain valid after release time so just double-check.

This small little additional feature is actually one that I have been looking forward to for a long time. In Visual Studio Team System and Team Foundation Server 2010, you will now be able to limit your test runs to specific test categories with a new command-line option on MSTest.exe and therefore in Team Build 2010 which calls MSTest.exe automatically for you.

Back in the day… You would need to create test lists (.VSMDI files) in VSTS 2005 and VSTS 2008 to basically “categorize” your automated unit tests by putting them into different lists. One handy thing about them is that the lists could be hierarchical which helps out at build time. When you wanted to run a specific subset of tests either locally using MSTest.exe or in Team Build, you would just specify the .VSMDI file to use and then the test list you wanted to run. Not too bad, but it’s a pain to keep up with those test lists. Serious pain. However, the thing that I hated absolutely most about them is that you could only edit the .VSDMI files if you purchased Visual Studio Team Suite or the Tester Edition. So that means that if you have just the VSTS Developer Edition then you are pretty much out of luck. For most places that I have seen, it’s usually the developers maintaining those test list files not the testers.

For this reason I actually prefer and will be recommending the Test Container and Category approach going forward in 2010. Test Containers are essentially files that contain tests in them. For example, unit tests (and other compiled tests) are stored in .dll files and ordered tests are in .orderedtest files. I like this approach. In automated builds I just want to specify which files contain the tests that I want to run and then if I want to limit the test run to just a subset I can just list which categories to run.

A great example of this is what I call the “BVT” category. These are the tests that you have identified to be your “smoke” tests that make sure a build is okay. If these tests fail then you’ve probably got a bad build. (BVT = Build Verification Tests) So I would limit the test runs on any CI or even the new Gated Check-In builds to just those BVT tests. Quick & dirty verification is really all you need for those builds leaving a more extensive automated test pass to happen during the nightly or weekly build. If you’re not familiar with the new Gated Check-In feature in TFS 2010, check out Brian’s blog post or Patrick’s blog post for more information. It’s a killer feature.

This does rely on one thing though… each “developer” of an automated test needs to make sure they add the correct attribute(s) to their test methods. You don’t have to keep up with maintaining the .VSDMI files any longer but you do have to make sure you mark each method appropriately.

You can even use test categories with the new types of automated tests available in 2010 like Coded UI Tests. It doesn’t just have to be unit tests.

How to Specify a Category in an Automated Test

This part is pretty easy. You just add as many TestCategory attributes to the test method as you need. Here’s an example in C# using multiple test categories for a test method called DebitTest:

[TestCategory(“Nightly”), TestCategory(“Weekly”), TestCategory(“Monthly”), TestMethod()] public void DebitTest() { }

Alternately, you can select a test in the Test View tool window and then set the category by using the Properties tool window in Visual Studio and it will add the appropriate attributes to the methods for you.

How to Specify which Categories to Run in an Automated Build with Team Build 2010

Okay… this part is easy too. :) Build definitions now have build properties that can be exposed to the end user in the Build Definition Details dialog or in the Queue Build dialog. This is handy because you could by default not set a filter to run under normal circumstances (triggered or default manual builds) or you can change it when manually queuing a build if you want that build to run differently. Either way it’s the same for setting the categories. If you’re using the default build process workflow that is available out of the box, then just scroll down through the property list until you reach the Testing section which includes a build property called Test Category. Leave it blank if you want to run all tests or specify the categories you’d like to limit it too:

According to the MSDN documentation for the Test Category switch, you can combine multiple categories in different combinations instead of just specifying one category. Very handy — here’s some examples:

• /category:group1 runs tests in the test category “group1”.
• /category:”group1&group2" runs tests that are in both test categories “group1” and “group2.” Tests that are only in one of the specified test categories will not be run.
• /category:”group1|group2" runs tests that are in test category “group1” or “group2”. Tests that are in both test categories will also be run.
• /category:”group1&!group2" runs tests from the test category “group1” that are not in the test category “group2.” A test that is in both test category “group1” and “group2” will not be run.

What I’m not sure about is whether you can specify test categories when using the old Upgrade Build Workflow template .xaml file… I’ll check on that and then update the blog post.

It’s worth noting that if you are going to use the test category method to limit test runs, you must use test containers.

Limiting Test Runs Based on Test Priorities

If you noticed in the screenshot above from Team Build, you can also limit your test run to tests that are in a specific priority range. How do you specify the range for your test methods? You can use the Properties window when selecting a test in the Test View tool window or you can add the Priority attribute manually to the test method. After that you just specify the range of priorities to use in the test run.

[TestCategory(“Nightly”), TestCategory(“Weekly”), TestCategory(“Monthly”)] [TestMethod()] [Priority(1)] public void DebitTest() { }

Additional Note: It appears that the product team is actually encouraging people to move away from the old .VSDMI approach in favor of categories. Check their note out:

Note

Test categories are recommended for use over the test lists functionality from earlier versions of Microsoft Visual Studio Team System Test, unless you have to create a check-in policy which requires a test list. For more information about check-in policies, see How to: Add Check-In Policies.

Take care and happy testing,

Ed Blankenship