SpecFlow Tips: Running SpecFlow scenarios in parallel with xUnit v2

Originally published at gasparnagy.com on February 23, 2016.

As mentioned earlier, one of the key new features of SpecFlow v2 is that you can run your tests in parallel within this same AppDomain. This is currently possible with NUnit v3 and xUnit v2. This post shows a step-by-step guidance about how this can be done.

First of all, I would like to highlight that even though you can speed up the execution of your tests with parallelization, if you have integration steps that depend on shared resources (e.g. on a shared database), you have to make extra considerations in order to be able to parallelize them. In fact, if you have never run your tests in parallel before, it is quite unlikely that they will run smoothly without any change. So just like with every new method, it makes sense to conduct some experiment on how this works before jumping into the parallelization of your real project.

If your project uses xUnit, you can switch off parallelization until you fix all related problems with the [assembly: CollectionBehavior(DisableTestParallelization = true)] global attribute.

As an example, we can use the “Calculator Sample” that you get when you add a new feature file to your project. Not the best example, but good enough for now. You can find the code sample for this post on GitHub: Sample_ParallelSpecFlowXUnit.

If you are not familiar with xUnit, there is a good introduction on their website. Here I am going to concentrate only on the really necessary steps.

If you are familiar with the general SpecFlow setup, you can have a quick look at the GitHub repo and jump to the “Ready to run” section directly.

Create a project

So first, you need to create a new class library project (I’ve called it Calculator.Specs) and add a few NuGet packages. Fortunately there is a composition package called SpecFlow.xUnit that does almost all of it. Start with adding that one. This adds the packages shown in the picture and in addition to that, it configures SpecFlow to use xUnit as a unit test provider. So the App.Config should look like this.

<specFlow>
  <unitTestProvider name="xUnit" />
</specFlow>

Actually this should be enough to start coding, but if you want to run your tests as well, you can add the xunit.runner.console package that provides a console runner or the xunit.runner.visualstudio package that shows your tests in the Visual Studio Test Explorer window. You can also add both of them…

Add scenarios

As I have said, in order to have some initial specs, we just need to add a new SpecFlow Feature File from the solution explorer, because that gives us a scenario for the “Addition” function of our calculator. You can push it up a bit by turning into a Scenario Outline, so that we have a bit more tests to play with.

Most of the test runner frameworks that allow parallel execution run the tests from the same class sequentially and only parallelize the tests in different test classes. xUnit works in the same way. This means that if we really want to see something running in parallel, we need to have at least two feature files! So let’s add a Multiplication.feature similarly to the addition.

Run the tests

As I have mentioned, you can run the tests in different ways. With the console runner you need to do something like this from the solution root:

> .\packages\xunit.runner.console.2.1.0\tools\xunit.console.exe .\Calculator.Specs\bin\Debug\Calculator.Specs.dll

xUnit runs the tests in parallel by default., so you don’t have to specify any extra arguments. If you want to force single-threaded execution, you need to provide the -parallel none option. This is quite convenient for experimenting.

The Visual Studio runner also runs the tests in parallel by default. To my knowledge, you cannot turn this off in the runner itself, but there are several ways to configure the default parallel behavior for xUnit described here.

Implement the step definitions and the calculator

As the specs imply, this is a kind of Reverse Polish notation style calculator, where you have to enter the operands first and then select the operation to be performed. So the implementation will probably have a stack for the operands and some methods for the operators, like Add() or Multiply(). You can query back the result from the calculator. So this is a stateful class. You can look at the implementation here.

The step definitions are also pretty simple. They keep the calculator instance in an instance field and do the necessary calls and assertions.

Ready to run

OK. So let’s run our scenarios. (I’m sure you have already done this during the implementation process.) They pass! Hurray!

But wait! Are they running really in parallel? Sometimes it is not so easy to see that. SpecFlow provides some help. If the tests run in parallel, all trace messages are prefixed with the thread id. If you run the tests single-threaded, there is no such prefix. So let’s go back to the console and check. For me it looks like this:

As you can see, the execution started with a multiplication scenario on thread #13 and an addition on thread #8. If you run the same test, but now with the -parallel none option, you would not see the thread numbers in the output.

That evil static state

The .NET AppDomains provide memory boundary. All static states (objects stored in static fields) are visible to any other code within the same AppDomain. So if you run the tests in parallel within the same AppDomain, as we are doing now, all shared fields can be accessed from all test threads, so they are potentially dangerous. Of course, static state is also a potential problem in the real application, but during the normal execution of the application, there is only one application instance running in the AppDomain. So using static fields for caching — if thread-safe — can be a valid usage. During testing however, every test represents an instance of the application that we expect to run in isolation. So even innocent caches might cause troubles.

We can simply simulate this problem by changing the step definition class field (holding the current calculator instance) to static.

You can get the same effect (or maybe even worse) if you change the stack of operands inside the calculator to static.

If you run the tests now (without the -parallel none option), you will see strange errors, like this.

Although static fields are usually problematic points of parallelization, they just represent the problem of shared state. Unfortunately you can also have shared state in a database too. That is much harder to fix.

Also don’t forget that even if you don’t use static fields, maybe one of the libraries you use (e.g. an ORM) does use static caches… These problems are typically hard to find and fix.

The Scenario Context

Another problem you might face when switching to parallel execution is the access to the ScenarioContext.Current static property. Currently SpecFlow throws an exception if you try to access this property from a parallel testing environment.

Let’s change the step definitions to use ScenarioContext.Current to store the calculator. (You can see the complete solution for this in the “scenariocontext” branch of the GitHub repo.)

If you run the tests in parallel with this, you will get the exception I’ve mentioned.

In order to solve this problem — if you really need to use the scenario context — you have to get the current context instance using context injection. Sounds complicated but it it easy: just declare a ScenarioContext parameter for the constructor of your step definition class and save it to an instance field. You can use the instance field in your step definitions, and these will also work with parallel execution. (See the scenariocontext-parallel branch for a complete solution.)

If you re-run the tests, you can see that they run again without errors also with parallel execution.

Next Steps

You can use this simple project for doing further experiments. You can add more feature files to increase the degree of parallelization. You can modify the parallel behavior of xunit and you can introduce further complexity as well, like logging the entered numbers into a (shared) file or to a shared database.

Once you have seen how the typical problems can be identified and solved, you can try to turn your project into a parallel testable state.

Gaspar Nagy is the creator of SpecFlow working as a trainer & coach. Check out his public SpecFlow courses or request an in-house private course for your team. He is a bdd addict and as such he’s editing a monthly newsletter about interesting articles, videos and news related to BDD, SpecFlow and Cucumber.