SpecFlow without code-behind files and a big step towards .NET Core support
SpecFlow uses generated code-behind files for executing the scenarios. Not because we like code-behind files, but because this was the only way we could easily integrate Cucumber-style Behavior Driven Development to the .NET developer toolset. But after almost 8 years, can we get rid of the code-behind files finally? And how is it related to .NET Core? Just keep on reading…
When I planned to make this post, I was hesitating about the title. What should I highlight? The fact that we have taken a big step towards the full .NET Core support that many people have been waiting for? Or shall I highlight that after so many years finally we can get rid of the code-behind files? This might be an important news for almost every SpecFlow users… I could not make a decision, so in the end I have chosen this hybrid one. I don’t dare to enter it to the headline analyzer. It would surely give negative score. Anyway…
This is a kind of “good news” post, but let me start from the beginning. If you want to have only the dessert, just jump ahead to the section “The good news: SpecFlow.xUnitAdapter“.
Why does SpecFlow use code-behind files?
I have told the story about the beginnings of SpecFlow a few times already, so I won’t repeat. The essence of it is that I wanted to have a native .NET version of Cucumber, that 1) integrates to the tools that .NET developers use (Visual Studio & co.) and 2) I wanted to have it as soon as possible, because my project needed it.
Cucumber was a stand-alone test execution environment. You installed the Cucumber gem and got the “cucumber” executable that you could use to run your scenarios.
An obvious way to port it would have been to make a “specflow.exe” that can run your scenarios in a .NET project. But running command-line tools did not fit to the way how usual .NET devs worked. We’ve “got” great IDE tools for editing, navigating, running, debugging and testing things. A console runner would not fit into that.
I had some past tool-writing experience and I knew that writing a proper test runner tool was not easy. You have to deal with the different .NET versions, processor architectures, threading models, AppDomain and assembly loading strategies and many more. I did not want to take this. Especially because there were some great test runner tools, like NUnit or xUnit, which already did this hard job.
So my super-pragmatic solution was to combine this two together: generate test classes and test methods from the feature files and let NUnit run them. Build-time generation was not well-supported at that time, so I decided to use the infrastructure of Visual Studio (called SingleFileGenerator) that allowed generating code-behind files from other content files. In our case generating the test classes from the feature files.
The code-behind files have a lot of disadvantages.
- They store redundant information, which might get out of sync with the feature files.
- They are not much source-control merge tolerant.
- They confuse people because it can be mixed up with step definition classes.
- They might need to be re-generated after a version upgrade or a configuration change.
- They… shall I really carry on?
At the same time, code-behind files had some advantages as well.
- They allowed to make SpecFlow work with many different test runner frameworks.
- They loosened the dependency from the different versions of the test runner frameworks. As long as their outer interface remained the same in code-level, the generated code just worked.
- They moved the complexity of parsing the feature files to generation-time, which allowed the runtime to be much simpler. In the end this allows to use SpecFlow in limited runtimes, like Silverlight — only the runtime part had to be ported.
- They removed the parsing overhead of feature files from the runtime test execution (the parser we have used until recently is extremely slow).
You could weight these arguments and it comes out that it would be still better without code-behind files. But this was not an option at that time.
Later on, we added compile-time generation support, but — thanks to the unpredictability of the Visual Studio-MsBuild integration — this has never got popular.
How does .NET Core come into the picture?
.NET Core is a brand new .NET framework that has been built from scratch mainly targeting server-side applications — an ideal target for SpecFlow! We have been interested in supporting .NET Core from the beginning, but this was not really easy. .NET Core has been changed quite much since it started, and these changes make tool development pretty much painful. I summarized this problem in another post earlier.
But besides these general issues, there were a few concrete ones as well that we had to face when thinking about the SpecFlow .NET Core support. Some of them are related to code-behind files. (Andreas Willich, who has been working a lot on the .NET Core support, is maintaining a checklist of the challenges we have to solve.)
The most painful problem is that the .NET infrastructure we have used for generating the code for code-behind files, the CodeDom, is not available in .NET Core. So porting the generator part of SpecFlow to .NET Core would be quite complex, because we either rewrite the entire codebase to leave out CodeDom or try to find (or write) a CodeDom replacement for .NET Core. Neither of these looked very promising.
The SpecFlow .NET Core support has changed the balance between the advantages and disadvantages of the code-behind files. So I started to look for alternatives.
Dynamic test generation
Let’s keep the original idea of using existing test runner tools to execute the scenarios. If we want to get rid of the code-behind files, we need dynamic test generation.
NUnit had dynamic test generation support for a long time, but it worked only on test-level. Once you had a test class, you could dynamically generate the tests for it. But in our case, we do not want only to generate the tests for a test class, but even to generate the test classes themselves.
We also needed a dynamic test generation support that does not work with the native test runner of the framework only (e.g. nunit-console for NUnit), but for external, 3rd party test runners as well, like the Test Explorer Window of Visual Studio.
xUnit for help
xUnit was developed by Jim Newkirk and Brad Wilson, with the goal of making a modern, extensible unit test framework. Probably this helped xUnit to become the first unit test framework available for .NET Core. And it was not only the first but it also became the de-facto standard unit test framework for .NET Core.
I have already tried to use xUnit to a dynamic test generation for the scenarios, however, neither xUnit nor SpecFlow was really prepared for this at that time. But a few months ago, after a status call with Andi Willich about the SpecFlow .NET Core support, I looked up the possibilities again. And after a few days of hacking, I could see a passing scenario in the Visual Studio Test Explorer Window, the first one ever without code-behind file!
The good news: SpecFlow.xUnitAdapter
The infrastructure I have created for running SpecFlow scenarios without code-behind files is now available as a NuGet package and can be used in any SpecFlow v2.1 project configured for xUnit. The plugin supports the normal .NET framework currently, but it will be used for .NET Core as well later.
(If your project does not use xUnit, as I have explained it in my post, you don’t have to use the assertions from the same unit test library that you use for running. You can keep your step definitions making NUnit assertions for example, but please add the xUnit NuGet packages as well, and reconfigure SpecFlow to use xUnit. This way you can get ready for removing the code-behind files with minimal changes.)
In order to be able to use it, you have to
- add the SpecFlow.xUnitAdapter NuGet package to your SpecFlow project,
- remove the “SpecFlowSingleFileGenerator” value for the “Custom Tool” setting in the properties of the feature files. This will automatically remove the code-behind files as well.
- configure the feature files to be copied to the output directory. (The plugin currently runs the feature files from the output folder.)
You can read the instructions on the github project page of the plugin: https://github.com/gasparnagy/SpecFlow.xUnitAdapter or you can watch my quick demo video.
The plugin is in beta phase currently and there are some limitations, but it should be enough to give a first try and explore the world without SpecFlow code-behind files. It feels different, really.
If you may test it and encounter into any problems, please report it on GitHub. If you find it useful, please consider to donate the plugin so that I can spend more time on improving SpecFlow. It is all free and open-source.
Next steps for .NET Core
The idea of replacing the code-behind files came from the .NET Core support, but finally the first results seem to be usable for normal .NET projects as well. But what are the next steps for .NET Core?
We still have a lot to do for the full support. Especially because we would like to do it in a way that we should be able to run all existing tests for .NET Core as well. But on a proof-of-concept level it works already, so hopefully we can share further good news about it soon!
Please ♡ this post and follow me for future stories. Thanks for reading.
Gaspar Nagy is the creator of SpecFlow working as a trainer & coach. Check out his public SpecFlow courses or request an in-house private course for your team. He is a bdd addict and as such he’s editing a monthly newsletter about interesting articles, videos and news related to BDD, SpecFlow and Cucumber.
Originally published at gasparnagy.com on April 24, 2017.
