Mutation Analysis with Stryker.NET and SonarQube

In Trendyol, we apply dozens of different practices to ensure that we provide the best value for our customers and one of these practices is mutation testing.

Today, we’ll start by talking about mutation testing in .NET and continue by importing mutation testing reports of a .NET project to SonarQube.

Mutation Testing

Mutation Testing is a type of software testing in which certain statements of the source code are changed/mutated to check if the test cases can find errors in the source code. The goal of Mutation Testing is to ensure the quality of test cases in terms of robustness that it should fail the mutated source code.

If the question is why the quality of tests is that much important, I would be grateful to quote from Uncle Bob’s Clean Code:

Tests are as important to the health of a project as the production code is. Perhaps they are even more important, because tests preserve and enhance the flexibility, maintainability, and reusability of the production code. So keep your tests constantly clean. Work to make them expressive and succinct.

If you let the tests rot, then your code will rot too. Keep your tests clean.

Each programming language has different kinds of mutation testing tools such as Pitest for JAVA, go-mutesting for GO and Stryker.NET for C#.

Stryker.NET

Stryker.NET is a mutation testing tool for .NET. It mutates the source code of .NET app (inserts bugs to the source code!) and runs tests against these updated source codes to measure the quality of the tests just like any other mutation testing tool.

To demonstrate how to analyze mutation testing reports with SonarQube I’ve prepared an example which is a .NET application that sums two integers.

The Github repository can be found here:

mstfymrtc/dotnet-mutation-testing-example

Here is our Utilities.cs class:

And our first unit test case:

Although this test passes, there is one more thing to consider. Let’s try to mutate our source code using Stryker.NET.

Start by installing Stryker.NET:

$ dotnet tool install -g dotnet-stryker

The next step is to run Stryker.NET in our example project to start mutation testing analysis. We start by navigating to the folder where our test project takes part. In our case, it’s tests folder.

$ cd tests
$ dotnet-stryker

By default, Stryker.NET generates a HTML report as below:

As a result, our mutation score is 0 and there is one survived mutant. Survived mutant means that Stryker.NET mutated our source code -in this case, it replaced plus (+) sign in our Sum method by minus sign (-)- and run the tests.

Although the source code and the logic of our Sum method was changed, our tests still passed which means there is a case that we didn’t cover.

One of the most important point in here is, code coverage is a liar. Having code coverage of 100% doesn’t mean you’ve tested all the scenarios. So this is where the mutation testing comes into question.

So let’s add one more test and try to kill the survived mutant. (Yup, so wild)

And run Stryker.NET again:

In Sum_GivenOnePlusTwo_ShouldCalculateProperly, we give 1 and 2 as a parameter and expect our result to be 3.

Because Stryker.NET replaced a+b with a-b, the result of our last test was 1–2 = -1 and that’s why our last unit test didn’t pass. So we were able to kill the mutant.

Until this point, we’ve talked about mutation testing and how to apply it in our .NET project. But today’s topic is much more exciting. As you can see, we’ve gone through the HTML report -which is generated by Stryker.NET for us- to analyze the mutation testing results.

Imagine you have a .NET project with lots of classes, logic and code. Wouldn’t it be hard to go through this plain old HTML report and take action?

What if we were able to view all the mutation testing results in SonarQube?

Importing Mutation Testing Results to SonarQube

To be able to import mutation testing results to SonarQube, I’ve developed an open source plugin:

Trendyol/sonar-stryker-net-plugin

Let’s start by installing the plugin to SonarQube.

At this point, the plugin is not published in the Sonar Marketplace yet, so we’ll go for manual installation. I’ll update the post when it’s published :)

Manually installing sonar-stryker-net-plugin to SonarQube:

1. Download the JAR file of the plugin from here.
2. Put the downloaded JAR in $SONARQUBE_HOME/extensions/plugins
3. Restart your SonarQube server.

After restarting the SonarQube server, go to the Quality Profiles section and follow the steps below to activate the plugin and its rules.

If the steps above are all OK, we continue by installing dotnet-sonarscanner globally.

dotnet tool install --global dotnet-sonarscanner

After installing dotnet-sonarscanner, let’s delete the last test we added and bring back our good old survived mutant to life.

We have to re-run mutation testing analysis in the tests directory, but this time with a different command:

dotnet-stryker --reporters "['html','json']"

The reason we add json argument before running Stryker.NET is; the sonar-stryker-net-plugin needs json report to parse it and then import mutation test results to SonarQube.

Finally; we can run dotnet-sonarscanner with the following bash script in our project’s root directory:

Yay! We were able to see our analysis results in SonarQube dashboard.

Conclusion

In this blog post, I tried to go over the basics of mutation testing in .NET and how to import its results to SonarQube with the help of sonar-stryker-net-plugin.

I hope the plugin helps speed up your team’s workflow just like us. Without being forgotten, contributions are welcomed for sonar-stryker-net-plugin!

Please feel free to contact me if you have any questions.

Thanks for reading & take care! 🖐🏼

References

https://www.guru99.com/mutation-testing.html

http://csharp.academy/mutation-testing/