Mutation Testing in .NET, part 2
A while ago I wrote about my attempt to write a mutation testing tool for .NET code. I ended up with something that kinda worked, but not well enough to be very useful.
Since then I’ve tried again. And with help from my colleagues I’m happy to say that the second attempt is a lot more stable. It’s now being used in a few teams in the company, helping to spot gaps in our tests.
This second attempt is called Fettle, and it’s available on Github.
To be able to use it right now you need code that:
- Runs on .NET Framework 4.x
- Uses NUnit 3 for its tests
But what changed to make it work this time?…
Fettle uses Roslyn to mutate the original C# source code (instead of modifying the compiled CIL byte-code like I did originally).
This means that it’s no longer affected when the byte-code doesn’t match the source code. Now, when a mutant survives, it’s clear to see that it’s genuine and what code was changed.
After the initial work on Fettle, I started running it over some of our code to spot where mutants were surviving. The trouble was, it took hours. And so it became an over-night automated job. This was ok, but having such slow feedback on a change wasn’t ideal.
So I decided to do some optimisations.
The biggest win was to run fewer tests. That is, after mutating some code, instead of running all tests, only run the tests that are necessary. That is, the tests that call (directly or indirectly) the code being mutated.
I did this by doing a bit of test impact analysis. When Fettle starts, it instruments the code being tested to record when each method is called. It then runs each test, checking which C# methods are run as part of each test.
This information is collated, so that when it’s time to make a mutation, Fettle knows which tests are associated with that method in the source code.
This is quite coarse-grained (it just works out what methods are called, not individual instructions) but it worked really well. Mutation testing went from a few hours to 12 minutes. The team are now able to do a few mutation testing runs throughout the day, spotting testing gaps much sooner and while the code is fresh in their minds.
Although my previous attempt was written in F#, I decided to write Fettle in C#.
I‘m a big fan of F# (and functional programming general). But, unfortunately, it’s not widely understood by my colleagues. By using C# I was able to maximise my potential pool of contributors.
And in the end I was able to get some invaluable advice and contributions from the people I work with.
Fettle has helped a few teams at the company to spot gaps in their testing, which is fantastic!
But it’s still very experimental. There are lots of fixes and improvements that can be made.
By making it available to the public I’m hoping that:
- People find it useful for their C# projects.
- Mutation testing becomes more widely known/talked about in the community.
If you have any thoughts I’d love to hear them, feel free to get in touch.