Mutation Testing in .NET, part 2

A while ago I wrote about my attempt to write a mutation testing tool for .NET code. I ended up with something that kinda worked, but not well enough to be very useful.

You can read the previous post here.
Also if you’re not clear on what mutation testing is, see
this even earlier post.

Since then I’ve tried again. And with help from my colleagues I’m happy to say that the second attempt is a lot more stable. It’s now being used in a few teams in the company, helping to spot gaps in our tests.

Introducing Fettle

This second attempt is called Fettle, and it’s available on Github.

To be able to use it right now you need code that:

  • Runs on .NET Framework 4.x
  • Uses NUnit 3 for its tests

But what changed to make it work this time?…

Stability

Le Pont du Gard by Hubert Robert, photo from Wikimedia

Fettle uses Roslyn to mutate the original C# source code (instead of modifying the compiled CIL byte-code like I did originally).

This means that it’s no longer affected when the byte-code doesn’t match the source code. Now, when a mutant survives, it’s clear to see that it’s genuine and what code was changed.

Fast Feedback

Photo by Böhringer Friedrich

After the initial work on Fettle, I started running it over some of our code to spot where mutants were surviving. The trouble was, it took hours. And so it became an over-night automated job. This was ok, but having such slow feedback on a change wasn’t ideal. 
 
So I decided to do some optimisations.

The biggest win was to run fewer tests. That is, after mutating some code, instead of running all tests, only run the tests that are necessary. That is, the tests that call (directly or indirectly) the code being mutated.

I did this by doing a bit of test impact analysis. When Fettle starts, it instruments the code being tested to record when each method is called. It then runs each test, checking which C# methods are run as part of each test.

This information is collated, so that when it’s time to make a mutation, Fettle knows which tests are associated with that method in the source code.

This is quite coarse-grained (it just works out what methods are called, not individual instructions) but it worked really well. Mutation testing went from a few hours to 12 minutes. The team are now able to do a few mutation testing runs throughout the day, spotting testing gaps much sooner and while the code is fresh in their minds.

Enabling Collaboration

The Conversation, by Camille Pissarro, photo from Wikimedia

Although my previous attempt was written in F#, I decided to write Fettle in C#.

I‘m a big fan of F# (and functional programming general). But, unfortunately, it’s not widely understood by my colleagues. By using C# I was able to maximise my potential pool of contributors.

And in the end I was able to get some invaluable advice and contributions from the people I work with.


Now What?

Wanderer Above the Sea of Fog, by Caspar David Friedrich, photo from Wikimedia

Fettle has helped a few teams at the company to spot gaps in their testing, which is fantastic!

But it’s still very experimental. There are lots of fixes and improvements that can be made.

By making it available to the public I’m hoping that:

  • People find it useful for their C# projects.
  • Mutation testing becomes more widely known/talked about in the community.

If you have any thoughts I’d love to hear them, feel free to get in touch.

Like what you read? Give Oli Wennell a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.