Mutation Testing in .NET, part 2

Oli Wennell
Jun 29, 2018 · 3 min read

A while ago I wrote about my attempt to write a mutation testing tool for .NET code. I ended up with something that kinda worked, but not well enough to be very useful.

You can read the previous post here.
Also if you’re not clear on what mutation testing is, see
this even earlier post.

Since then I’ve tried again. And with help from my colleagues I’m happy to say that the second attempt is a lot more stable. It’s now being used in a few teams in the company, helping to spot gaps in our tests.

Introducing Fettle

This second attempt is called Fettle, and it’s available on Github.

To be able to use it right now you need code that:

  • Runs on .NET Framework 4.x
  • Uses NUnit 3 for its tests

But what changed to make it work this time?…

Stability

Image for post
Image for post
Le Pont du Gard by Hubert Robert, photo from Wikimedia

Fettle uses Roslyn to mutate the original C# source code (instead of modifying the compiled CIL byte-code like I did originally).

This means that it’s no longer affected when the byte-code doesn’t match the source code. Now, when a mutant survives, it’s clear to see that it’s genuine and what code was changed.

Fast Feedback

Image for post
Image for post
Photo by Böhringer Friedrich

After the initial work on Fettle, I started running it over some of our code to spot where mutants were surviving. The trouble was, it took hours. And so it became an over-night automated job. This was ok, but having such slow feedback on a change wasn’t ideal.

So I decided to do some optimisations.

The biggest win was to run fewer tests. That is, after mutating some code, instead of running all tests, only run the tests that are necessary. That is, the tests that call (directly or indirectly) the code being mutated.

I did this by doing a bit of test impact analysis. When Fettle starts, it instruments the code being tested to record when each method is called. It then runs each test, checking which C# methods are run as part of each test.

This information is collated, so that when it’s time to make a mutation, Fettle knows which tests are associated with that method in the source code.

This is quite coarse-grained (it just works out what methods are called, not individual instructions) but it worked really well. Mutation testing went from a few hours to 12 minutes. The team are now able to do a few mutation testing runs throughout the day, spotting testing gaps much sooner and while the code is fresh in their minds.

Enabling Collaboration

Image for post
Image for post
The Conversation, by Camille Pissarro, photo from Wikimedia

Although my previous attempt was written in F#, I decided to write Fettle in C#.

I‘m a big fan of F# (and functional programming general). But, unfortunately, it’s not widely understood by my colleagues. By using C# I was able to maximise my potential pool of contributors.

And in the end I was able to get some invaluable advice and contributions from the people I work with.


Now What?

Image for post
Image for post
Wanderer Above the Sea of Fog, by Caspar David Friedrich, photo from Wikimedia

Fettle has helped a few teams at the company to spot gaps in their testing, which is fantastic!

But it’s still very experimental. There are lots of fixes and improvements that can be made.

By making it available to the public I’m hoping that:

  • People find it useful for their C# projects.
  • Mutation testing becomes more widely known/talked about in the community.

If you have any thoughts I’d love to hear them, feel free to get in touch.

Compare the Market

The people behind comparethemarket.com and the Meerkat App

Oli Wennell

Written by

Embarassing Dad, husband, diagram doodler. Tech lead at comparethemarket.com

Compare the Market

The people behind comparethemarket.com and the Meerkat App

Oli Wennell

Written by

Embarassing Dad, husband, diagram doodler. Tech lead at comparethemarket.com

Compare the Market

The people behind comparethemarket.com and the Meerkat App

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store