Mutation Hunt — Exterminating mutant code through unit tests

It is pretty widely accepted in software development that Unit Testing is a wonderful tool to give you feedback about your code’s quality during the software lifecycle. As time goes by, your code changes and your tests blink a red light if any of the prior expected behaviors was violated, so that you can fix your logic, or update the expected behavior.

But what if your tests fail to show your code’s changes affected the software behavior? let’s think of a practical example:

That’s it, a really simple function that receives two numbers and multiplies them. Never will one fail to test this! I’m a bold QA/Developer and will write a single simple test for that!

Bravo! My code is tested!

Is it?

What if my production code becomes x + y ? What if it becomes x to the power of y? Is this test going to show my logic changed?

It’s pretty clear my choice was not exactly the best one, but sometimes things are not that obvious, and that’s why we need to conduct mutation testing to help us think about threats.

Mutation tests perform changes on your production code in order to show fragile parts of your system and test code. Running mutation tests for that specific pair of Test-Production Code will run your tests against mutated versions of your code and see if the test “kills” the mutant. Let’s take a look at the output of the mutation test.

Looking at the output you can see that three mutations were created, and they were tested against your test code. Two of them were killed by the test (x / y and x // y), however, the last mutation survived, which means if that rule changes in the way described by the mutation your test won’t fail, and the methods that consume them may receive incorrect data.

It’s simple to correct that, let’s just change the numbers in the test:

And retry the mutation tests:

All mutants were killed successfully! But you may imagine this is only possible with really simple tests. Well, I have tested a slightly bigger project using this technique. Do you remember this contract testing repo I have mentioned in the article Closing a Safer Deal — Contract and Collaboration Tests working together to improve security? Well, I have tested the unit test class using this technique and decided to improve some of the tests. Do you think there’s any extra test to be implemented? Feel free to fork the repo and show me ways to improve the test suite. By doing that we can think together about ways of improving our tests and our software as a whole.