Image from https://www.flickr.com/photos/thisisengineering/

Mutation Testing in Kotlin

Today most of us know why testing is necessary in order to ensure the behavior of our code goes as expected, but how can we be sure that our tests are checking what we need to? Here is where mutation testing comes in handy

What is mutation testing?

Mutation Testing is a technique used to evaluate the quality of the tests of an application and identify tests to add through some modifications in the code.

The idea is to make changes in our code that alter the execution flow. One simple alteration is to reverse the condition of an “if” and run the tests to see if any of them fail and catch the mutant.

Why is this interesting?

Let´s see a real-world example

This doesn´t look like a non tested project, right? It´s not that big and you have a 67% it´s just fine…BUT how about the mutations?

This value is not so good anymore, maybe we have a lot of tests that are not useful 🤷‍♀️

At the podcast “Programar es una mierda” they asked me about test coverage. In my experience, it’s not a reliable tool to measure the correctness of our tests because it is easy to fake the results. However, Mutation testing seems to be a more robust metric.

Coverage is a widespread metric, but Mutation Testing is not so much. I only know a handful of projects that are using it in production (if you know any company that uses it, I would love to know about it to have more data) and that’s why I thought I would try to use it in a production project.

And that’s what I’m going to tell you:

Mutation testing in Kotlin

What did I use?

• PIT Mutation Testing
• jUnit
• Gradle with Kotlin

Configuration

plugins {
...
	id("info.solidsoft.pitest") version "1.7.0"
...
}
...
pitest {
    setProperty("junit5PluginVersion", "0.12")
    setProperty("testPlugin", "junit5")
    setProperty("targetClasses", listOf("target package"))
    setProperty("outputFormats", listOf("HTML"))
}

Execution

To execute it by command line

./gradlew pitest

With this configuration, we will get console information with data, such as the percentage of mutants captured by our tests.

Detailed Report

If we want a detailed and more visual report of surviving mutants (indicating exactly in which class and which line of code) we will have to add the following configuration:

pitest {
	setProperty("junit5PluginVersion", "0.12")
	setProperty("testPlugin", "junit5")
	setProperty("targetClasses", listOf("target package"))
	setProperty("outputFormats", listOf("HTML"))
}

Tricks to improve the execution performance

The execution will take a while ⌛ , to improve that time we can modify the configuration to use more than one thread and reuse previous executions, like this:

pitest {
	setProperty("junit5PluginVersion", "0.12")
	setProperty("testPlugin", "junit5")
	setProperty("targetClasses", listOf("target package"))
	setProperty("outputFormats", listOf("HTML"))
	setProperty("threads", 2)
	setProperty("withHistory", true)
}

With all this, we will get a Gradle task that will not fail even if the percentage of mutants we capture is 0 🧐. To make the task fail if our percentage is lower than a given data, we will have to have a configuration like this:

pitest {
	setProperty("junit5PluginVersion", "0.12")
	setProperty("testPlugin", "junit5")
	setProperty("targetClasses", listOf("target package"))
	setProperty("mutationThreshold", 40)
	setProperty("outputFormats", listOf("HTML"))
	setProperty("threads", 2)
	setProperty("withHistory", true)
}

And here we stay for now 😊 although I don’t want to finish without thanking Vero and Mavi for their contributions to building this section.

Also a huge thanks to Cris for helping me review the post.