A complete guide on Mutation Testing

This article was originally published by Sylvain Tiset’s article : A complete guid on Mutation Testing

Mutation testing is a type of software testing that is used to design new software tests and evaluate the quality of existing software tests. Let’s see why it’s needed, what is it and how to use it in a project.

Mutation testing (generated by Microsoft Bing AI)

Isn’t Unit Tests enough?

Unit testing is the process where you test the smallest functional unit of code. With the help of unit tests, the program under tests is less likely to be on errors.

With many unit tests, we are able to calculate the code coverage, a metric that can help us understand how much of the source code is tested. It’s a very useful metric that can help assess the quality of the test suite.

The remaining question is : Is a unit test enough to ensure the absence of bugs in the tested function?

Let’s look at an example:

public class User {

public static int LAWFUL_AGE = 18;

   private int age;

   public User(int age) {
       this.age = age;
   }

   public int getAge() {
       return age;
   }

   public boolean isLawfulAge() {
       return age >= LAWFUL_AGE;
   }
}

The User class has one method that is tested in this UnitTest class:

class UserTest {

  @Test
   void isLawfulAge_WhenIsLawfulAge_ShouldReturnTrue() {
       // Arrange
       User user = new User(18);
       
       // Act
       boolean isLawfulAge = user.isLawfulAge();
       
       // Assert
       Assertions.assertThat(isLawfulAge).isTrue();
   }

   @Test
   void isLawfulAge_WhenIsUnderAge_ShouldReturnFalse() {
       // Arrange
       User user = new User(15);
       
       // Act
       boolean isLawfulAge = user.isLawfulAge();
       
       // Assert
       Assertions.assertThat(isLawfulAge).isFalse();
   }
}

Code coverage will return 100% on isLawfulAge() method. But what happens if the method is modified this way?

public boolean isLawfulAge() {
     return age == LAWFUL_AGE;
 }

Tests are still green! A mutation has been introduced in the code and the tests didn’t catch it.

That’s where mutation testing comes from.

What is mutation testing?

Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behaviour of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill, it’s the mutation score.

Mutation testing by Geeksforgeeks

Purposes

The objectives of mutation testing are the following:

• To identify pieces of code that are not tested properly.
• To identify bugs or hidden defects that can’t be detected using other testing methods.
• To assess the quality of the test cases by calculating the mutation score.

Types

There are 3 types of mutation testing:

• Statement mutation
• Value mutation
• Decision mutation

Statement mutation

During a statement mutation a statement is deleted or replaced by some other element.

// Original program
if (a < b) c = 10;
else c = 20;

// Mutant program 1: the else statement has been completely removed
if (a < b) c = 10;

// Mutant program 2: assignement variable has been changed
if (a < b) d = 10;
else d = 20;

Value mutation

During a value mutation, the values of variables are changed in order to detect potential errors in the program.

// Original program
function multiplyByTwo(value) {
  return value * 2;
}

// Mutant program 1: Small value to higher value
function multiplyByTwo(value) {
  return value * 10;
}

// Mutant program 2: Higher value to small value
function multiplyByTwo(value) {
  return value * 0.5;
}

Decision mutation

During decision mutation, logical or arithmetic operators are changed.

// Original program
bool isPositive(int number) => number > 0;

// Mutant program 1: Changing comparison operator
bool isPositive(int number) => number >= 0;

// Mutant program 2: Negative the result
bool isPositive(int number) => !(number > 0);

Pros and Cons

• It brings a good level of error detection in the program
• It helps developers to write better test cases
• It is costly and time-consuming

How to use mutation testing in a project

It will be very time consumming to modify manually every code under test to see if the tests are robust or not. Fortunetely, tools exist and can help us save time to automatise the run of the mutation tests.

Tools for mutation tests

Here are some tools for mutation testing, depending on which programming language is the project:

• Pitest — Java, Kotlin
• Stryker-mutator — Javascript, C# and Scala
• Mutates — Python
• Infection — PHP

An example of use with Pitest / Stryker

Let’s take the example of michaelpmcmillan’s repository:

git clone https://github.com/michaelpmcmillan/stryker-mutator-hello-world.git
cd ./stryker-mutator-hello-world/dotnet/StrykerDemo.Tests
dotnet tool restore

dotnet test --collect:"XPlat Code Coverage"
dotnet reportgenerator -reports:./TestResults/**/*.xml -targetdir:./TestResults/CoverageReport/ -reporttypes:Html

dotnet stryker

The example has the following method with the associated unit tests:

public string Categorise(int ratingScore)
{
    if(ratingScore < 0) return Rating.OutOfBounds;
    if(ratingScore < 3) return Rating.ReallyBad;
    if(ratingScore < 5) return Rating.FairlyBad;
    if(ratingScore < 7) return Rating.FairlyGood;
    if(ratingScore <= 9) return Rating.ReallyGood;
    if(ratingScore == 10) return Rating.Perfect;

    return Rating.OutOfBounds;
}

[Theory]
[InlineData(-2, Rating.OutOfBounds)]
[InlineData(2, Rating.ReallyBad)]
[InlineData(4, Rating.FairlyBad)]
[InlineData(6, Rating.FairlyGood)]
[InlineData(8, Rating.ReallyGood)]
[InlineData(10, Rating.Perfect)]
[InlineData(20, Rating.OutOfBounds)]
public void When_Categorise_ThenMapRatingToResult(int rating, string expectedCategorisation)
{
    string result = new ScoreCategoriser().Categorise(rating);
    result.Should().Be(expectedCategorisation);
}

As you can see, the code coverage is 100%. Good. Now let’s use the mutation testing to see how robusts our tests are.

Stryker report

As you can see now, the mutation score is only of 68.75%. Looking in details in each row, we are able to see which mutation wasn’t killed. For instance, the first test is weak to equality mutation. It’s easy to solve it by adding a unit test case:

[InlineData(0, Rating.ReallyBad)]

Running Stryker again, we now have a 75% mutation score.

Stryker report

Thanks for reading this article, hoping you learned something on mutation testing. If you liked the content, feel free to let claps, comments or to share the article.