Mutation Testing : The Most Comprehensive Way to Test Your Software
Change is inevitable, therefore it’s very rare that you would write some piece of code and never touch it again. As the requirements change, so will your code.
Testing your software is a great way of knowing whether or not your code behaves as expected. Tests are there to ensure you will catch unintended results of future modifications as you continue down the development process. But tests are also written by people and people make mistakes. Even if you write a high amount of tests that seem to check every single case, it’s unrealistic to believe that you’ve actually checked every single case.
Coverage is a metric that is widely used to determine what percentage of your code has been tested by a test suite. High coverage is interpreted as it’s less likely to have an undetected bug somewhere in your code. While coverage is important, it only tells you the percentage of code that is executed by your tests, which is usually not enough to determine test quality.
Mutation Testing is a type of white box testing that can give you a pretty good idea of the quality of your tests. The whole point of mutation testing is to show if your tests are able to detect subtle errors that may be introduced in the future. Mutation testing frameworks take your original source code and introduce some modifications to it, called mutations. Unit tests are run against both the original source and the mutated version and results are compared.
All tests pass after mutation → Mutation survived (Whoops!)
One or more tests fail after mutation → Mutation killed (Yay!)
Mutation Operators
For every mutation, a mutation operator is applied to a piece of code. Mutation operator performs a slight modification on code in order to change its behavior. Here are some common mutation operators:
- Replace boolean expression with true or false
- Replace the return value with a constant expression
- Negate a conditional check (convert less than to greater then)
- Replace arithmetic operators with others.
Mutations are usually introduced to code one by one rather than multiple of them at the same time. The reason for this is that multiple mutations may render the code completely irrelevant. Also it’s possible that one mutation may accidentally kill another mutation, leading to passing tests when they should not have.
Mutation Testing Frameworks
Since it would be tedious to do this manually, we have frameworks to automatically embed mutations and run all our tests for us. They also provide a detailed analysis of the results, including the amount of mutations, percentage of killed mutations, which mutations were introduced to which parts of the code, etc.
In order to demonstrate the process with an example, I will use PITest in Java. It’s an easy to use open-source library developed by Henry Coles. Here are some of the frameworks you may use in various programming languages:
Mutation Testing with PITest
PITest may be used as a plain command line tool but it also has great support for dependency frameworks such as Gradle and Maven. I will use it in a Maven project but you may also follow PITest’s official documentation to see how it works with other dependency managers.
Grab PITest Plugin and JUnit Dependency.
MVNRepository is a great place to pick up dependencies of external libraries. It contains dependencies for other dependency managers as well, not just Maven. Let’s get PITest 1.5.0 plugin and JUnit.
Step 1 — Write some code and unit tests for it
I wrote a simple piece of code that simply checks if a voltage level is dangerous or not. Dangerous voltage levels may be different depending on the type of application, so this field is configured through VoltageLevelAnalyzer’s constructor.
Now let’s write a simple test case that checks if isDangerous actually returns true when given voltage value is higher than the lower limit:
Step 2 — Test the tests with PITest
Now that we have a test case, let’s use PITest’s mvn plugin to generate mutation coverage:
mvn org.pitest:pitest-maven:mutationCoverage
This command will fail if some of your tests are already failing before mutations, if not, it will generate a detailed report in the form of html. Reports are generated under:
./target/pit-reports/timestamp
./target/pit-reports/timestamp/packagename/
Let’s take a look at the VoltageLevelAnalyzer.java.html under packagename folder:
Yikes! Looks like 4 mutations have been introduced to our code and our test was only able to kill one of them. As you can see, report shows which type of mutation operator was applied to which line and whether it survived or not. Let’s go over each mutation:
- Mutation 1 — getDangerousVoltageLowerLimit’s return value was replaced with 0 but since we didn’t test this method, it had no coverage.
- Mutation 2 — isDangerous method’s return value was replaced with true. Since we only tested with a value that is higher than the voltage limit, this mutation survived.
- Mutation 3 — isDangerous method’s >= operator was changed (equal part of the operator was removed). This mutation survived because we didn’t write a test to check the edge case where voltage level is equal to the lower limit.
- Mutation 4 — >= operator was negated (converted to <=). We managed to kill this mutation since our one test covers it.
Let’s write some more tests to fight these mutations!
Running the mutation analysis again, we can see that all mutations are now killed:
Disadvantages of Mutation Testing
Most of the disadvantages are specific to larger projects. If the original program is fairly big, then the number of generated mutants are going to be extremely high. It may take a long time to generate these mutants in order to perform mutation testing and after it’s done, killing all mutants may not be feasible especially if you’re adapting this method to an already evolved code-base.
Conclusion
You may think of mutation testing as an additional layer of safety measure. On top of your line coverage, mutation coverage can be used to improve the quality of your tests. It gives you a guideline to shape your unit tests.
Does that mean if you kill all mutations, your software is bug-free? Of course not. Mutation testing may help you catch some bugs before they can make it into production, but it doesn’t guarantee a bug-free system. It’s simply a way of gauging how good your test suite is at detecting anomalies, and it does that by assuming your code is logically correct. Regardless, the most important thing is to add as many of these technologies as possible to your arsenal in order to minimize potential headaches. Writing perfect software is a myth. Perfection is an ever-moving goal and all we can do is pursue it by improving our code and mitigating potential side-effects as much as possible.
