Test the quality of your tests using Stryker
“WebAssembly.Studio is an online IDE (integrated development environment) that helps you learn and teach others about WebAssembly. It’s also a Swiss Army knife that comes in handy whenever working with WebAssembly.” — Michael Bebenita
When I started working on my project in late May, the overall test coverage was at around 20% meaning several thousands uncovered lines of code. Adding tests to an existing code base can be challenging and while measuring test coverage can help you find untested code, it does not tell you much about the actual quality of your tests.
What is mutation testing?
Mutation testing is basically testing the quality of your tests to see if they are effective in catching bugs. Consider the assert function below.
To figure out if any tests that are covering the assert function is truly effective, we can change (mutate) certain statements in the source code to introduce bugs (called mutants). In this case, we reverse the boolean expression in the if-statement to introduce a bug.
We can then execute the tests to find out how effective they really are. If they are all still passing — that means the mutant has survived and that your tests did not catch the introduced bug. Are there any failing tests? Great, that means the mutant was killed and that your tests did catch the bug. The better your tests are, the less mutants will survive.
“The only way to know that a test actually works is when it fails when you make a code change. — Simon de Lang”
We need a framework for this
As shown by the simplified example above, this process can be done manually by making small changes in your source code and then running your tests. However, this would soon prove to be very time consuming. We need a mutation testing framework that can automate this process. A mutation testing framework will automatically go through your source code, create mutants where possible and then execute your tests for each of the introduced mutants.
Adding Stryker to the WebAssembly Studio project (that uses React, TypeScript, Webpack and Jest) worked like a charm. Let me know if you need any help with a similar setup or ask for help over at gitter.
A more in-depth introduction to Stryker can be found at stryker-mutator.io
But wait, what about test coverage?
Don’t get me wrong, measuring test coverage is great — it can help you find untested parts of your code. Test coverage does however not tell you much about the actual quality of your tests or give any sort of indication when they are good enough. Even if you reach 100% test coverage, that basically only tells you that all lines where executed during the test run — not that they actually work as expected. Mutation testing helps you to verify that your assertions are good and points out if you have missed anything important.
Lets look at an example where test coverage is at 100% but where we are still missing some test cases.
The following tests suite for the inRange function gives us 100% coverage. But is this really good enough?
If we mutate the above example using Stryker, two mutants will actually survive. Since we do not assert on the boundary values, the test suite would still pass if someone would change for instance
>. Things like this is pretty easy to miss when writing tests, making Stryker the perfect tool for ensuring that you don’t miss anything.
Finally, lets make it not only 100% covered, but also 100% mutant free by adding two more test cases.
How has this been useful?
Using Stryker has proven to be really useful for me during my GSoC project. A majority of the tests I’ve written has been added to cover existing code written by other contributors. By continuously running mutation testing using Stryker I have been able to know when my tests are good enough and prove to myself that they are effective in catching bugs.
Some final thoughts:
- Mutation testing helps you to know when tests are good enough. This is especially useful when you are writing tests for advanced functionality that others may have written.
- Mutation testing helps you prove that your tests are effective.
- Mutation testing makes you think about your assertions. Do you assert that all important functionality actually works as expected? Test coverage only lets you know what parts of the code you are executing during a test run.
- If the project you are working on is extensive (like WebAssembly Studio), mutating the whole project will generate a lot of mutants and naturally take a really long time to finish. Consider mutating only a single/a few source code files that are related to your changes.