JavaScript Mutation Testing

Test the quality of your tests using Stryker

Olle Lauri Boström
Aug 2, 2018 · 4 min read

This summer I have been working on a Google Summer of Code project with the goal of increasing the test coverage and the test quality in an open source project called WebAssembly Studio.

“WebAssembly.Studio is an online IDE (integrated development environment) that helps you learn and teach others about WebAssembly. It’s also a Swiss Army knife that comes in handy whenever working with WebAssembly.” — Michael Bebenita

When I started working on my project in late May, the overall test coverage was at around 20% meaning several thousands uncovered lines of code. Adding tests to an existing code base can be challenging and while measuring test coverage can help you find untested code, it does not tell you much about the actual quality of your tests.

In this article I will talk a bit about JavaScript mutation testing, how it has been useful for me during my GSoC project and how it can help you to analyze your tests to make them more effective in catching bugs.

What is mutation testing?

Mutation testing is basically testing the quality of your tests to see if they are effective in catching bugs. Consider the assert function below.

To figure out if any tests that are covering the assert function is truly effective, we can change (mutate) certain statements in the source code to introduce bugs (called mutants). In this case, we reverse the boolean expression in the if-statement to introduce a bug.

We can then execute the tests to find out how effective they really are. If they are all still passing — that means the mutant has survived and that your tests did not catch the introduced bug. Are there any failing tests? Great, that means the mutant was killed and that your tests did catch the bug. The better your tests are, the less mutants will survive.

“The only way to know that a test actually works is when it fails when you make a code change. — Simon de Lang

We need a framework for this

As shown by the simplified example above, this process can be done manually by making small changes in your source code and then running your tests. However, this would soon prove to be very time consuming. We need a mutation testing framework that can automate this process. A mutation testing framework will automatically go through your source code, create mutants where possible and then execute your tests for each of the introduced mutants.

Introducing Stryker

Mutation testing is still a relatively new thing in the JavaScript world. Stryker is one of few (possibly the only) actively maintained JavaScript frameworks for performing mutation testing. It’s open source and has support for a bunch of different test runners and test frameworks such as Jest, Mocha, Jasmine and Karma. It also comes with plugins for Webpack, Babel and TypeScript support.

Adding Stryker to the WebAssembly Studio project (that uses React, TypeScript, Webpack and Jest) worked like a charm. Let me know if you need any help with a similar setup or ask for help over at gitter.

Code Search icon by Pravin Unagar, Search icon by Hassan ali and Bug icon by Ben Davis (from the Noun Project)

A more in-depth introduction to Stryker can be found at stryker-mutator.io

But wait, what about test coverage?

Don’t get me wrong, measuring test coverage is great — it can help you find untested parts of your code. Test coverage does however not tell you much about the actual quality of your tests or give any sort of indication when they are good enough. Even if you reach 100% test coverage, that basically only tells you that all lines where executed during the test run — not that they actually work as expected. Mutation testing helps you to verify that your assertions are good and points out if you have missed anything important.

An example

Lets look at an example where test coverage is at 100% but where we are still missing some test cases.

Checks if a number is in a range

The following tests suite for the inRange function gives us 100% coverage. But is this really good enough?

The tests suite gives us 100% coverage

If we mutate the above example using Stryker, two mutants will actually survive. Since we do not assert on the boundary values, the test suite would still pass if someone would change for instance >= to >. Things like this is pretty easy to miss when writing tests, making Stryker the perfect tool for ensuring that you don’t miss anything.

Finally, lets make it not only 100% covered, but also 100% mutant free by adding two more test cases.

Improved test suite after running mutation testing

How has this been useful?

Using Stryker has proven to be really useful for me during my GSoC project. A majority of the tests I’ve written has been added to cover existing code written by other contributors. By continuously running mutation testing using Stryker I have been able to know when my tests are good enough and prove to myself that they are effective in catching bugs.

Some final thoughts:

  • Mutation testing helps you to know when tests are good enough. This is especially useful when you are writing tests for advanced functionality that others may have written.
  • Mutation testing helps you prove that your tests are effective.
  • Mutation testing makes you think about your assertions. Do you assert that all important functionality actually works as expected? Test coverage only lets you know what parts of the code you are executing during a test run.
  • If the project you are working on is extensive (like WebAssembly Studio), mutating the whole project will generate a lot of mutants and naturally take a really long time to finish. Consider mutating only a single/a few source code files that are related to your changes.

Can’t wait to get started?

Head over to Strykers Quickstart guide to get started killing some mutants. Leave a comment here if you need any help setting this up in your project, or hit me up over at gitter.

Olle Lauri Boström

Written by

Google Summer of Code 2018 @ WebAssembly Studio - github.com/ollelauribostrom

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade