Mutation Tests with Pitest

Photo by Braňo on Unsplash

Brief

At this article I wanna explain a little about about the importance of tests over an application, and after I’ll focus in a specific test type called mutation tests. And how example I’ll use the Pitest a Java framework.

A little about test

I believe that a test is a think auto descriptive. But in short words it’s how certificate if your application is doing what is expected. So, from this point we can deep into more details about tests. The software industry follows a pyramid test that shows from where we should start and in theory end, in theory because, exists much test types and it’s very hard to mantain an application with much tests, mainly if there’re more tests than unit tests. Remember that each behavior changed at application you’ll to change tests too. So not only when increase tests of service or ui, the development stay slower, but, it can happen with unit tests when occurs scope changes frequently. Bellow an example of pyramid test:

From: https://martinfowler.com/bliki/TestPyramid.html

Looking for the picture above we can see that unit tests are faster because your scope is restrict to a function in the major of cases. And to build a good unit test you can follow the pattern Triple A(Arrange, Act, Assert), where basically you configure fake objects, then call the method and do assertions to make sure that the method is executed as expected.

Mutation Test

Mutation tests comes to improve your code and your unit test, it’s because, it changes conditions, returns and many other features in your source code and one time that changes your code, the idea is that your test braek too. It’s the real guarantee that source code and test it’s alright. A example is generate a report and see that every tests ran 👌

Report from Junit

But when we run the mutation tests we saw an unexpected result like:

Report overview

Pitest package summary

Class report

Looking for reports above really appears that are something wrong with our tests, so we now we can have an overview about Pitest.

Pitest

It’s a framework to do mutation tests and it’s simple to configurate and easy to use.
All of operators are enable by default, but you can refine according with your necessities. The tests that passed by a mutation can have the following states: killed, survived, no covered, non viable, time out, memory error, run error. A test is considered strenght if a test is killed or in other words failed.
A test is considered weak when a test stay passing state lived or when your state turn a time out(probabilly your code fall into an infinit loop). In some cases occurs that Pitest cannot generate mutation into JVM so, it’s stay with state no viable. Some tests can fall in a memory error where, it’s possible that heap and permgem it’s not enought to run the tests, at this cases you could or increase heap and permgem space or decrease the scope of your tests. Like a configuration in unit tests. In this case you could use the tags from Junit 5(if you are using it) that says at Junit User Guide:

Used to declare tags for filtering tests, either at the class or method level; analogous to test groups in TestNG or Categories in JUnit 4. Such annotations are inherited at the class level but not at the method level.

Some examples of mutators:

CONDITIONALS_BOUNDARY

Changes source code from:

if ( i >= 0 ) {
    return "foo";
} else {
    return "bar";
}

To:

if ( i > 0 ) {
    return "foo";
} else {
    return "bar";
}

EMPTY_RETURNS

Changes source code from:

    public Set<String> listAllStudentsByName() {
        return students.stream().map(Student::getName).collect(Collectors.toSet());
    }

To:

public Set<String> listAllStudentsByName() {
      return new HashSet<String>();
}

Also it provides other mutators like: FALSE_RETURNS, INCREMENTS, INVERT_NEGS, MATH, NEGATE_CONDITIONALS, NULL_RETURNS, PRIMITIVE_RETURNS, TRUE_RETURNS

Add Pitest to a pipeline

If you thought in put a step into a pipeline to execute when occurs a change in your code during a pull request, so you’ll must use the arcmutate 😒.

Arcumate have a good documentation to put in a pipeline, but it’s paid or you can use for 30 days. And the license value to use is increased by team size…

Final Toughts

It’s a amaizing type of test and framework. But, there’s the point to put in a pipeline and generate cost, so to put in a pipeline depends from your budget. If you’ve it’s a good way, think that doesn’t have much to be configured. Anyway you can create a profile or tag or use another approach to use at least locally, it’s a first step. And ever you want you can get main branch and run the tests. Here there’s a github repository with a sample configured: https://github.com/ivanzito/test-samples/