Testing your tests: PHP Mutation Testing with Infection Framework

How reliable are the tests we write? Are they covering the edge cases? What happens when the code changes? Do the tests still pass?

Welcome to testing our tests. Of course to make any tests reliable it has to cover all the edge cases that an application may find itself in. Testing both the correct input flow and wrong input flow, throwing unexpected inputs, dealing with exceptions and security issues come into the picture. All these steps and thought processes help us write good tests and various test cases to cover it all.

But what if we could automate at least some of that burden of testing our tests? Enter Mutation Testing. Mutation testing exactly does what we need. It modifies the code at runtime and then runs our tests to see if they really cover the modifications. If the tests still pass then we have some holes in our tests.

One of the tools for this is: Infection (https://infection.github.io/)

Infection uses Abstract Syntax Tree (AST) mutations to change the code at runtime.

Let’s go ahead and see what it does.

You can install Infection in diffrent ways as shown here: https://infection.github.io/guide/installation.html

But for the purpose of this article, we’ll install it using composer globally.

Run the following command through your terminal. Make sure you have composer installed in your pc first. If not you can get it through: https://getcomposer.org/download/

composer global require infection/infection

Also do not forget to include the path to ~/.bash_profile (or ~/.bashrc) if you are on linux.

export PATH=~/.composer/vendor/bin:$PATH

For windows the path is added automatically when composer is installed.

Now to test the infection framework, let’s setup a new laravel project.

I did mine on C:\laragon\www\laravel.

Then let’s run “infection” command on the same directory.

As shown in the screenshot it’ll ask you to configure it for the first time.

I chose to test it for “app” folder.

Now it’ll ask you a bunch of questions. If you’d like to exclude any directories inside the “app” directory you selected you can do so. For now we’ll just skip that. Timeout by default is 10 which is fine for us. Then finally the log file path, for now root is fine.

It then creates a configuration file based on our choices.

Now it runs the initial tests, processes the source code inside our “app” directory and then generates mutants.

Mutants are the modified instances of our code that the framework will test against our tests.

For me with Laravel I got the “Class CreateUsersTable does not exist” error while it was processing code files. It seems to have some issue with Laravel’s migration file. But luckily it only happens for the first time. Next time you run “infection” it works fine.

Here we see that 30 total mutations were generated.

7 killed, which means the tests covered the mutated codes.

16 were not covered by tests. This is obvious because we haven’t written test cases for them at all. Laravel only ships with two default test cases. 7 more mutants escaped the detection. Those ones show that our tests are not taking into account those cases.

Let’s look at the generated report in detail. Let’s open the file “infection-log.txt” that is in our root folder.

We can see mutation types and the actual mutation done line by line.

Some mutation types we can see are:

[M] MethodCallRemoval

[M] ProtectedVisibility

[M] FunctionCall

MethodCallRemoval removes method call from a single line and then runs our tests.

ProtectedVisiblity changes the method visibility from public to protected then runs our tests.

FunctionCall removes the function call from a line and then runs our tests.

List of all the available mutators are given on the official documentation here: https://infection.github.io/guide/mutators.html

So we can see, that it help us automate a lot of work that we do manually when we write tests with good coverage. These kinds of feedback about our tests help us write better tests. They also help to figure out which type of test cases we are missing in our suite.

I hope you enjoyed learning about Mutation testing in PHP. If you did, please subscribe for more! And please give some claps for this article :) How about a share to facebook and or twitter too? ;) It will help spread the information to other people who may need it. Thanks!

Don’t forget to check out my other series in medium:

PHP Test Driven Development Part 1: Introduction

PHP Software Architecture Part 1: MVC

Cheers :)