PIT Mutation Testing on CI/CD Pipeline
We as the Search team at Trendyol, have the motivation to switch to a practice that we can perform one-click deployments. Our CI/CD Pipelines should be very strong and reliable in order to apply a one-click deployment model so that we are constantly taking new steps to improve our CI/CD pipelines.
In this article, I would like to give details about a fault-testing tool that we started to use within Java projects at Search Team. PIT Mutation Testing is an open-source fault-finding software testing tool for Java that introduces bugs into the code on purpose and then checks if the test suite can find them or not. PIT is created by Henry Coles, a software developer based in Edinburgh, but has benefited from contributions from numerous others.[4]
Currently, we are using methods such as code review and code coverage to increase the quality of our APIs, but sometimes these methods could be insufficient.
What’s wrong with code coverage?
With the help of code coverage tools, we can check whether the code we have written is covered by any of the tests or not but we may not be sure how valid and complete the tests are. Let’s check with an example of how code coverage can be insufficient in some cases.
Here we have a method basically returning a boolean regarding our parameter could be equal to, bigger than, or smaller than 5.
In the first test method as shown below, assume that we forgot to add an assertion for the use case when x=5.
In the second test method, assume that we’ve covered all of the use cases but we forgot to add assertions.
For the executions of both test methods above, the code coverage will be %100 which is misleading. It is clear that we need a more efficient method to make sure that tests are written properly.
What is PIT test and how it works?
Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation continues to live. The quality of your tests can be gauged from the percentage of mutations killed. [5]
PIT runs your unit tests automatically against modified versions of your application code. When the application code changes, it should produce different results and cause the unit tests to fail. If a unit test does not fail in this situation, it may indicate an issue with the test suite. [5]
Mutant Creation Process: The PIT Testing tool introduces bugs on purpose by directly modifying the byte code and creates Mutants.
- Mutation Operators/mutators: A mutator is an operation applied to the original code. Basic examples include changing a
'>'operator by an'<', replacing'and'by'or'operators, and substituting other mathematical operators for instance. [1] - Mutants: A mutant is a result of applying the mutator to an entity (in Java this is typically a class). A mutant is thus the modified version of the class, that will be used during the execution of the test suite.[1]
Test Execution on Mutants: After the mutant creation process, the PIT Test runs only the tests that may kill the mutants. It does not run all of the test cases on mutants since PIT is aware of which test belongs to which code according to the coverage data.
Line coverage identifies code that is definitely not tested. If there is no test coverage for the line of code where a mutant is created, then none of the tests in the suite can possibly kill it. Pitest can mark the mutant as surviving without doing any further work.[4]
Mutations killed/survived: When executing the test suite against mutated code, there are 2 possible outcomes for each mutant: the mutant is either killed, or it has survived. A killed mutant means that there was at least 1 test that failed as the result of the mutation. A survived mutant means that our test suite didn’t catch the mutation and should thus be improved.[1]
Type of the Mutators (Default Group)
There are different groups of Mutation Operators (Defaults, Stronger, All) which have a different number of mutation operations. In our team, we started using the “Defaults” group which has mutation operators below[2];
- Conditionals Boundary Mutator — The conditionals boundary mutator replaces the relational operators with their boundary counterpart. (like <,<=,>, >=)
- Increments Mutator — Mutate increments, decrements and assignment increments, decrements of local variables (stack variables). It will replace increments with decrements and vice versa.(like i++, i--)
- Inverts Negative Mutator — Inverts negation of integer and floating-point numbers. (like 1, -1)
- Math Mutator — Replaces binary arithmetic operations for either integer or floating-point arithmetic with another operation. (like +, -, /, *)
- Negate Conditionals Mutator — Mutate conditionals (like != with ==)
- Void Method Call Mutator — Removes method calls to void methods.
- Empty Returns Mutator — Replaces return values with an ‘empty’ value for that type.
- False Returns Mutator — Replaces primitive and boxed boolean return values with false.
- True Returns Mutator — Replaces primitive and boxed boolean return values with true.
- Null Returns Mutator — Replaces return values with null.
- Primitive Returns Mutator — Replaces int, short, long, char, float and double return values with 0.
Initial Configuration
Let’s make the initial configuration in pom.xml to enable Pitest in our project.
- Add “pitest-maven” and “maven-scm-plugin” dependencies.
2. Add SCM configuration including your repo, path and API information. (SCM configuration is to detect the diff between the branches to run PIT test only for the changed code, will be described in detail later)
3. Add the “pitest-maven” plugin to the build plugins section.
Here are the explanations of the configurations that we use to provision and optimize the PIT test execution for our projects.
- targetClasses — Source code path to be mutated.
- targetTests — Test classes path that will be executed on created mutants.
- outputFormats — XML for sonarQube, HTML to view the report on local.
- threads — The number of threads to use when mutation testing. By default, a single thread will be used.[3]
- avoidCallsTo — This is to state the packages that will be excluded by mutation analysis.
- excludedClasses — Matching classes will be excluded from mutation[3].
- maxDependencyDistance — PIT can optionally apply an additional filter to the supplied tests, such that only tests a certain distance from a mutated class will be considered for running. e.g A test that directly calls a method on a mutated class has a distance of 1, a test that calls a method on a class that uses the mutated class as an implementation detail has a distance of 2, etc.[3]
With the above 3-step configurations, Pitest would be enabled for the project.
Running PIT test at Local
This step is for running Pitest locally before the newly added code is pushed to the remote repository. Add a file named “pitest.sh” including the below script to the root directory of your project.
With the above script, there will be two options that can be run at a terminal;
- Running command “./pitest.sh”
This command will execute the script in the “pitest.sh” file which;
- Finds the code change between the origin(current branch) and the destination branch(in our script that is stated as origin/develop)
currentBranch=$(git rev-parse --abbrev-ref HEAD)- Runs pitest on the changed classes by using scmMutationCoverage goal.
mvn clean install org.pitest:pitest-maven:scmMutationCoverage
-DoriginBranch=${currentBranch}
-DdestinationBranch=origin/develop
-Dinclude=ADDED,MODIFIED
-DtimestampedReports=false- The HTML report file will be opened in the default browser when the execution completed.
open target/pit-reports/index.html
2. Running command “./pitest.sh all”
This command includes the ‘all’ parameter and will execute the other script in pitest.sh.
- Runs pitest for all project code by using mutationCoverage goal. Please note that the first execution will take a long time but the subsequent executions will be much faster with the help of the “-DwithHistory” parameter.
mvn clean install org.pitest:pitest-maven:mutationCoverage
-DwithHistory
-DtimestampedReports=false- The HTML report file will be opened in the default browser.
open target/pit-reports/index.html
Run PIT test on Pipeline for the changed code only
We are using GitLab to run a pipeline for our projects. When a code is developed on a feature branch and pushed to the remote repository, the pipeline builds the feature branch code, dockerizes it, and deploys it to the stage K8S. During this flow;
We run and report Pitest at the “Feature Build” stage, then the “Sonar Check” stage will check the gate and break the pipeline if the PIT results not OK as shown below.
Feature Build Stage
Feature Build stage that is located in gitlab-ci.yml includes 3-steps to run Pit test on only changed code.
- Install git, fetch remote branches (You may prefer to put these into your build docker image)
apt-get update -qy && apt-get upgrade -qy
apt-get install -y git
git fetch --all- Run Pitest by using the scmMutationCoverage goal. ScmMutationCoverage goal will find the code diff and then run Pitest for that changed code.
mvn clean install org.pitest:pitest-maven:scmMutationCoverage
-DoriginBranch=origin/$CI_COMMIT_BRANCH
-DdestinationBranch=origin/develop
-Dinclude=ADDED,MODIFIED
-DtimestampedReports=false- Run Sonar to report. “-Dsonar.pitest.mode=reuseReport” should be added for sonar to report PIT test output to SonarQube.
sonar:sonar -Dsonar.projectKey=$CI_PROJECT_NAME:$CI_COMMIT_REF_NAME -Dsonar.projectName=$CI_PROJECT_NAME:$CI_COMMIT_REF_NAME
-Dsonar.host.url=$SONARQUBE_ADDRESS
-Dsonar.login=$SONARQUBE_TOKEN
-Dsonar.pitest.mode=reuseReportSonar Check Stage
We have already run “mvn clean install”, then run pitest and report the results to SonarQube in the “Feature Build” Stage. In the “Sonar Check” stage, Sonar will check whether the code coverage and the PIT test results are OK or not.
Please note that pitest bugs decrease “Reliability” and “Maintainability” metrics in SonarQube so that defining quality gates for these metrics will be enough to break the pipeline.
As shown below, the Quality Gate failed since the pit test bug decreased Reliability from A to C, decreased Maintainability from A to B.
Run PIT test on Pipeline for all project code
The key point running PIT testing for all project code is to create a branch having a name that starts with “pitest-” and push the code to the remote repository. “Project Pitest” stage is triggered for only branches starting with “pitest-”. We run and report Pitest at the “Project Pitest” stage, then the “Sonar Check” stage will check the gate and break the pipeline if the PIT results not.
Project Pitest Stage
This stage includes 2 steps to run Pitest for all project codes.
- Runs the mutationCoverage goal.
mvn clean install org.pitest:pitest-maven:mutationCoverage- Runs Sonar to report. “-Dsonar.pitest.mode=reuseReport” should be added for sonar to report Pitest output to SonarQube.
sonar:sonar -Dsonar.projectKey=$CI_PROJECT_NAME:$CI_COMMIT_REF_NAME -Dsonar.projectName=$CI_PROJECT_NAME:$CI_COMMIT_REF_NAME
-Dsonar.host.url=$SONARQUBE_ADDRESS
-Dsonar.login=$SONARQUBE_TOKEN
-Dsonar.pitest.mode=reuseReportSonar Check Stage
During the build operation, we run “mvn clean install”, then run pitest and report the results to sonarQube. Here we check whether the code coverage and the PIT test results are OK or not.
Mutation Analysis results can be seen under the Measures tab, on the Mutation Analysis section in SonarQube.
Configuring SonarQube for PIT testing
The default Java profile(Sonar Way) does not include Mutation Analysis rules. To be able to see the PIT reports of feature branches, the default Java profile should be updated to a new profile including Mutation Analysis rules.
- Install Mutation Testing Plugin from Update Center
- Extend Sonar Way profile, rename as “Sonar Way + Mutation Analysis”, add Mutation Analysis Rules to this newly added profile
- Make the new profile as the default
Bonus — Performance
Mutation testing is a computationally expensive process and can take quite some time depending on the size of your codebase and the quality and speed of your test suite.[6]. You can speed up the execution by;
Increasing the thread count — Configure the thread count in the pitest-maven plugin in pom.xml. (Default is 1, limited with the number of the CPUs)
<threads>2</threads>We do not increase the thread count as of now since Pitest execution on only changed code takes 2 min 44 sec for the codebase mentioned in Chart1 and takes 15 sec for the codebase mentioned in Chart2 which both are acceptable for us. We may increase up to 2 threads in the future.
Please note that using more threads will lower the execution time but will increase the heap size needed since mutants will be created and loaded into different class loaders to prevent conflicts on parallel executions. (PIT means Parallel Isolated Tests). In the chart1 below, you can see that maven heap size should be configured as 4096m for 4 threads execution which is not acceptable.
Setting appropriate maxDependecyDistance value— PIT can optionally apply an additional filter to the supplied tests, such that only tests a certain distance from a mutated class will be considered for running. e.g A test that directly calls a method on a mutated class has a distance of 1, a test that calls a method on a class that uses the mutated class as an implementation detail has a distance of 2, etc.[3]
We have set the maxDependencyDistance as 1 which saved us a considerable amount of execution time.
<maxDependencyDistance>1</maxDependencyDistance>We have tested pitest for two projects having different scales which may help you compare and estimate the pitest execution time on your codebase.
Please note that running Pitest for all project codebase on pipeline may take a very long time according to your lines of code and test count. Try running all project code pitest locally by using the “-DwithHistory” parameter.
Conclusion
We as the Search team at Trendyol, have the motivation to switch to a practice that we can do one-click deployments. Our CI/CD Pipelines should be very strong and reliable in order to go to a one-click deployment model so that we are constantly taking new steps to improve our CI/CD pipelines. Although we use some methods like code review and code coverage to increase the quality of our unit tests, sometimes these methods could be insufficient. Mutation testing is the best way to understand that the tests are valid and complete which is making our pipelines more stronger and reliable.
Special thanks to Sedef Tulum for her support on the Pitest integration and to Akif for his support on Sonar integration.
