Mutation Testing for Smart Contracts - A step by step guide
In my previous article, I wrote about the core concepts of mutation testing. With this post, I will show you the mutation testing theory can be applied in practice using the mutation testing tool Vertigo.
Setup
To start with mutation testing, we need to do three things:
- Install Truffle and Ganache
- Install Vertigo
- Get a project to test
Install Truffle and ganache by executing the following commands:
$ npm install -g truffle
$ npm install -g ganache-cli
Then install Vertigo by executing the following command:
$ pip3 install eth_vertigo
With that out of the way, the only thing left to do is to get a Truffle project to run our mutation testing on.
In this article, I will use the MetaCoin truffle box as the target of the mutation testing run. You can easily download this project by executing the following commands in your terminal:
$ mkdir metacoin
$ cd metacoin
$ truffle unbox metacoin
Configuration
We need to do a minor configuration tweak to ensure that Vertigo can run optimally. Vertigo can run the analysis for multiple mutants in parallel if there are enough test networks configured. Unfortunately, the default configuration for the MetaCoin box does not yet provide these networks, so you’ll have to add them yourself.
The following snippet shows my truffle-config.js
after I’ve added two networks to use with mutation testing. Go ahead and copy-paste this snippet into the truffle-config.js file in the metacoin directory.
You can also just add the two networks, if you already have some values configured in your project’s truffle-config.js
.
Let the magic happen
First, we need to start the two networks that will be used by Vertigo. You can do so by running the following commands in separate terminal windows (bonus: You can try to use tmuxinator with this config, to make life a bit easier):
$ ganache-cli -p 8545
$ ganache-cli -p 8546
With everything set up, we can start the mutation testing by simply running the following command:
$ vertigo run --network vertigo_test_network_1 --network vertigo_test_network_2
The next step is very important!
Sit back. Enjoy. Drink some ☕
PS. You can optionally add the commandline option --output output.txt
to the command, to write the analysis results to a file named output.txt
.
Interpreting The Results
Once the analysis is finished you will be presented with the following information:
...
[*] Running analysis on 5 mutants
100%|████████████████████████████████████████████████████| 5/5 [00:17<00:00, 4.38s/mutant]
[*] Done with campaign run
[+] Report:
Mutation testing report:
Number of mutations: 5
Killed: 4 / 5Mutations:[+] Survivors
Mutation:
File: /Users/walker/Development/metacoin/contracts/MetaCoin.sol
Line nr: 19
Result: Lived
Original line:
if (balances[msg.sender] < amount) return false;Mutated line:
if (balances[msg.sender] <= amount) return false;
...
The important bit is the line that says Killed: 4/5. This line shows the number of killed mutants (4 in this case), compared to the total number of valid mutants (5 for MetaCoin). See my previous article for an explanation of these terms.
Just one mutation slipped through the cracks! The mutation score for this project is 80%; pretty good.
Stay tuned for a deep dive into the detailed analysis results that Vertigo provides and how you can use them to improve your test suite’s quality.
Looking to the future
Vertigo is a brand new mutation testing tool, and while features like parallelization and equivalent mutant detection help improve usability, there are still some things to improve:
Test Networks
The current setup process requires you to add network configurations to `truffle-config.js` and directly run ganache instances.
One of the next things that will be added to Vertigo is a feature that automates all this and gives a more streamlined experience.
Analysis Speed
For big projects like openzeppelin-contracts the mutation testing process can take some time. And sure, you can use features like mutant sampling to reduce the number of mutants that you need to test. However, it would be better if we spend less time on each mutant altogether. A soon-to-be-added feature achieves just that, by leveraging fine-grained coverage data and optimizing which unit tests to execute.
Of course, there is more in the pipeline! But you will have to stay tuned to hear about that!
Thanks to Bernhard Mueller for reviews on drafts of this article!