Your own property based testing framework — Part 2: Runners
In Part 1, we covered the concept of generators. Without them property based testing would not be possible. Alone they do not offer a lot.
In this article, we will cover an other main piece of property based testing frameworks: runners.
In fast-check there is one main runner called fc.assert. Let’s see how to add such feature into our minimal framework.
Same article on dev.to at https://dev.to/dubzzz/your-own-property-based-testing-framework-part-2-runners-12fl
First of all, we will have to define the concept of property. For simplicity, a property can be seen as a super generator:
In our case, properties will be created using the following helper:
Now, let’s consider a simple example to understand how we want to use our minimal version of fast-check. The code under test will be an implementation of isSubstring with obviously a bug into it in order to check that our framework can find it. As a user we would like to be able to write the following test:
In terms of typings we have the following signature to fulfill on assert:
By default, in most frameworks, runners run the property a hundred times and stop if everything is working fine after those hundred runs.
A basic implementation for the runner can be written as follow:
Additionally in property based testing, seed is supposed not to be fixed except if specified on call-site. Implementation above can be updated as follow:
In the previous article, we did not cover the reason why we opted for pure random generators. In property based we want properties to be reproducible no matter the seed, no matter the hardware and no matter the unix time… But we also want to have independent runs for each iteration in the loop.
For instance, in the implementation defined above, we call generate with the following instances of Random:
runId = 0Call withnew Random(prand.xoroshiro128plus(seed))runId = 1Call withnew Random(prand.xoroshiro128plus(seed)).jump()runId = 2Call withnew Random(prand.xoroshiro128plus(seed)).jump().jump()- …
jump offsets a random number generator. In the context of xoroshiro128plus calling jump is equivalent to 2^64 calls to next. In the case of pure-rand neither jump nor next alter the original instance, they both create a new instance of the generator while keeping the original one unchanged.
No matter how many times our property will call the passed mutable random generator, we will always ignore it to build the generator required for the next iteration. While it might appear strange at first glance, this feature is important as we don’t really know what will happen to this instance of our random generator. Among the possible scenario:
- relying on the offset applied by the property to the passed instance of
Randomis problematic as it makes replays difficult to implement except if we re-generate all the values one-by-one whenever replaying stuff - instance of
Randommight be kept and re-used later by the property and itsGenerator(we will see that it might be the case in some implementations of shrink), thus calls to generate in subsequent iterations might alter it
We can now use our small framework on the property we discussed earlier in this section. As we were expecting, it finds an issue and reports it. When running it locally you should have an output similar to:
Given all the work above, you should be able to write and test properties as if you were using fast-check.
A playable example is available at https://runkit.com/dubzzz/part-2-runners