Introduction to Property Based Testing

Published in

Criteo R&D Blog

6 min readMar 23, 2018

Another test philosophy introduced by QuickCheck

Property based testing has become quite famous in the functional world. Mainly introduced by QuickCheck framework in Haskell, it suggests another way to test software. It targets all the scope covered by example based testing: from unit tests to integration tests.

In order to introduce property based testing, this article will use fast-check — a property based testing framework for JavaScript — but examples can easily be adapted to other frameworks and languages.

It is good to notice that this test framework helped me to discover bugs in famous npm packages with only a few lines of code — checkout fast-check documentation for more details.

Existing ways

Multiple ways are currently available in the industry in order to prove the correctness of a code.

Axis:

Feature compliance: Does it really test the feature I am trying to provide?
Input scope covered: How well do I cover all possible inputs?

As we can see in the diagram above, there is maybe a gap to fill. This is where property based testing comes into play.

Definition

Property based testing relies on properties. It checks that a function, program or whatever system under test abides by a property. Most of the time, properties do not have to go into too much details about the output. They just have to check for useful characteristics that must be seen in the output.

A property is just something like:

for all (x, y, ...) such as precondition(x, y, ...) holds property(x, y, ...) is true

Here is a simple hello world property:

for all (a, b, c) strings
the concatenation of a, b and c always contains b

In other words, without any precise knowledge of the shape and content of the strings a, b and c I can surely say that b is a substring of a + b + c.

In the example above “rt” is a substring of “azerty”

The executor understanding of a property can be summarized as:

Property based testing: Executor understanding

Benefits

They are numerous:

Cover the scope of all possible inputs: by construct, it does not restrict the generated inputs if not asked to do so. Consequently it can theorically generate all possible inputs and cover the whole range of strings, integers or whatever type required by the system under test.
Shrink the input in case of failure: whenever it fails, the framework tries to reduce the input to a smaller input. For instance: if the condition of the failure is the existence of a given character in a string it should return the one-character string having only this character. This is certainly one of the most interesting features of such approach as most of the time the failure can be summarized by a very small case.
Reproducible and replayable: each time it runs a property test, a seed is produced in order to be able to re-run the test again on the same datasets. Any run failure causes the framework to print both the failing case and the seed in order to be able to fully reproduce the run.

It is also important to note that it does not — by any means — replace unit testing. It only provides an additional layer of tests that might prove very efficient to reduce some boilerplate tests.

Hands on property based testing

A full “hands on fast-check” is available at https://fast-check.dev/docs/category/quick-start/.

Bootstrapping an empty project:

mkdir sample-fast-check
cd sample-fast-check
npm init --yes
npm install fast-check --save-dev

Let’s consider the property we described in the previous section:

for all (a, b, c) strings
the concatenation of a, b and c always contains b

Let’s create a file called script.js with the code below:

Example of property for “contains”

You also need to implement a basic contains method (eg.: const contains = (q, text) => text.indexOf(q) !== -1;).

Then you can run the property script with:

node script.js

Content of script.js:

fc.assert(<property>(, parameters)): It executes the test and checks that the property stays true for all the a, b, c strings produced by the framework. In case of failure, it shrinks the input to the minimal failing example to help the user during the analysis. By default, it runs the property check against 100 generated inputs.
fc.property(<...arbitraries>, <predicate>): It describes the property. arbitraries are the instances responsible to build the inputs while predicate is the function doing the test against those inputs. predicate should either return a boolean or not return anything and throw in case of issue.
fc.string(): It is a string generator — also called arbitrary in property based. An arbitrary is responsible to generate and shrink values.

If you wonder what are the generated inputs you can replace fc.assert by fc.sample as follows and wrap it into a console.log:

Extract the values generated to check the property

You might get generated inputs like:

{a: ")|2", b: "", c: "$&RJh%% "}
{a: "\\\"", b: "Y\\\"\\\"", c: "$S#K3"}
{a: "$", b: "\\\\cx%wf", c: "'t4qRA"}
{a: "", b: "", c: "n?h.0%"}
{a: "6_#7", b: "b", c: "4%E"}
...

Shrinking in action

Let’s suppose that we have the following implementation of contains — which obviously does not work. The idea is to see what the framework generates in case of failure and how it shrinks the input:

Possible (buggy) implementation of “contains”

Multiple inputs will be generated and as soon as it finds a failing case it will start the shrinking process. If we take the example above, when running it locally I got a first failure for the entry
{"a":"","b":"1%Eg+]]2","c":"\\\"F#\\\" g"},
then the shrinking process took place:

{a: ")|2", b: "", c: "$&RJh%% "} -> success
{a: "\\\"", b: "Y\\\"\\\"", c: "$S#K3"} -> success
{a: "$", b: "\\\\cx%wf", c: "'t4qRA"} -> success
{a: "", b: "", c: "n?h.0%"} -> success
{a: "6_#7", b: "b", c: "4%E"} -> success
{a: "", b: "l%Eg+]]2", c: "\\\"F#}\\\" g"} -> first failure encountered: start shrinking process
{a: "", b: "", c: "\\\"F#}\\\" g"} -> success
{a: "", b: "+]]2", c: "\\\"F#}\\\" g"} -> failure
{a: "", b: "]2", c: "\\\"F#}\\\" g"} -> failure
{a: "", b: "2", c: "\\\"F#}\\\" g"} -> failure
{a: "", b: " ", c: "\\\"F#}\\\" g"} -> success
{a: "", b: ")", c: "\\\"F#}\\\" g"} -> failure
{a: "", b: "%", c: "\\\"F#}\\\" g"} -> failure
{a: "", b: "#", c: "\\\"F#}\\\" g"} -> success
{a: "", b: "$", c: "\\\"F#}\\\" g"} -> failure
{a: "", b: "\\\"", c: "\\\"F#}\\\" g"} -> success
{a: "", b: "#", c: "\\\"F#}\\\" g"} -> success
{a: "", b: "$", c: ""} -> failure
{a: "", b: "\\\"", c: ""} -> failure
{a: "", b: "!", c: ""} -> failure

Going further

Property based testing is a useful and powerful tool. I personally used it for multiple problems going from testing famous npm repositories to personal codes including UI. Here are some of the interesting topics I covered with this kind of tool.

Bug in famous npm packages: js-yaml

While js-yaml is a 1 million downloads per day package, property based testing was able to find a problem in it. The story is available at https://github.com/nodeca/js-yaml/pull/398 and the test at https://runkit.com/dubzzz/multiple-properties-for-js-yaml

Basically I adopted a black-box technic to test it against the most simple property possible: whatever the object I convert into yaml, I should be able to have it back by reading the generated yaml.

UI testing

Certainly one of the most surprising usage is to derive it for UI testing. Video below relies on fast-check and Selenium.

More at: https://medium.com/criteo-labs/detecting-the-unexpected-in-web-ui-fuzzing-1f3822c8a3a5

Turning fast-check into an automatic QA tester

Conclusion

I highly recommend that you give it a try on some snippets. Some code examples based on fast-check are available at: https://github.com/dubzzz/fast-check/tree/main/examples in the files called test.js.

By using it you will see how simply it can be used to diagnose and find issues.

Additional readings:

Frameworks I tried in other languages:

Hypothesis — Python
RapidCheck — C++
ScalaCheck — Scala