Property based testing

Writing 100s of unit tests with only a single test.

Photo by Martin Oslic on Unsplash

The goal of this article is to introduce you to the concept of property based testing, PBT, and how it differs from regular unit tests. PBT can be a powerful addition to your unit tests and help prevent bugs in your code.

The Basics

What is a property?

When I’m talking about a property, I’m referring to this definition:

Property: an attribute, quality, or characteristic of something

Typically for us developers, we will be looking at the property of a function. For example, a property of the function below is that it always returns 0.

function getZero() {
 return 0;
}

Is it a unit test or a property test?

I like to distinguish between unit tests and property tests with this simple definition:

Unit test — testing with constants
Property test — testing with variables

If I was going to explain PBT in as few words as possible I would say:

A regular unit test, but the values are generated for you.

Normally, a unit test is a test using only constant values. The benefit of using constants is that the result of the test will never change and you only have to run it once. This is not the case with a property test.

With a property test, we replace all of the constants with variables. We want to be able to run a test multiple times and get different outcomes.

Constants: 1 + 2 = 2 + 1
Variables: x + y = y + x

Generated variables

When the variables are generated for us, we can create as many tests as we want. A property test is run inside a loop and each run of this loop will generate new variables for us and test our property.

In the example below, we have one single test. This is because there is only one reason why this test could fail.

assert( add(1, 2) ).toBe( add(2, 1) )

While the test above could pass, it is not certain whether or not it would pass for all other cases.

assert( add(1, 3) ).toBe( add(3, 1) )

The above test could possibly fail because of some unknown business logic.

If we remove the constants however, we have an infinite amount of tests. This is because there are an infinite amount of cases that could fail.

assert( add(x, y) ).toBe( add(y, x) )

In this property test, the case add(1, 3) === add(3, 1) will also be tested (given enough loops that is)

Photo by Osman Rana on Unsplash

Example

Now that we have a better understanding of a property based test, let us look at an example of using a property based test.

Simplifications

To make the examples shorter, we will add two simplifications:

1. all tests are run inside a loop.
2. inside this loop, all our variables are defined to a random value

const x = randomBoolean()

The task

Imagine you are given the task of testing a function, isRaining. This function takes in two boolean variables and returns a new boolean that indicates if it is in fact raining.
function isRaining(groundIsWet, isCloudy)

You are told that this function should only return true when the ground is wet and it is cloudy.

The Property

The property we want to prove here is:

Only isRaining(true, true) should return true.
All other cases should return false.

The Property Test

if (isRaining(x, y)) {
 expect(x && y).toBeTruthy()
} else {
 expect(x && y).toBeFalsy()
}
// or
expect(
 isRaining(x, y) ? x && y : !(x && y)
).toBeTruthy()

Given enough loops, we should be able to confidently say that all cases are tested. And if the test does not fail, we can say that the property has been proved.

Function modifications

Let us now assume that we need to add a new parameter to our isRaining function, aboveFreezing. If it is freezing outside, it would be snowing and not raining. Therefore, the only case that should return true is now isRaining(true, true, true)

This requires only a minor change to our property test

if (isRaining(x, y, z)) {
 expect(x && y && z).toBeTruthy()
} else {
 expect(x && y && z).toBeFalsy()
}
// or
expect(
 isRaining(x, y, z)
 ? x && y && z
 : !(x && y && z)
).toBeTruthy()

As we can see, adding new variables to isRaining requires very little modification to our property test. In fact, adding 5 new variables would not require much effort at all. This is however not the case if we had instead written unit tests.

With unit tests we would have to write 2^n unit tests. The 2 comes from the fact that a boolean variable has two possible values: true or false. The n is the amount of variables we have.

2^n

With only 3 variables, we require 8 unit tests. In my opinion, going beyond 3 variables seems like an unreasonable amount of tests to both write and maintain.

Issues

By now you should have an understanding of what some of the benefits of PBT are. Now we will discuss some of the issues and concerns with this approach.

The loop takes time

The biggest disadvantage of PBT is that looping takes time.

While unit tests are constant and quick to run, property tests are dynamic and have to be run multiple times. Reducing the amount of loop iterations will improve performance, but that would reduce the amount of cases being tested.

Finding the failing case

There is no guarantee that a failing case will be tested. If we test the properties of the functionadd, defined below, by providing two random numbers, we will most certainly never find the case that fails.

function add(a, b) {
 if (a === 1 && b === 100) {
   throw “you will never test this case!”
 }
 return a + b;
}

The only way we could hope to find this particular failing case, is if we have an appropriate loop size.

An appropriate loop size
As mentioned earlier, the amount of cases for boolean variables is 2^n. For our isRaining function with 4 variables, we would have a total of 16 cases. A loop size of around 64 will most likely cover all cases. For regular numbers in JavaScript, the amount of loops is much higher.

The amount of cases for a function that accepts a single number is around Number.MAX_SAFE_INTEGER. On my machine, that is 9007199254740991. This is an unfeasible amount of test cases to run and this is only taking into account positive numbers.

Dealing with the issues

The loop

The performance penalty of the loop is something that you can not avoid. You will eventually have to run your tests. To remedy this issue, you could run your unit tests separately from your property tests.

You could accomplish this with a simple naming convention.

Unit test: something.spec.js
Property test: something.properties.js

You can now specify which type of tests you wish to run by supplying a test regex to your test runner.

You can also have your default loop size be based on the environment. Tests run locally could use small loop sizes, while build servers could use larger numbers.

Finding the failing cases

PBT can help you find failing cases, but there is no guarantee that it will. For the function add, guaranteeing that it will find the failing case would require testing 9007199254740991² cases.

Program testing can be used to show the presence of bugs, but never to show their absence!

-Edsger W. Dijkstra

The only thing we can do here is to set some restrictions to our property tests. If we limit the range of possible input values for add, we can greatly increase our likelihood of finding a failing case. Let us say that we know that add will only be used for values from 0 to 100. The amount of cases is now 100². A tremendous reduction in cases.

Libraries

There are many libraries out there that will get you started with PBT. You can find many if you search for quick check <your programming language>

Here are some:

QuickCheck — the mother of all PBT libraries, written in Haskell
fast-check — JavaScript and TypeScript
junit-quickcheck — Java
ScalaCheck — Scala

GitHub

I’ve created a minimal GitHub repository, property-based-testing, that shows the above examples implemented using fast-check