A QuickCheck Primer for PHP Developers

Don’t write tests. Generate them.

— John Hughes

Meet QuickCheck

QuickCheck is a testing library originally developed in Haskell in 1999. Like the xUnit family it has been implemented in many different languages.

It takes a completely different approach to example-based testing which everyone is familiar with from PHPUnit. This approach is often called generative testing because inputs are randomly generated but the more fitting term is actually property-based testing because that is what it’s really all about.

A Trivial Example

Property-based testing means that we’re not explicitly listing our expected outputs for our given inputs but describing our functions in terms of properties that should be true for all inputs.

Sounds good, but how does it work in practice? Let’s consider the example of addition. If we were to implement our own add function we may write unit tests like the following:

function testAddZero() {
  $this->assertEquals(42, add(42, 0));
  $this->assertEquals(23, add(0, 23));
}
function testAddPositive() {
  $this->assertEquals(42, add(23, 19));
  // …
}

This is tedious and error-prone to write, to maintain, hard to read and whoever writes it will probably not think of the combinations of inputs which might actually catch a non-trivial bug.

“Nooo, “ you’re probably yelling at me right now “use @dataProvider you dummy!” and you’re right, we can be much more efficient by doing so but there’s no escaping the fact that we’re still enumerating inputs and outputs by hand, with all the drawbacks that brings.

Enter Property-Based Testing

So let’s write property-based tests for our addition function. We have to come up with some properties for addition that will be true no matter what.

That’s easy of course, addition is commutative and associative so we can test those. We can also test the identity property of zero.

$zeroIdentityProperty = Gen::forAll(
 [Gen::ints()],
 function($x) {
   return $x == add($x, 0);
 });
$commutativeProperty = Gen::forAll(
 [Gen::ints(), Gen::ints()],
 function($a, $b) {
   return add($a, $b) == add($b, $a);
 });
$associativeProperty = Gen::forAll(
 [Gen::ints(), Gen::ints(), Gen::ints()],
 function($a, $b, $c) {
   return add($a, add($b, $c)) == add(add($a, $b), $c);
 });

Now we can run as many tests as we want for those properties and PhpQuickCheck will take care of generating inputs for us.

Quick::check(1000, $zeroIdentityProperty);
Quick::check(1000, $commutativeProperty);
Quick::check(1000, $associativeProperty);

And if you want to run more tests, just turn it up. Let’s do a billion tests of the associative property.

Quick::check(1e9, $associativeProperty,
  ['max_size' => PHP_INT_MAX]);

We’re setting the max_size option because it defaults to 200 which means we’re only producing integers in the range [-200, 200]. Obviously there’s really no point in testing those values a billion times.

Honey, I Shrunk the Kids

When PhpQuickCheck successfully falsifies a given property, i.e. finds a failing case, it attempts to find the smallest input that still triggers the failure.

Consider a sort function. An obvious and useful property of a sort function is that the resulting array elements should be in ascending order. We can easily implement a predicate function to test that property:

function isAscending(array $xs) {
  $lastIndex = count($xs) - 1;
  for ($index = 0; $index < $lastIndex; ++$index) {
    if ($xs[$index] > $xs[$index + 1]) {
      return false;
    }
  }
  return true;
}
$sorted = Gen::forAll(
  [Gen::ints()->intoArrays()],
  function($xs) {
    return isAscending(sort($xs));
  });

If our sort function is broken this test will fail eventually and since we’re randomly generating inputs it will fail with a different array every time.

Even for this simple property it would be quite annoying to get a large array of random values with only a few elements that fail the test. You would still have to figure out which elements caused the error.

Obviously if you’re dealing with more complicated inputs, outputs and properties it can be virtually impossible to find the signal amidst the noise of a randomly generated failing case.

Shrinking Inputs

Once a property fails PhpQuickCheck will go into the so-called shrink-loop. This just means that it will re-run the property with smaller and smaller inputs in an effort to find the smallest failing value.

For example, even though the sort example given above will fail with a different array every time, QuickCheck will always shrink the input to an array with the two elements [0,-1] or [1,0].

Here is such an example output from PhpQuickCheck (JSON encoded for readability). As you can see the original input that caused the property to fail is [-3,6,5,-5,1] which got shrunk to [1,0].

{
    "result": false,
    "seed": 1411398418957,
    "failing_size": 6,
    "num_tests": 7,
    "fail": [
        [-3,6,5,-5,1]
    ],
    "shrunk": {
        "nodes_visited": 24,
        "depth": 7,
        "result": false,
        "smallest": [
            [1,0]
        ]
    }
}

This shrinking process is automatic and works out of the box for all generators. For example:

$dayTimeGen = Gen::tuples(Gen::choose(0, 23), Gen::choose(0, 59))
  ->fmap(function($tuple) {
    return sprintf('%02d:%02d', $tuple[0], $tuple[1]);
  });

Here both minutes and hours will shrink independently to zero. Imagine a property that fails if minutes are above 42 — this would result in “00:42” as the minimal failing value.

tl;dr

Property-based testing can be a great addition to your existing testing practice. It can save a lot of tedium and allows you to run as many tests as you want or need.

In future posts we will explore more realistic examples, learn how shrinking works, and figure out how to test stateful systems.

Meanwhile, I’d recommend to watch Testing the Hard Stuff And Staying Sane, a great talk by John Hughes, one of the original QuickCheck developers, to see how powerful property-based testing can be.

And check out PhpQuickCheck if you want to try it for yourself.