Generative Testing in JavaScript

The Case for Property-based Testing

Published in

JavaScript Inside

7 min readAug 14, 2016

Introduction

Generative testing is a widespread approach in the functional world, mainly due to the popularity of Haskell’s QuickCheck and Clojure’s QuickCheck implementation test.check.

Traditional unit tests enable us to verify assumptions for a given function. This approach, also known as example based testing, is based on manually defined inputs and expected outputs. In other words a developer has to figure out a set of inputs/outputs to assert that the function under test is working as intended. This also means that one would have to write a large number of tests just so to be able to verify a wide range of cases.

So while unit tests might verify a reasonable set of assumptions, you can’t manually catch all the possible cases that need to be tested. Thinking beyond a subset of expectations becomes very inefficient at some point. There’s a limitation here. This is where generative testing comes into play.

The traditional approach

To clarify things, let’s start with an example.

const add = (a, b) => a + b

So we have our own handcrafted add function now, which means we can write some tests to verify everything works as expected. A typical test for add might look something like this (you would probably group expectations like “should work with zero” etc. but this should suffice for now):

describe('add', () => {
  it('should work', () => {
    assert(add(1, 1), 2)
    assert(add(1, 3), 4)
    assert(add(0, 0), 0)
    assert(add(-1, 1), 0)
    assert(add(10, -1), 9)
  })
})

These couple of assertions should verify everything is working as expected with the inputs/outputs having been defined manually. Nothing new up until now, but we will build upon this example in the following section.

Property-based Testing Introduction

To highlight the difference between unit and generative tests we can state that generative tests are not defined manually and that they are verified against invariants not against values as this is the case with an example based approach.

In short: with generative testing we specify properties and the testing library generates those tests for us.

The key part is to define a property that can be verified, in other words we need to define a function that should always returns true for any given input.

Writing property functions isn’t as trivial as compared to say traditional unit tests. Take for example the previously defined add function. Writing a property function like

(a, b) => a + b === add(a, b)

is useless, as we’re reimplementing the add function that we’re actually trying to test.

A more suitable approach would to be verify that adding zero works.

const zeroProperty = x => add(x, 0) === x

Addition is also commutative, so we can also base our assumptions on this fact to validate that add actually works.

const commutativeProperty = (x, y) => add(x, y) === add(y, x)

Another good approach for writing a property function is using multiple functions to describe their inverse relationship.

const result = x => decodeSomething(encodeSomething(x)) === x

It is also valid to test a function against another existing function. For example you might test your own min function against Math.min.

Further more you might also write a property function that describes the relationship between the inputs and the outputs. Take the following assumption f.e.:

const result = add(4, 0) + add(0, 4) === add(4, 4)

Defining useful property functions is the hard part here. The rest is handled by a generative testing library that generates tests upon a given data specification.

Property-based Testing in Detail

Now that we have a high level idea about what property-based testing is all about it’s time to see how this works in a JavaScript setup.

The following examples are based on testcheck-js by Lee Byron, but there other alternatives like jsverify.

We have a handlePayment function that expects two arguments balance and payment, balance representing the account total and payment representing a sum that will either be added or deducted from the total balance.

const handlePayment = (balance, payment) =>
  (!payment || balance - payment <= 0) ? balance : balance - payment

In a traditional fashion you would start to write a couple of tests that verify certain expectations. For example your test might look something like this. You might test that handlePayment handles zero as well as negative inputs besides checking for the standard case (positive balance and payment).

import { assert } from 'chai'
import handlePayment from '../src/handlePayment'describe('handlePayment', () => {
  it('should handle zero inputs', () => {
    assert.equal(handlePayment(0, 1), 0)
    assert.equal(handlePayment(1, 0), 1)
  })
  it('should handle negative inputs', () => {
    assert.equal(handlePayment(-1, 1), -1)
    assert.equal(handlePayment(10, -1), 11)
  })
  it('should handle positive inputs', () => {
    assert.equal(handlePayment(200, 1), 199)
    assert.equal(handlePayment(10, 11), 10)
  })
})

We run the tests and see everything is green.

What property function would we need to define for verifying our assumptions?

Let’s verify that zero payment always returns balance.

import { check, property, gen } from 'testcheck';
import handlePayment from './handlePayment'const result = check(property([gen.int], (x) => {
  return handlePayment(x, 0) === x
}))console.log(result)

This is the log output:

{ result: true, ‘num-tests’: 100, seed: 1471133618254 }

testcheck-js ran one hundred tests against handlePayment and all cases passed. Now let’s verify if handlePayment works correctly when working with a balance of 100 and random payment inputs.

const result = check(property([gen.intWithin(0, 100)], x => {
  return handlePayment(100, x) === 100 - x;
}))console.log(result)

The result output shows clearly that a case has failed.

{ result: false,
  'failing-size': 22,
  'num-tests': 23,
  fail: [ 100 ],
  shrunk: 
   { 'total-nodes-visited': 7,
     depth: 0,
     result: false,
     smallest: [ 100 ] } }

If we take a closer look, we’ll find a shrunk property. This is a powerful feature that most generative testing libraries encompass. Instead of returning a large dataset of cases, where maybe one or a couple of cases failed, testcheck-js will try to find the smallest failing test (see the smallest property inside shrunk) as soon as one case fails. In this specific case the smallest value that failed is 100. This gives us very specific data to find out if the problem is inside the predicate function verifying our handlePayment or in handlePayment itself or if the dataset we generated isn’t explicit enough.

The dataset set should be fine. Let’s check the handlePayment function.

Obviously the case where balance and payment might be equal, in this failing case it’s [100, 100], isn’t handled properly. We verified that balance should return balance when payment is larger than the balance but didn’t cover this specific case. Let’s fix updatePayment.

const handlePayment = (balance, payment) =>
  (!payment || balance - payment < 0 // fix. only less than zero
  ) ? balance : balance - payment

This will the solve the problem.

Let’s verify that paymentTransaction can handle a negative balance as well as a negative payment.

const { strictNegInt : strNegInt } = genconst result = check(property([strNegInt, strNegInt], (x, y) => {
  return handlePayment(x, y) === x - y;
}), {seed: 50})console.log(result)

If you take a closer look at our new property, we added an options object, where we defined a seed, to be able to reproduce the generated tests. (Thanks Sune Simonsen for highlighting the fact)

Running the newly added test results in a new case failing.

{ result: false,
  'failing-size': 1,
  'num-tests': 2,
  fail: [ -2, -1 ],
  shrunk: 
   { 'total-nodes-visited': 1,
     depth: 0,
     result: false,
     smallest: [ -2, -1 ] } }

A balance of-2 and a payment of -1 would have a positive outcome on the balance. We need to refactor handlePayment again.

const handlePayment = (balance, payment) =>
  (!payment ||
  (balance <= 0 && balance - payment < balance) ||
  (balance > 0 && balance - payment < 0)) ?
    balance : balance - payment

Running the test again finally results in all cases passing.

{ result: true, 'num-tests': 100, seed: 50 }

Let’s refactor the handlePayment one last time, just to make it more readable.

const handlePayment = (balance, payment) =>
  payment &&
  ((balance <= 0 && balance - payment > balance) ||
  balance - payment >= 0) ?
     balance - payment : balance

The tests still verify the property.

{ result: true, ‘num-tests’: 100, seed: 50 }

The two failed cases can be added to the previously defined unit tests.

We haven’t covered how to integrate testcheck-js with mocha or jasmine. This is out of scope for this writeup but there are specific mocha and jasmine testcheck-js wrappers available, see mocha-check and jasmine-check for more information on the topic. jsverify also has integration for mocha and jasmine.

Summary

Generative or property-based testing isn’t a replacement for traditional unit testing. While the example-based approach should verify common assumptions, the property-based approach should help identify missed cases. Furthermore traditional tests are easier to write, while the generative approach requires more thinking in regards of defining properties. Using them both back to back will yield the greatest benefits. The easiest way to get started with generative testing is to try out a library, for example jsverify or testcheck-js and start writing a couple of tests to fully understand the benefits.

I have added a couple of links that might be beneficial for getting up and running with property-based testing.

Edit

Thank you Sune Simonsen for testing the examples and actually verifying that the final handlePayment test would actually fail. Check this example showing all the examples in this post by using unexpected-check.

Also read Generative Testing Redux: Reducers to see an example for property-based testing your redux reducers.

Feedback or questions via twitter