An introduction to property based testing with JS Verify

Recently, my team was tasked with investigating property based testing with Javascript. We looked at several libraries and concluded JSVerify was the best candidate. At first, it was unclear exactly how to get started with JSVerify, and it wasn’t until I saw an example of the source within a folder labeled, “examples” that I understood how to get going. Once I started working with it, though, the rest seemed to fall into place.

The purpose of this guide is to help one get started in quickly evaluating JSVerify. It purposely leaves out parts of JSVerify that are not needed to conduct property based testing. Instead, you’ll find out how to quickly write property tests and include them with your favorite testing suite. I’ve also included working examples to help you grok! Check for links located in some of the captions of code snippets to see them live in your browser.

Quick note: Property vs unit testing

With unit tests, your input and outputs are well defined so you can use them to ensure the “happy path” for your functions. With Property tests, your inputs (and sometimes outputs) are “fuzzy”, so you can ensure that your functions behave predictably given certain constraints (aka rules, aka properties)

Guide

To get the most out of the guide, you should have a basic understanding of nodejs, used some test suites before, and know some javascript.

The first thing to do is add jsverify as a devDependency to your project:

npm install -D jsverify

Test construction

You can create a test using checkForall which takes n args, with the last one being the predicate function the randomly generated arguments are passed to. The test is considered passing when the predicate returns true 100% of the time.

If all return true, the result of checkForAll is true

If any return false, the result of checkForAll is an Object containing additional data about what went wrong.

All tests created using checkForall will run 100 times by default. To change it, you have to use a different API which will be covered later.

For example:

checkForall(jsc.integer, (a) => … )
// A randomly generated integer will be passed to a predicate function as a.
checkForall(jsc.integer, jsc.integer, (a, b) => … )
// Two randomly generated integers will be passed to a predicate function as a, b.
checkForall(jsc.integer, jsc.integer, jsc.integer, (a, b, c) => … )
// Three randomly generated integers will be passed to a predicate function as a, b, c.

Working Example

Should you run the following code:

(Run this live in your browser)

You will find the following output:

OK, passed 100 tests
OK, passed 100 tests
{ additionIsCommutative: true,
  multiplicationIsDistributive: true }

Let’s see what happens if we assert that subtraction is commutative by adding this function to our suite:

(Run this live in your browser)

Running this gives us the following output:

OK, passed 100 tests
OK, passed 100 tests
Failed after 1 tests and 4 shrinks. rngState: 095d6bac8937ef2140; Counterexample: 0; 1;  [ 0, 1 ]
{ additionIsCommutative: true,
  multiplicationIsDistributive: true,
  subtractionIsCommutative:
   { counterexample: [ 0, 1 ],
     counterexamplestr: '0; 1',
     shrinks: 4,
     exc: false,
     tests: 1,
     rngState: '095d6bac8937ef2140' } }

Because subtraction is not commutative, the test fails! The failure gives us an object that tells us that it passed in 0, 1 as arguments, and it was the first test. exc is false because an exception was not thrown. Theoretically, you could assert that a given property test must pass x percent of the time. (Perhaps an async function takes less than 100 ms 90% of the time maybe?)

We’ll explore more about this object later, for now, I think its best to know that failures will return a(truthy)object but they could be used for further inspection and more complex assertions.

Usage with test runners

You can assert or expect that checkForall returns true, or you can use assertForall to throw an error because throwing errors will fail tests.

Mocha

With mocha alone, you can wrap it inside an it() function.

Chai (Should / Expect)

The following works with Chai BDD Library:

Jasmine

You can wrap jsc.checkForall() Jasmine's toBeTrue() expectation.

Tape

When using tape’s plan, you want to use checkForall and t.equal to ensure that tape counts the test as passing.

Customizing tests

To customize the tests, we use the forAll api, and then wrap it with check (hence, checkForAll is shorthand for those) For example:

jsc.checkForall(jsc.integer, jsc.integer, (a, b) => a + b === b + a)
// is the same as
jsc.check(jsc.forall(jsc.integer, jsc.integer, (a, b) => a + b === b + a))

However, you can pass an options object to forall() which allows you to customize the test

What is interesting about the rngState is that if you give the same state, it will "randomly" generate the same data every time.

Generating Random Data

So far we’ve been generating random integers using jsVerify’s built-in integer function. To generate random data we need to feed a Type called an arbitrary. In the field of Mathematics an arbitrary is defined as undetermined; not assigned a specific value.

In a sentence: “So far, we have been using arbitrary integers! :)”

Much of JSVerify’s API is filled with functions for crafting and using them. If you have a strong background in mathematics as well as functional programming then they will make sense to you. Personally, I do not, so with this guide, I am only going to cover ones that you absolutely need in order to generate random data for testing. I will also disclaim with confidence that there may be a better way for creating arbitraries (if there is, please share!)

You can see a list of arbitraries here: https://github.com/jsverify/jsverify#primitive-arbitraries

I personally like to think of these as Objects that describe a Type, with a set of additional instructions on how to randomly generate the data, as well as how to report on failures. (correct me if I’m wrong)

Custom Arbitraries

Let’s create an Arbitrary user! Since our user is an object literal, we tell JSVerify to create an arbitrary record.

See the live example here

Some closing thoughts

At first I was actually trying to create random data similar to what I see with chance.js or data faker, but I couldn’t figure out how to combine generators together. Ultimately, decided to avoid 3rd party random generators for now because they break rngState, and I’m not sure if it breaks anything else that JsVerify does.

However, I think that for the most part when using property based testing, we’re looking more at constraints, rules, and boundaries. With that, I feel it safe to assume that its better to generate variables that are less predictable. The whole focus of property based testing is about describing constraints/rules (aka properties) about a given function and ensuring those are indeed true.

Please let me know if there are any corrections needed. I also would like to mention that in the next guide we will be creating a full test suite and taking a closer look at arbitraries.