An introduction to property based testing with JS Verify
Recently, my team was tasked with investigating property based testing with Javascript. We looked at several libraries and concluded JSVerify was the best candidate. At first, it was unclear exactly how to get started with JSVerify, and it wasn’t until I saw an example of the source within a folder labeled, “examples” that I understood how to get going. Once I started working with it, though, the rest seemed to fall into place.
The purpose of this guide is to help one get started in quickly evaluating JSVerify. It purposely leaves out parts of JSVerify that are not needed to conduct property based testing. Instead, you’ll find out how to quickly write property tests and include them with your favorite testing suite. I’ve also included working examples to help you grok! Check for links located in some of the captions of code snippets to see them live in your browser.
Quick note: Property vs unit testing
With unit tests, your input and outputs are well defined so you can use them to ensure the “happy path” for your functions. With Property tests, your inputs (and sometimes outputs) are “fuzzy”, so you can ensure that your functions behave predictably given certain constraints (aka rules, aka properties)
Guide
To get the most out of the guide, you should have a basic understanding of nodejs, used some test suites before, and know some javascript.
The first thing to do is add jsverify as a devDependency to your project:
npm install -D jsverify
Test construction
You can create a test using checkForall
which takes n
args, with the last one being the predicate function the randomly generated arguments are passed to. The test is considered passing when the predicate returns true 100% of the time.
If all return true, the result of checkForAll
is true
If any return false, the result of checkForAll
is an Object
containing additional data about what went wrong.
All tests created using
checkForall
will run 100 times by default. To change it, you have to use a different API which will be covered later.
For example:
checkForall(jsc.integer, (a) => … )
// A randomly generated integer will be passed to a predicate function as a.checkForall(jsc.integer, jsc.integer, (a, b) => … )
// Two randomly generated integers will be passed to a predicate function as a, b.checkForall(jsc.integer, jsc.integer, jsc.integer, (a, b, c) => … )
// Three randomly generated integers will be passed to a predicate function as a, b, c.
Working Example
Should you run the following code:
You will find the following output:
OK, passed 100 tests
OK, passed 100 tests
{ additionIsCommutative: true,
multiplicationIsDistributive: true }
Let’s see what happens if we assert that subtraction is commutative by adding this function to our suite:
Running this gives us the following output:
OK, passed 100 tests
OK, passed 100 tests
Failed after 1 tests and 4 shrinks. rngState: 095d6bac8937ef2140; Counterexample: 0; 1; [ 0, 1 ]
{ additionIsCommutative: true,
multiplicationIsDistributive: true,
subtractionIsCommutative:
{ counterexample: [ 0, 1 ],
counterexamplestr: '0; 1',
shrinks: 4,
exc: false,
tests: 1,
rngState: '095d6bac8937ef2140' } }
Because subtraction is not commutative, the test fails! The failure gives us an object that tells us that it passed in 0, 1
as arguments, and it was the first test. exc
is false because an exception was not thrown. Theoretically, you could assert that a given property test must pass x percent of the time. (Perhaps an async function takes less than 100 ms 90% of the time maybe?)
We’ll explore more about this object later, for now, I think its best to know that failures will return a(truthy)
object but they could be used for further inspection and more complex assertions.
Usage with test runners
You can assert
or expect
that checkForall
returns true
, or you can use assertForall
to throw an error because throwing errors will fail tests.
Mocha
With mocha alone, you can wrap it inside an it()
function.
Chai (Should / Expect)
The following works with Chai BDD Library:
Jasmine
You can wrap jsc.checkForall()
Jasmine's toBeTrue()
expectation.
Tape
When using tape’s plan
, you want to use checkForall
and t.equal
to ensure that tape counts the test as passing.
Customizing tests
To customize the tests, we use the forAll
api, and then wrap it with check
(hence, checkForAll
is shorthand for those) For example:
jsc.checkForall(jsc.integer, jsc.integer, (a, b) => a + b === b + a)
// is the same as
jsc.check(jsc.forall(jsc.integer, jsc.integer, (a, b) => a + b === b + a))
However, you can pass an options object to forall()
which allows you to customize the test
What is interesting about the
rngState
is that if you give the same state, it will "randomly" generate the same data every time.
Generating Random Data
So far we’ve been generating random integers using jsVerify’s built-in integer
function. To generate random data we need to feed a Type called an arbitrary
. In the field of Mathematics an arbitrary is defined as undetermined; not assigned a specific value.
In a sentence: “So far, we have been using arbitrary integers! :)”
Much of JSVerify’s API is filled with functions for crafting and using them. If you have a strong background in mathematics as well as functional programming then they will make sense to you. Personally, I do not, so with this guide, I am only going to cover ones that you absolutely need in order to generate random data for testing. I will also disclaim with confidence that there may be a better way for creating arbitraries (if there is, please share!)
You can see a list of arbitraries here: https://github.com/jsverify/jsverify#primitive-arbitraries
I personally like to think of these as Objects that describe a Type, with a set of additional instructions on how to randomly generate the data, as well as how to report on failures. (correct me if I’m wrong)
Custom Arbitraries
Let’s create an Arbitrary user! Since our user is an object literal, we tell JSVerify to create an arbitrary record
.
Some closing thoughts
At first I was actually trying to create random data similar to what I see with chance.js or data faker, but I couldn’t figure out how to combine generators together. Ultimately, decided to avoid 3rd party random generators for now because they break rngState
, and I’m not sure if it breaks anything else that JsVerify does.
However, I think that for the most part when using property based testing, we’re looking more at constraints, rules, and boundaries. With that, I feel it safe to assume that its better to generate variables that are less predictable. The whole focus of property based testing is about describing constraints/rules (aka properties) about a given function and ensuring those are indeed true.
Please let me know if there are any corrections needed. I also would like to mention that in the next guide we will be creating a full test suite and taking a closer look at arbitraries.