Introduction to Property Based Testing With Hypothesis
A standard set of test cases looks like this: Given input X I expect result Y. Repeat for different Xs and Ys until you have confidence your code works. In practice there are at least two common problems: This testing can lead to a large amount of code that must be maintained and all too often we miss bugs because of assumptions we made about what inputs needed testing.
Property Based Testing is a method of writing tests that tries to improve testing quality while minimizing testing code. The basic concept is you describe the inputs to your function and then declare what properties that must be true to declare your code correct.
Let’s walk though an example using Hypothesis. Hypothesis a property based testing library for Python written by David R. Maclver. Many of its core ideas are come from Haskell’s QuickCheck library.
Assume I am writing an application to be an online history archive. Below is a simple model for describing an event. I included code to encode and decode this object with JSON as I would like my app to communicate with some kind of front end.
The question now: Does my encoder/decoder actually do what I think it does?
Some reasonable test code would be:
This test passes and would get us full test coverage. However, for production ready code we may want to try different examples to ensure our code works in a variety of different situations. Realistically though, this is a fairly simple bit of code and many of us would stop here.
Besides only testing one case, the problem is that a lot of the code here is irrelevant to what’s being tested. All that code setting up my data is boilerplate. At the end of the day, I don’t care about specific data. I care about certain properties being true. Specifically, I expect that if I encode an object to JSON I should be able to load that code back into my system without losing data. No matter what the specific values are in my object.
Let’s see a property based version of this test.
Using the @given decorator, I tell hypothesis what kind of data to give my test. With that I construct my Event, dump it to JSON and load that JSON back in. I then assert that I got back an equal Event.
Hypothesis by default will run 200 different examples against this test. It Explores the space of possible inputs and ensuring that for all those cases the property I wrote remains true.
Running this test actually highlights a problem!
While testing my code Hypothesis found a failure. Despite working with random data Hypothesis gave me a very simple example to debug. This is because rather than giving me the first failure it finds Hypothesis will simplify that failure as much as it can.
So what’s happening with my program? If you look closely, you will see that this date is from the first century AD. I am using dateutil to parse my ISO formatted date strings and it turns out as of this writing dateutil mis-interprets this date as 1999 rather than 99. I have written a patch to fix this here https://github.com/dateutil/dateutil/pull/96.
This is an obscure bug in my decoder. That is the point though! I would not have thought to test this case because I assumed that the date parsing would work fine since I was relying on a solid and popular library to do it for me.
Property Based Testing can lead to smaller and more powerful tests. However, this method requires a different way of thinking than you may be used to. If you are interested in reading more here are some resources to get you started. I actually was provided these by David in the #hypothesis irc!
http://fsharpforfunandprofit.com/posts/property-based-testing-2/
http://research.microsoft.com/en-us/projects/pex/patterns.pdf
In addition, I highly recommend the Hypothesis docs. They do a fantastic job of walking you though what this library is capable of.
