Property-based testing driven development

Property-based testing is a cool thing which will be merged into Elixir sometimes in the future. Why not try it out? In the article, I will share idea how I use it.

Why do we need Property-Based Testing?

First of all Property-Based Testing does not replace Unit Test. While Unit Test let us think about edge cases of the function given input, Property-based testing let us think about properties of the function instead. Sometimes, we can’t cover all the edges in unit tests in many reasons. Then property-based testing could fill these gaps.

How to define properties ?

I’ve found that defining properties is very tricky part. Some functions could be pretty simple and easy to think of but some could be very hard especially complex functions which we might end up with complex property based testing accordingly. We have to keep in mind that complex test might cause bugs, increase maintainability and decrease readability as well which is the counterpart. We really need to find balance of this spectrum.

One of basic example of property to make everybody understand could be encryption and decryption function, let’s say

decrypt(encrypt(text)) == text

This property should be always true for every given texts.

In property-based testing, there is also a concept of Generator. In the above example, We can set the generator to feed x number of random texts into our function. And again, this property should be always true.

In this article, I will introduce you, one common pattern in property-based testing called ‘Test Oracle’

Test Oracle is the testing approach where we test our function against another implementation and expect output of our function will be always the same as the output from another implementation.
awesome_function(x) == blackbox_function(x)

I will pick one kata implementation in Elixir and use it as blackbox_function. Then, I will try to implement this kata by myself. The kata I’ve chosen is Poker Hands. , From Fred Wu, who is my old friend. Once I’ve read the requirement, It was quite clear and looked straightforward. After I’ve got everything I wanted, both the original implementation and the requirement of kata. Now, I’m ready to start by using property-based testing to drive my implementation. In the end, the goal is my function should always output the same as original no matter what input.

“people playing cards” by Inês Ferreira on Unsplash

Poker Hands Requirements in summary

I’ve never play Poker before. Then, my implementation would be follow the rules only from the requirement without prior game knowledge. Basically, in this game there are 2 hands — left hand & right hand. Each hand draws 5 cards. After that, we compare between left and right hand cards. To indicate the winner, we indicate by rank of the patterns and value of the cards. For example: 5 cards of the same suit with consecutive values aka. “StraightFlush” beats 2 pair of cards which have same suit aka. “2 Pairs”. (You can see more details in the full requirement page

When I’ve looked into the original function from Fred Wu. There is function name vs where the first argument is Left Hand and 2nd argument is Right Hand. So simple!

PokerHands.vs("2H 3D 5S 9C KD", "2C 3H 4S 8C KH")
#=> "Left wins!"

Is the requirement clear? Shall we start?

Let’s start!

  1. Add stream_data library to mix.exs in our project then run

And then run:

> mix deps.get

2. Copy Fred Wu’s implementation to my project 😆 So that I can call `PokerHand` and start to write property-based testing


  • [line:2] deck() is a list of cards. 1 deck has 52 cards.
  • [line:2–3] StreamData.uniq_list_of(deck(), length: 10)
    We will draw 10 cards from the deck and those cards shouldn’t be the same card. Then we assign 5 cards to Left hand and the rest 5 cards for Right hand
  • [line: 7] assert ProperPoker.vs(left_hand, right_hand) == PokerHands.vs(left_hand, right_hand) My function ProperPoker.vs/2 should always return the same result as Fred Wu’s PokerHands.vs/2

3. Nothing has been implemented anything yet. The test should be failed. Next step, I will start to write simple unit test for ProperPoker.vs/2 and implement it in a simple way. Just always returns Left wins! and run the test.

1st Unit Test
Failed property based testing.

4. Start implementing the patterns — High Card, Two Pairs, … from the failed tests which will be occurred every time we run the property-based testing. This is the flow:

4.1 We run property-based testing

Property-Based Test failed

4.2 Test failed.

4.3 We have to check which pattern that make the test failed in requirement document []. In the example above. Left hand has Full House. (3 cards in the same suit and one pair) Left hand supposes to win. I haven’t implement Full House yet. That’s why my function cannot recognize this pattern.

4.4 Write unit test for this pattern → Implement → Make Unit Test pass

Randomized with seed 837036

4.5 We have to run property-based testing. To see whether our implementation is correct or not. In order to replay the same failed case. We have to use the same randomized seed number as before. With the same seed, The generator always produces the same set of cards. Otherwise the generator will randomly generate new cards each time we run the test and the test might be failed because of different patterns.

> mix test --seed 837036

4.6 Once the property-based testing passed — start 4.1 over again we will see other failed cases according to unimplemented patterns. We have to start over until every patterns are implemented.

Hidden requirement has been found

After I have implemented all patterns. My expectation is all the tests should always pass. But the property-based testing still got failed result sometimes. It made me quite surprised. With the test failed, it could imply that there are something different between Fred Wu’s code and mine. I started to investigate. And then I’ve found that there is something special for the rule related consecutive values with Ace.

“A-2–3–4–5” counts as consecutive and Ace acts as lowest value
“10-J-Q-K-A” counts as consecutive and Ace acts as highest value

It’s called WHEEL. []

“pink and yellow Ferris wheel under clear sky” by Denisse Leon on Unsplash

After, I’ve implemented it with unit test first to make this test case more explicit and run property-based testing again. Finally, All tests passed!

If you want to see the source code, here is Github

Looks good in KATA but what about REAL life?

There are many situations that we can use Test Oracle pattern. For example:

  • Refactor legacy code. You can implement new code and run the property-based against legacy code. To make sure that new code still have the same properties as old version.
  • Simple implementation vs Optimized implementation. Many situations that you have to optimize your code for performance and sacrifice readability. You can implement simple and readability version and test against optimized version. To make sure that optimized version still working as expected. In additional, Elixir has process and message passing mechanism which always introduce complexity to the application. This approach could be a good test tool.
  • Codebase maintainability is also one concern of this pattern as well. Since, our codebase will contain 2 different versions of the same thing. We have to think wisely in order to use it.

Last Thought?

I definitely feel more confident on my code with property-based testing. In the same time, to introduce more tests mean more time for maintaining them.

There are few questions which always to think of — which function should have property-based testing? — How important is it? which properties should we test? — How important is it?

One trick to answer those question is thinking on business side. For example, What would be the business cost if this function behave wrongly?

Thanks & Hope this helpful