Basic property-based testing in Go

tristin
tristin
Oct 20, 2018 · 7 min read

Imagine it’s your job is to write tests for your coworker’s black-boxed code.

Your coworker comes to you on Monday and says, “I’ve written the Add function. Can you write some tests to make sure it works?” You think for a few minutes and come up with the following ideas about addition:

  • x + y is the same as y + x
  • 0 + x is equal to x

Wonderful. You write your three unit tests, and your coworker runs them.

package mainimport (
"testing"
)
// Read more about this style of unit testing at:
// https://github.com/golang/go/wiki/TableDrivenTests
func TestAdd(t *testing.T) {
type TestCase struct {
name string
input1 float64
input2 float64
wantOutput float64
}
tests := []TestCase{
{
name: "adding 1 and 2 is 3",
input1: 1,
input2: 2,
wantOutput: 3,
},
{
name: "adding 2 and 1 is 3",
input1: 2,
input2: 1,
wantOutput: 3,
},
{
name: "adding 0 to 3 is 3",
input1: 0,
input2: 3,
wantOutput: 3,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
gotOutput := Add(test.input1, test.input2)
if gotOutput != test.wantOutput {
t.Errorf("output mismatch: got %f but expected %f",
gotOutput,
test.wantOutput)
}
})
}
}

“Great tests,” your coworker says. “I found some bugs and fixed them. The tests pass now.”

// New and improved Add! Lots of very smart people
// are telling me this is the most robust Add function
// they've ever seen.
func Add(x, y float64) float64 {
if x == 1 && y == 2 || x == 2 && y == 1 {
return 3
}
if x == 0 {
return y
}
return -1
}

Your coworker comes to you on Tuesday and says, “Last night we found out Add isn’t working. A customer sent us a negative number, and it brought the whole service down. Could you write some more tests?” You think a little harder and update the tests.

tests := []TestCase{
{
name: "adding a -1 decrements the number by 1",
input1: -1,
input2: 2,
wantOutput: 1,
},
{
name: "adding a number to -1 decrements the number by 1",
input1: 500,
input2: -1,
wantOutput: 499,
},
{
name: "adding 1 and 2 is 3",
input1: 1,
input2: 2,
wantOutput: 3,
},
{
name: "adding 2 and 1 is 3",
input1: 2,
input2: 1,
wantOutput: 3,
},
{
name: "adding 0 to 3 is 3",
input1: 0,
input2: 3,
wantOutput: 3,
},
}

“Awesome,” your coworker says. “These tests found a few more bugs. I fixed those, and the tests pass again.”

// Just a tremendous Add function. Truly one of the best.
// Passes all our test cases.
func Add(x, y float64) float64 {
if x < 0 {
return y - 1
}

if y < 0 {
return x - 1
}

if x == 1 && y == 2 || x == 2 && y == 1 {
return 3
}

if x == 0 {
return y
}

return -1
}

Your coworker comes to you on Wednesday and says, “Last night we found out Add isn’t working.” You put your head into your hands and cry. There’s got to be a better way! The good news is there is, and it’s called property-based testing.

Property-based testing

What is property-based testing? A simple, and maybe inaccurate, definition could be: property-based testing is using your test framework to generate bazillions of unit tests with randomized input for you. You can use property-based testing to enhance your existing unit tests and feel more confident about your code.

Let’s time-travel back to Monday and approach our testing strategy differently…

Your coworker comes to you and says “I’ve written the Add function. Can you write some tests to make sure it works?” You think about the properties of addition (or consult Google) and you come up with the following:

  • Commutative: x + y is the same as y + x
  • Associative: (x + y) + z is the same as x + (y + z)
  • Identity: 0 + x is x
  • Distributive: x * (y + z) is the same as x * y + x * z

These rules are the same for any numbers we can throw at addition. Instead of writing unit tests for all numbers, we can let our test framework do it for us.

Now that you Googled properties of addition, you come up with the following tests using Go’s testing/quick package.

package mainimport (
"testing"
"testing/quick"
)
func TestAdd(t *testing.T) {
// If a property-testing function returns false,
// it means that property failed for some input.
commutative := func(x, y float64) bool {
return Add(x, y) == Add(y, x)
}
associative := func(x, y, z float64) bool {
return Add(x, Add(y, z)) == Add(Add(x, y), z)
}
identity := func(x float64) bool {
return Add(0, x) == x
}
distributive := func(x, y, z float64) bool {
return z*Add(x, y) == Add(z*x, z*y)
}
if err := quick.Check(commutative, nil); err != nil {
t.Errorf("commutative fail: %v", err)
}
if err := quick.Check(associative, nil); err != nil {
t.Errorf("associative fail: %v", err)
}
if err := quick.Check(identity, nil); err != nil {
t.Errorf("identity fail: %v", err)
}
if err := quick.Check(distributive, nil); err != nil {
t.Errorf("distributive fail: %v", err)
}
}

Each property-testing function (commutative, associative, identity, and distributive) will take in any valid float64s generated by the testing/quick package and throw them at Add. Now your coworker’s Add function won’t pass the tests unless it’s really working.

Note: Given a large enough or small enough input for this particular function, these tests may fail because of floating-point precision. One solution I found was to generate float32s and cast them to float64, as below. Depending on your use case, this could be okay. There are some third-party libraries available for dealing with these kinds of issues. GOPTER is one I’ve stumbled across, but have not used.

commutative := func(x, y float32) bool {
x64 := float64(x)
y64 := float64(y)
return Add(x64, y64) == Add(y64, x64)
}

Things to try

What are some properties you might test on the functions below? I’ve put my thoughts directly underneath.

// Join puts together two slices of ints.
// Example:
// xs := []int{1, 2, 3}
// ys := []int{4, 5, 6}
// Join(xs, ys) // => [1, 2, 3, 4, 5, 6]
func Join(xs, ys []int) []int {
return append(xs, ys...)
}
  • xs == Join(xs, [])
  • ys == Join([], ys)
  • length(xs) + length(ys) == length(Join(xs, ys))
  • sum(xs) + sum(ys) == sum(Join(xs, ys))
// InsertBeginning inserts a single value
// into the beginning of a list.
// Example:
// x := 1
// xs := []int{4, 2, 0}
// InsertBeginning(x, xs) // => [1, 4, 2, 0]
func InsertBeginning(x int, xs []int) []int {
return append([]int{x}, xs...)
}
  • length(InsertBeginning(x, [])) == 1
  • length(InsertBeginning(x, xs)) == length(xs) + 1
  • InsertBeginning(x, xs)[0] == x
  • x + sum(xs)== sum(InsertBeginning(x, xs))
// RandomStringFromCharacterSet creates a string of size `n`,
// using only characters from the string `set`.
func RandomStringFromCharacterSet(n int, set string) string {
// implementation...
}
  • What properties does RandomStringFromCharacterSet have?

These are simple examples for property-based testing. Testing functions that take basic data types is not complicated to implement. However, we may want to test a function that accepts structs.

Using structs

So far, the examples have been simple. But sometimes our functions take structs as parameters, not just basic types. How can we do property-based testing on these functions?

We can have Go generate random instances of a struct by implementing the Generator interface, and then make assertions the same way we did before.

Let’s use an example of taking a request from a client and validating it.

package main
import (
"errors"
"fmt"
)
type ClientRequest struct {
requestId string
username *string
password *string
}
// ProcessClientRequest validates a request and returns the error code
// and error message if the request could not be processed. Otherwise it returns
// a status 200.
func ProcessClientRequest(req *ClientRequest) (int, error) {
// A request must have a non-empty requestId
if len(req.requestId) < 1 {
return 400, errors.New("requestId too short: must be at least length 1.")
}

// A password is required if a username is present
if req.username != nil && req.password == nil {
return 400, errors.New("password required")
}

// A username is required if a password is present
if req.password != nil && req.username == nil {
return 400, errors.New("username required")
}

return 200, nil
}

If ProcessClientRequest was a black box to us, we could figure out that when the status is not 200 there should be an error, and when the status is 200 there should be no error. (Check out HTTP Status Codes if you’re interested.)

package mainimport (
"math/rand"
"reflect"
"strconv"
"testing"
"testing/quick"
)
func (r ClientRequest) Generate(rand *rand.Rand, size int) reflect.Value {
var id string
var username *string
var password *string
// I'm using strconv as a quick way to generate a string of some length
// greater than 0.
if rand.Float64() < 0.5 {
id = ""
} else {
id = strconv.FormatInt(rand.Int63(), 2)
}
if rand.Float64() < 0.5 {
username = nil
} else {
s := strconv.FormatInt(rand.Int63(), 2)
username = &s
}
if rand.Float64() < 0.5 {
password = nil
} else {
s := strconv.FormatInt(rand.Int63(), 2)
password = &s
}
req := ClientRequest{id, username, password}
return reflect.ValueOf(req)
}
func TestProcessClientRequest(t *testing.T) {
status4xxIsClientError := func(req ClientRequest) bool {
status, err := ProcessClientRequest(&req)
if err != nil {
return status > 200
}
if err == nil {
return status == 200
}
return false
}
status200HasNoError := func(req ClientRequest) bool {
status, err := ProcessClientRequest(&req)
if err != nil && status == 200 {
return false
}
return true
}
if err := quick.Check(status4xxIsClientError, nil); err != nil {
t.Errorf("status4xxIsClientErrorfail: %v", err)
}
if err := quick.Check(status200HasNoError, nil); err != nil {
t.Errorf("status200HasNoError: %v", err)
}
}

Final thoughts

Property-based testing provides a way to enhance your normal unit tests. By making use of Go’s testing/quick, you can potentially find edge cases in your functions you haven’t handled. Check out this talk on Youtube to learn even more about the testing/quick library.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade