Go: Fuzz Testing in Go

Vincent Blanchon
Oct 25 · 5 min read
Illustration created for “A Journey With Go”, made from the original Go Gopher, created by Renee French.

Fuzzing is a testing technique that loads our program with random data. It is complementary to the usual testing and allows developers to find bugs that would be hard to find with manual generated input. Fuzzing is quite easy to set up in Go programs and can be adapted to almost all kinds of code.

Fuzzing projects

Two projects are available in the Go community for fuzzing: gofuzz by Google and go-fuzz by Dmitry Vyukov, that also works for Google. Both are useful and applicable for different usage. Let’s review them one by one:

  • gofuzz provides a package that can populate your Go struct with random values. It is now your responsibility to write your tests and call this package to get randomized data. This package is perfect when you want fuzzing structured data. Here is an example with fuzzing a struct 50,000 times with random data where the pointers/slice/map have 50% chance to be set as null:
fuzzing structured data
  • go-fuzz is based on the American Fuzzy Lop that has found hundreds of bugs in the most famous software/libraries. Go-Fuzz will run continuously and generate random strings based on the samples you have provided. Then, you will have to parse those strings and explicitly flag them as useable for the test or not. Any interesting generated data that add code coverage or any crash will be reported by the tool. This tool perfectly suits programs that manage strings like XML, JSON, images, etc. Here is a preview of the tool running and finding issues, called crasher:
fuzzing with go-fuzz

Each package has its advantages, and at least one of the two tools should suit your programs and the project you are working on. Let’s review the second tool, which has a more complicated workflow.

Go-Fuzz by Example

Let’s start with an example with the resolution of a bug in the package encoding/xml thanks to fuzzing testing. Here are the steps to reproduce this issue:

  • define theFuzz method that will receive the generated data:
// +build gofuzz

package fuzzing

import "encoding/xml"

type
X struct {
D string `xml:",comment"`
}

func FuzzXMLComment(data []byte) int {
v := new(X)
if xml.Unmarshal(data, v) != nil {
return -1
}
if _, err := xml.Marshal(v); err != nil {
panic(err)
}

return 1
}
  • define an initial corpus that will be used by the tool:
<a>
<!-- my comment -->
<b>foo</b>
</a>

Then, since the bug has been merged in Go 1.6, make sure you revert the commit 97c859f8da0c85c33d0f29ba5e11094d8e691e87 in your standard library — using Go 1.5 that had the bug is not compatible with the last version of go-fuzz. Your mini project should follow this structure:

fuzzing encoding/xml

You now can run go-fuzz-build and go-fuzz -bin=./main.zip -workdir=. to start fuzzing:

fuzzing encoding/xml

After the first few seconds, go-fuzz already found a crasher that is stored in the crashers/ folder:

crasher recorded during fuzzing

The crasher file contains the string that raised a panic:

<a><!------></a>

The .output file contains the panic that occurred:

panic: xml: comments must not contain "--"

Indeed, as per the specification of the XML, there are two constraints in the comments.

the string “ -- " (double-hyphen) must not occur within comments. […]
Note that the grammar does not allow a comment ending in --->

This panic has been fixed in the standard library thanks to go-fuzz, and it now returns an error. Let’s now dive into the package to understand how it managed to find this issue.

Go-Fuzz workflow

As seen in the previous section, the workflow of go-fuzz is composed of two steps:

  • building the tool from the instructions defined in your code via the command go-fuzz-build:

Since the build embeds the Fuzz methods, do not forget to run go-fuzz-build it if you modify one of those methods.

  • running the tool continuously and collecting interesting input and crashes via the command go-fuzz -bin=./my-package.zip -workdir=. :

The corpus generation is the focus of the core of go-fuzz. Dmitry Vyukov made a diagram of this core in the GopherCon 2015:

workflow of the corpus generation

The corpus generation loops on the initial corpus and uses two methods:

  • mutation, that applies small modifications on the bytes to the corpus such as removal, insertion, duplication, swapping, flipping, etc. Here is an example of different mutations that happened before finding the crash:
<a><!------

<a><!------>

<a><!------a>

<a><!------/a>

<a><!------</a>

<a><!------></a>
  • versifying, which is the most advanced method. It learns the structure of the text (numbers, alphanumeric, list key-values, etc.) and then applies the mutation of the different parts. Here is an example of the previous corpus with a mutation on the string only:
<!--  /my commentcomment -->
<b>foo</b>
</a>

Here is another one on the tags:

<>
<!-- my comment -->
<b>foo<:b>
[/a]

Go-fuzz mostly uses mutation (90% of the iterations) while running, but the combination of the two helps to discover bugs:

On xml text after 2.5 hours of fuzzing:
Without versifier fuzzing discovered 902 inputs.
With versifier fuzzing discovered 1055 inputs and versifier discovered 83 inputs.
Versifier generated new inputs + increased fuzzing efficiency by 25%

This workflow is quite efficient and easy to integrate. It can help any packages that deal with text, whether Go standard library or your code.

Fuzzing integration

Fuzzing has been used in the Go standard library since Go 1.5 and has already found more than 200 bugs since then. However, although some packages already have some Fuzz functions such as encoding/csv or image/png, there is no native integration with Go. A discussion has been opened on Github to make fuzzing a first-class citizen in Go.

In terms of tools available online for continuous integration with fuzz testing, two are working with Go and Go-fuzz:

Both have similar prices along with a Github integration. They come with a free account that will allow you to test fuzz testing in your pipeline.

Vincent Blanchon

Written by

French Gopher in Dubai

A Journey With Go

A Journey With Go Language Programming

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade