Detecting cycles: safer JSON marshaling in Go

A large number of cycles

JSON marshaling/unmarshaling in Go is a very powerful feature, but there are limits to the Go structures that can be marshaled. Most of these limits will trigger an error, but having a cycle in your data structure results in a nasty stack overflow:

runtime: goroutine stack exceeds 250000000-byte limit
fatal error: stack overflow

runtime stack:
runtime.throw(0x1131e3, 0xe)
	/usr/local/go/src/runtime/panic.go:616 +0x100
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1054 +0x9a0
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64p32.s:366 +0xc0

goroutine 1 [running]:
reflect.Value.Field(0x1025c0, 0x1040a0d0, 0x199, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/reflect/value.go:780 +0x1a0 fp=0x184c0340 sp=0x184c0338 pc=0xb1740
encoding/json.fieldByIndex(0x1025c0, 0x1040a0d0, 0x199, 0x10414054, 0x1, 0x1, 0x21140, 0x6e27, 0x10f4a0, 0xf9560)
	/usr/local/go/src/encoding/json/encode.go:834 +0x40 fp=0x184c0380 sp=0x184c0340 pc=0xe4cc0
encoding/json.(*structEncoder).encode(0x104462e0, 0x1045a000, 0x1025c0, 0x1040a0d0, 0x199, 0x100)
// *** many more lines of this, until *** //
	/usr/local/go/src/encoding/json/encode.go:653 +0x60 fp=0x184c1ad8 sp=0x184c1ab8 pc=0xed1c0
...additional frames elided...

Because the stack is cut off, these crashes can be extremely difficult to debug in a complex application, particularly if you’re composing together data from many different sources. In such cases it would be useful to be able to identify when there is a cycle, and then be able to see where it is in the structure.

I encountered a bug like this yesterday, and spent hours trying to pin down exactly where the cycle had been inserted. In the end, I put together a package to check for and helpfully print out the location of such cycles: github.com/theothertomelliott/acyclic

Extending the example from earlier, we can check for cycles early to allow us to error out before hitting the panic.

package main
import (
  "encoding/json"
  "fmt"
  
  "github.com/theothertomelliott/acyclic"
)
func main() {
  value := &struct {
    A string
    B interface{}
  }{
    A: "a string",
  }
  // Add a cycle
  value.B = value
  // Check for the cycle
  err := acyclic.Check(value)
  if err != nil {
    fmt.Println(err)
    return
  }
  // With a cycle, we don't try to marshal
  _, _ = json.Marshal(value)
}

The above code will output an error with the path to the cycle.

What if we want to see more detail on where the cycle is? Or we have more than one cycle? We can print the structure to see the cycles:

acyclic.Print(value)

This will output the structure with all cycles replaced with a helpful marker:

*{
  A: "a string"
  B: <CYCLE>
}

Once I applied this to my original stack overflow issue, I found that I’d been using the wrong variable name when inserting into a map. A problem so obvious I just couldn’t see it when reading the code. Being able to see exactly where the cycle was in the structure allowed me to fix the problem in less than a minute.

There are differing opinions on whether or not Go should provide a more helpful message here, and the overflow may be improved in a future version. But for now, checking for cycles when marshaling (or printing) complex structures can save you a lot of time and frustration!