Pitfalls of GoLang interface streaming to JSON (part2)

Photo by Aaron Burden on Unsplash

In the first part of this short series, I covered the basic techniques required to stream an interface type to and from JSON in Go. Previously I promised that I would clean up the functions, nobody likes an if statement in their streaming code, and I’ll talk about why reflection doesn’t really help. I’ll also try and convince you that not using reflection isn’t that bad a thing.

If you have not read part 1, stop here and go for a read

Pitfalls of GoLang interface streaming to JSON (part1)

Ok, so I assume that you have followed the steps and now have a working round-trip for a couple of structures. One of the items I glossed over during the first part was the choice of the type used in the interface definition.

type Type string
type MyInterface interface {
   Type() Type
}

I create a type alias for string and a simple function that needs to be implemented that returns this type. If you followed along you’ll also notice that I used this type in the streamed version of the struct.

func (x StructX) MarshalJSON() ([]byte, error) {
   var xr struct {
      X               string      `json:"x"`
      MyInterface     MyInterface `json:"my_interface"`
      MyInterfaceType Type        `json:"my_interface_type"`
   }
   xr.X = x.X
   xr.MyInterface = x.MyInterface
   xr.MyInterfaceType = x.MyInterface.Type()
   return json.Marshal(xr)
}

and yet in the decoder I listed by hand these types. This is clearly error prone but without some form of lookup this is the best you can do. So now we add a simple lookup map

var lookup = make( map[Type]MyInterface )

And a function that we can use to register our types

func Register(iface MyInterface) {
   lookup[iface.Type()] = iface
}

This map is a global and so any mutations will need to be protected but as you will see we are only mutating this map at module initiation time and the Go runtime ensures that the order of init calls is correct.

Effective Go - The Go Programming Language

func init() {
   Register( StructA{} )
   Register( StructB{} )
}

Now when we first import a module or run our main the init functions get called and our map gets populated. We can now redefine decoder as

func (x *StructX) UnmarshalJSON(b []byte) error {
   var xr struct {
      X               string          `json:"x"`
      MyInterface     json.RawMessage `json:"my_interface"`
      MyInterfaceType Type            `json:"my_interface_type"`
   }
   err := json.Unmarshal(b, &xr)
   if err != nil {
      return err
   }
   x.X = xr.X
   myInterface, ok := lookup[xr.MyInterfaceType]
   if !ok {
      return fmt.Errorf("unregistered interface type : %s", xr.MyInterfaceType)
   }
   err = json.Unmarshal(xr.MyInterface, myInterface)

   if err != nil {
      return err
   }
   x.MyInterface = a
   return nil
}

This looks great but doesn’t work! We run into the same problem as before

panic: json: Unmarshal(non-pointer main.StructA)

It’s that pointer interface duality thing all over again. The issue is that our map has a value-based entry, and we need a pointer but if we store a pointer then we will be sharing the same instance of the interface. My head starts to spin at this point. There are a number of ways to solve this. We can rely on reflection and create a new instance of the class, we can add another method that returns a new instance of the type, or we can use a lambda function to return a new value. Perhaps the easiest it to add another method to our interface.

type Type string
type MyInterface interface {
   Type() Type
   New() MyInterface
}

var lookup = make(map[Type]MyInterface)

func Register(iface MyInterface) {
   lookup[iface.Type()] = iface
}

and now we implement the functions

func (_ StructA) New() MyInterface {
   return &StructA{}
}

func (_ StructB) New() MyInterface {
   return &StructB{}
}

and update the decoder

func (x *StructX) UnmarshalJSON(b []byte) error {
   var xr struct {
      X               string          `json:"x"`
      MyInterface     json.RawMessage `json:"my_interface"`
      MyInterfaceType Type            `json:"my_interface_type"`
   }
   err := json.Unmarshal(b, &xr)
   if err != nil {
      return err
   }
   x.X = xr.X
   myInterfaceFunc, ok := lookup[xr.MyInterfaceType]
   if !ok {
      return fmt.Errorf("unregistered interface type : %s", xr.MyInterfaceType)
   }
   myInterface := myInterfaceFunc.New()
   err = json.Unmarshal(xr.MyInterface, myInterface)

   if err != nil {
      return err
   }
   x.MyInterface = myInterface
   return nil
}

and now it all works, we can register new streaming types and have them constructed automatically. Note you still have to be careful in that there is no direct way to ensure that an interface points to a pointer or a value. You can do this with reflection but that makes for a more complex environment.

You may ask, why did you not use reflection? Well unlike reflection in Java and some other languages you can’t just create an instance of a type by name. You have to first register those types in a map so that you can duplicate the type. This is why the golang gob API requires a register function.

gob

with this implementation

https://cs.opensource.google/go/go/+/refs/tags/go1.19:src/encoding/gob/type.go;l=836

Since we are in complete control here it is marginally clearer to use an additional method on the interface.

So that’s it we can now round-trip interfaces to JSON and have explored delayed decoding and how to construct types by name.

Full code follows

package main

import (
   "encoding/json"
   "fmt"
)

type Type string
type MyInterface interface {
   Type() Type
   New() MyInterface
}

var lookup = make(map[Type]MyInterface)

func Register(iface MyInterface) {
   lookup[iface.Type()] = iface
}

func init() {
   Register(StructA{})
   Register(StructB{})
}

type StructA struct {
   A float64 `json:"a"`
}
type StructB struct {
   B string `json:"b"`
}
type StructX struct {
   X           string      `json:"x"`
   MyInterface MyInterface `json:"my_interface"`
}

type StructXRAW struct {
   X           string          `json:"x"`
   MyInterface json.RawMessage `json:"my_interface"`
}

func (_ StructA) Type() Type {
   return "StructA"
}

func (_ StructB) Type() Type {
   return "StructB"
}

func (_ StructA) New() MyInterface {
   return &StructA{}
}

func (_ StructB) New() MyInterface {
   return &StructB{}
}

// Check that we have implemented the interface
var _ MyInterface = (*StructA)(nil)
var _ MyInterface = (*StructB)(nil)

func (x StructX) MarshalJSON() ([]byte, error) {
   var xr struct {
      X               string      `json:"x"`
      MyInterface     MyInterface `json:"my_interface"`
      MyInterfaceType Type        `json:"my_interface_type"`
   }
   xr.X = x.X
   xr.MyInterface = x.MyInterface
   xr.MyInterfaceType = x.MyInterface.Type()
   return json.Marshal(xr)
}

func (x *StructX) UnmarshalJSON(b []byte) error {
   var xr struct {
      X               string          `json:"x"`
      MyInterface     json.RawMessage `json:"my_interface"`
      MyInterfaceType Type            `json:"my_interface_type"`
   }
   err := json.Unmarshal(b, &xr)
   if err != nil {
      return err
   }
   x.X = xr.X
   myInterfaceFunc, ok := lookup[xr.MyInterfaceType]
   if !ok {
      return fmt.Errorf("unregistered interface type : %s", xr.MyInterfaceType)
   }
   myInterface := myInterfaceFunc.New()
   err = json.Unmarshal(xr.MyInterface, myInterface)

   if err != nil {
      return err
   }
   x.MyInterface = myInterface
   return nil
}

func main() {
   // Create an instance of each a turn to JSON
   xa := StructX{X: "xyz", MyInterface: StructA{A: 1.23}}
   xb := StructX{X: "xyz", MyInterface: StructB{B: "hello"}}

   xaJSON, _ := json.Marshal(xa)
   xbJSON, _ := json.Marshal(xb)
   println(string(xaJSON))
   println(string(xbJSON))

   var newX StructX
   err := json.Unmarshal(xaJSON, &newX)
   if err != nil {
      panic(err)
   }
   err = json.Unmarshal(xbJSON, &newX)
   if err != nil {
      panic(err)
   }
}