How to Unmarshal an Array of JSON Objects of Different Types into a Go Struct

Alain Drolet
Mar 3 · 11 min read

Abstract

Example code of a complete solution can be found (in the Go Playground) here:
https://play.golang.org/p/6etRTUZ5xgw

The example code contains extra comments and is ready to be compiled and run.

Context

The specific difficulty addressed in this article is the fact that the objects are not of the same types, but could belong to a common category.

If we were to describe this using Java (OO) terminology we could say that the problem is to serialize a list of objects whose types are subclasses of a common superclass.

You could think of all objects being birds, but of different species. Or being transportation means: like bicycles, cars, trains, or planes, all being used for transportation but with some characteristics specific to their exact subtypes.

In this article I’ll be using the somewhat contrived example of a wheel set. The set knows 3 wheel subtypes.

In the set all wheel subtypes share some common attributes:

  • ID, a string naming the wheel instance
  • Radius, the external radius of the wheel in whatever unit you like
  • Type, an enum value to tell us how to handle the type-specific attributes

Then each subtypes has extra attributes. In this article we have:

  • Solid Wheel
    This type of wheel is what you could have found on chariots in antique Rome. It has only one attribute: material. Material value in this case could be “wood”.
  • Spoked Wheel
    Think of a bicycle or motorcycle wheel with spokes. Its attributes are: the number of spokes, and the inner radius. The difference between the external radius and the inner one, give you the tire thickness.
  • Tire Wheel
    It is the standard car wheel with a tire. Its attributes are: an inner radius similar to the spoked wheel, and a fill type. The fill type is an enum that has the values:
    - “inflated” like a common tire, and
    - “plain” for a solid rubber tire.

In Java we could have the common attributes defined in a superclass and the subtype attributes defines in subclasses.

In C one could consider using a union to implement a such model.

Go is not object oriented, and is very type-safe (which I like). So, a priori mixing types freely does not look like a match made in heaven, but with some creativity we can solve this.

The Goal

// JSON list of wheels
{
"name": "my-wheel-list",
"wheels": [
{
"id": "chariot_wheel_0142",
"radius": 17,
"type": "solid",
"material": "wood"
},
{
"id": "bicycle_wheel_8450",
"radius": 20,
"type": "spoked",
"spoke_count": 30,
"inner_radius": 19
},
{
"id": "car_wheel_2743",
"radius": 20,
"type": "tire",
"inner_radius": 17,
"fill_type": "inflated"
}
]
}

A Solution

One obvious way is to not use a single json object for all the wheel subtypes. The json object could have three embedded objects, and only one would be set, but that is not my goal today (well at least in the json representation).

Another possible simple solution would be to model wheel objects with Go maps. In a map-based solution the attributes are modeled using Go maps with strings as keys.

As indicated above I like the type-safe characteristic of Go, so I’m looking for better than maps of maps with string being used as keys all over.

So if you can agree with the constrains I set on this problem, let’s see how we can solve it.

The Go Model

The wheel objects will look like this in Go:

type Wheel struct {
Id string `json:"id"`
Radius int `json:"radius"`
Type int `json:"type"`
attr interface{}
}
// WARNING:
// The type Wheel will be renamed as more details of the solution are exposed!
// The attribute Type will also be redefined to be of an enum type

type SolidAttributes struct {
Material string `json:"material"`
}

type SpokedAttributes struct {
SpokeCount int `json:"spoke_count"`
InnerRadius int `json:"inner_radius"`
}

type TireAttributes struct {
Fill FillType `json:"fill_type"` // FillType is an enum made of int. See details further down.
InnerRadius int `json:"inner_radius"`
}

The Go representation is made of a struct, Wheel, that contains the common attribute, Id and Radius. It has a Type attribute to tell us what subtype of wheel that instance represents.

To store the subtype specific attributes we will be using three other structs.
A pointers to one of these will be stored in the private attribute of Wheel called, attr.

Tags (e.g. `json:"id"`) will also be used to map from the json attribute names to/from the struct attribute names.

Unmarshaling

For this we will be using the encoding/json package.

It provides a function that does the json to Go conversion on basic types and structs. The function signature is:

func Unmarshal(data []byte, v interface{}) error

A priori we could try some code like this:

var wheelJson = `
{
"id": "bicycle_wheel_8450",
"radius": 20,
"type": 1,
"spoke_count": 30,
"inner_radius": 19
}
`
wheelObj := &Wheel{}
err := json.Unmarshal([]byte(wheelJson), wheelObj)

This would populate the attributes Id, Radius, and Type of the wheelObj struct.

The Go attribute, attr, would remain a nil pointer, because it is private and the json package cannot operate on private attributes, and because the json sting does have a such attribute anyway.

The other json attributes: spoke_count and inner_radius would just be skipped since they do not exist in the Wheel struct nor is there any json tags with such names.

This however is still a start. It gives us the common attributes and we know what subtypes we are trying to unmarshal.

The json package allows you to define an UnmarshalJSON and a MarshalJSON method on a type.
If any is found it will be used and the default unmarshaler or marshaler functions will not.

We could try to use a such custom unmarshaler to parse the json once to get the common attributes and in particular the type, then call it a second time to populate an object specific to the type we discovered in the first pass.

This two passes parsing could look like this:

func (w *Wheel) UnmarshalJSON(b []byte) error {            // Line A
// populate common attributes in w
err := json.Unmarshal(b, w) // Line B
...
// second pass to get attributes of specific sub-types
switch w.Type {
case WheelType_Spoked:
subTypeAttr := &SpokedAttributes{}
err := json.Unmarshal([]byte(wheelJson), subTypeAttr)
...

This is almost right except that our custom function defined on Line A, is also the one being called on line B. This would give us an infinite recursion and a stack overflow. Not really what we want!

Parsing json requires the handling of many primitive types and the recursive handling of structs. The easiest solution path is to leverage this well done functionality from the json package. What we want is only to add a reasonably simple wrapper on top of what json already provides us.

This can be achieve by defining a new type (basically and alias of Wheel). Different types can have different methods attached to them. With two types that refer to the same Wheel struct we can now control when the default (and fancy) marshaler is called and when our custom marshaller is called.

We will define our 2 types by renaming the Wheel struct, InnerWheel (feel free to find a better name!) and we will define a new Wheel type like this:

type Wheel InnerWheel

The bulk of the code is likely to refer to the Wheel type, but for most use, Wheel or InnerWheel could be used.

The final code for the type definition and the unmarshaler looks like this.

Note that this code refers to code that converts enum from string to int.
We will expand on this in a short while.

type Wheel InnerWheel  // Wheel is a new type that uses the same struct definition as InnerWheel

type InnerWheel struct {
Id string `json:"id"`
Radius int `json:"radius"`
Type WheelType `json:"type"`
attr interface{}
}

type SolidAttributes struct {
Material string `json:"material"`
}

type SpokedAttributes struct {
SpokeCount int `json:"spoke_count"`
InnerRadius int `json:"inner_radius"`
}

type TireAttributes struct {
Fill FillType `json:"fill_type"`
InnerRadius int `json:"inner_radius"`
}

func (w *Wheel) UnmarshalJSON(b []byte) error {
// populate common attributes using default (InnerWheel) json Unmarshaler
err := json.Unmarshal(b, (*InnerWheel)(w))
if err != nil {
return err
}

// then unmarshal the sub-type specific attributes in an object stored in the InnerWheel attr attribute.
switch w.Type {
case WheelType_Solid:
subTypeAttr := &SolidAttributes{}
err := json.Unmarshal(b, subTypeAttr)
if err != nil {
return err
}
w.attr = subTypeAttr
case WheelType_Spoked:
subTypeAttr := &SpokedAttributes{}
err := json.Unmarshal(b, subTypeAttr)
if err != nil {
return err
}
w.attr = subTypeAttr
case WheelType_Tire:
subTypeAttr := &TireAttributes{}
err := json.Unmarshal(b, subTypeAttr)
if err != nil {
return err
}
w.attr = subTypeAttr
default:
return fmt.Errorf("Wheel.UnmarshalJSON: unexpected type; type = %s (%d)", w.Type.String(), w.Type)
}

return nil
}

The idea of using an alias type and a two-pass Unmarshaler was borrowed from the excellent article by Jon Calhoun on Advanced Encoding and Decoding Techniques.

If the current article is of interest to you, then you should probably read Jon’s article as well.

He is explaining similar techniques and more.

Note that this approach could be used with objects that do not share a lot in the superclass. A common Type attribute is all we really need.

A program that uses wheel objects, once it checked which sub-wheel type it is, could get the subtype specific attributes in a type-safe way by using a getter method on the wheel struct.

One such method could be:

// SolidAttr returns a pointer to a SolidAttributes struct.
// The object is read from the private `attr` attribute of the Wheel/InnerWheel.
// It is an error to call this method if Wheel.Type is not equal to WheelType_Solid.
func (w *Wheel) SolidAttr() (*SolidAttributes, error) {
solidAttr, ok := w.attr.(*SolidAttributes)
if ! ok {
err := fmt.Errorf("Implementation Error: Wheel %s failed assertion of attr field as Solid.", w.Id)
return nil, err
}
return solidAttr, nil
}

You should also have the getters: SpokedAttr() and TireAttr().

Marshaling

Basically we want to build a json string that represents the common attributes followed by the subtype specific attributes.

"{" + common-attributes-string + "," + specific-attributes-string + "}"

For this we create a wrapper marshaller method on the Wheel type. It will call the default marshaller of the InnerWheel giving us the string for the common attributes.

Then we call the marshaler on the object in the attr attributes and we get the specific attributes string.

This part is given to us free by the json package.

Our “toughest” job is to concatenate both the common and specific strings. The bulk of the complexity (if you see complexity here) is to be careful about building the overall string when parts could be missing.

The Wheel marshaler looks like this:

// MarshalJSON returns the json string representation of the Wheel object in the receiver w.
func (w *Wheel) MarshalJSON() ([]byte, error) {

commonStr, comErr := json.Marshal((*InnerWheel)(w))
if comErr != nil {
return []byte(""), comErr
}

attrStr, attErr := json.Marshal(w.attr)
if attErr != nil {
return []byte(""), attErr
}

commExists := len(commonStr) > 2
attrExists := len(attrStr) > 2 && attrStr[0] == '{' // if attr is nil, attrStr is set to "null"

buf := make([]byte, 0, (len(commonStr)+len(attrStr)-1))

buf = append(buf, byte('{'))

if commExists {
buf = append(buf, commonStr[1:len(commonStr) - 1]...)
}

if commExists && attrExists {
buf = append(buf, byte(','))
}

if attrExists {
buf = append(buf, attrStr[1:len(attrStr) - 1]...)
}

buf = append(buf, byte('}'))

return buf, nil
}

Converting Enums From String To Int And Back

Creating a new type (typically an alias of int) and using it for all the members of an enum set, helps to make the code type-safe.

If you want to print text labels instead of the int values or convert the enum back and forth between their int and string or json representations you need to do some work.

This is documented on many sites on the web. For completeness here is how I did mine. Feel free to explore variations on the theme!

The key is to have a list of int values and a way to map these values to the label and vice versa.

An enum set with labels can be implemented with the following four building blocks:

  • A type for the enum set.
  • A const block to define each enum values.
    It is often initialized using Go’s iota.
  • A slice of string to represent the labels for each enum.
    The const block and the slice must be exactly aligned!
  • A map where the key is a label, and the values are the enum values.
    The list of keys must be an exact match of the label list above. Order in a map is irrelevant.
// WheelType is the list of wheel subtypes we understand
type WheelType int

const (
WheelType_Solid WheelType = iota
WheelType_Spoked
WheelType_Tire
)

var wheelNames = [...]string{
"solid",
"spoked",
"tire",
}

var wheelValues = map[string]WheelType{
"solid": WheelType_Solid,
"spoked": WheelType_Spoked,
"tire": WheelType_Tire,
}

With this we can implement a Stringer interface to convert an enum value to its text label. This way the Stringer method String() can be used in functions of the fmt package, and in marshalers.

// String converts a WheelType value to its string representation
func (t WheelType) String() string {

// This test will help detect when the wheelNames array is out of sync with the list of WheelType values
// At the very least you should avoid a panic
if t < 0 || int(t) >= len(wheelNames) {
return ""
}
return wheelNames[t]
}

To convert from a text label (e.g. as found in a json object) to its enum value we can define a function like:

// WheelTypeValue take a string as an argument and attempt to return the equivalent WheelType enum value.
// If it is impossible to convert the string an error is returned, and the value returned should be ignored.
func WheelTypeValue(name string) (WheelType, error) {
val, found := wheelValues[name]
if !found {
// we must return some value, even on error, let's pick the default value WheelType_Solid
return WheelType_Solid, errors.New(fmt.Sprintf("name '%s' not found in enum WheelType", name))
}

return val, nil
}

To convert to and from json we will use the following marshaler and unmarshaler methods.

// MarshalJSON converts an enum instance of WheelType to its string representation surrounded by double-quotes.
func (t WheelType) MarshalJSON() ([]byte, error) {
buffer := bytes.NewBufferString(`"`)
buffer.WriteString(t.String()) // calling our Stringer
buffer.WriteString(`"`)
return buffer.Bytes(), nil
}

// UnmarshalJSON attempts to unmarshal a quoted json string to its enum value
// An error is returned if the string does not represent a known WheelType value
func (t *WheelType) UnmarshalJSON(b []byte) error {
// unquote the argument; we could possibly use strconv.Unquote
var valname string
err := json.Unmarshal(b, &valname)
if err != nil {
return err
}

val , err := WheelTypeValue(valname) // calling out text-to-val function
if err != nil {
return fmt.Errorf("Cannot convert '%s' into a value of the WheelType enum", valname)
}

*t = val // populate the attribute value of the parent struct
return nil
}

This concludes this article.
I hope it can be useful to some of you.

Remember you can read and experiment with the full example in Go Playground:
https://play.golang.org/p/6etRTUZ5xgw

Alain Drolet

Written by

Software Architect § I spent the bulk of my career designing Network Management systems for telecom § www.linkedin.com/in/alain-drolet-505694