Dynamic JSON umarshalling in Go

When dealing with third-party JSON data (e.g. from an API), it is sometimes the case that field values can have multiple types depending on the context. A common case is when a field can hold either the ID of a sub-object or embed the entire object in an expanded view.

Here is an example of the Spacely Sprockets API:

// Simple API response
{
“desc”: “Sub-object is an ID”,
“sprocket”: 42
}
// Expanded API response
{
“desc”: “Sub-object is an object”,
“sprocket”: {
“id”: 42,
“size”: “large”,
“gears”: 15
}
}

This is bad API design. Please don’t do it. It forces client developers to jump through hoops to unmarshal data. This is especially problematic for statically typed languages like Scala and Go, because your data object’s field must be somehow typed to handle more than one type of data.

For those wondering what a better schema would be, I would point to the Facebook Graph API. They use partial objects everywhere, consistently containing ID, Name, and CreateTime fields. This object can be expanded via query shaping without requiring a different object on the client side to unmarshal. Another alternative is to have two fields (e.g. sprocket and sprocket_id) that store an object and its ID respectively, but this is redundant, can be error-prone, and requires flow-control logic on the client side to figure out which field was passed.

In this post, I’m going to demonstrate a reasonably elegant way of dealing with dynamic JSON field types in Go. The code used in the examples is available in my Github repo.


Unmarshalling

In our example, we need to handle two cases for the “sprocket” field:

  1. sprocket is an integer ID
  2. sprocket is a Sprocket object

Since we can’t have a variable of both types, we’re going to build a Sprocket object and a SprocketWrapper object that embeds a Sprocket (I’ll explain why in a second):

// Example is the top-level API response for this demo
type Example struct {
Desc string `json:”desc”`
Sprocket SprocketWrapper `json:”sprocket”`
}
// Sprocket is a sprocket object in our business logic
type Sprocket struct {
ID int `json:”id”`
Size string `json:”size,omitempty”`
Gears int `json:”gears,omitempty”`
}
// SprocketWrapper wraps a Sprocket object with 
// additional functionality
type SprocketWrapper struct {
Sprocket
Partial bool `json:”-”`
}

Notice that the top-level API object specifies “sprocket” as type SprocketWrapper. When when our code attempts to unmarshal a JSON response…

var example flexjson.Example
err := json.Unmarshal([]byte(exampleJSON), &example)

… Go’s json package will attempt to call UnmarshalJSON() on our wrapper object. In order to correctly unmarshal either an ID or an object, we can now override the default behavior:

func (w *SprocketWrapper) UnmarshalJSON(data []byte) error {
if id, err := strconv.Atoi(string(data)); err == nil {
w.ID = id
w.Partial = true
return nil
}
return json.Unmarshal(data, &w.Sprocket)
}

If an ID is passed, as in the first example response above, we detect it by calling Atoi() on the raw data and store it in the Sprocket object’s ID field. We also mark the object as being a partially populated object using a flag. This allows our API client code to differentiate between a partial object and a sprocket with no Size or Gear information. If an object is passed, the attempt to convert the raw data to an int fails, and we fall through to the packages default unmarshal method on the underlying Sprocket object.

Notice that having the wrapper allows us to preserve the default unmarshalling behavior for Sprocket. This makes things much cleaner.


Marshalling

If you need to marshal your business objects back into JSON for any reason, there are two options for dealing with a partial Sprocket:

  1. Marshal into an object with only the ID field populated and mark that object as partial in the JSON
  2. Mimic the API and output an integer ID

While the first method cleans up the technical debt of the poorly constructed API, the second method may be necessary if you need to PUT/POST back to the same API. It’s also more fun to implement, so let’s do that.

To achieve our goal, let’s write a custom MarshalJSON() method for our wrapper that is aware of partial data:

func (w SprocketWrapper) MarshalJSON() ([]byte, error) {
if w.ID == 0 {
return []byte(“null”), nil
}
if w.Partial {
return []byte(strconv.Itoa(w.ID)), nil
}
return json.Marshal(w.Sprocket)
}

This method can output three different things:

  1. If the object is empty (i.e. it’s ID field is zero), output JSON null
  2. If the object was constructed as a partial, only output the integer ID
  3. If the object is populated, fall through to the default marshal function and output the object’s JSON representation.

Again, notice that we still have Sprocket’s default marshal behavior, thanks to our wrapper object. Additionally, the Sprocket object itself isn’t cluttered up with any fields or methods that are not part of its business logic. This decouples API-level hackery from your core business logic code. Since Go’s MarshalJSON() hook allows any type of data to be output, it’s relatively straightforward to switch between integer and object values when constructing your JSON.

Conclusion

The JSON encoding format allows for schema-less objects, which sometimes makes life difficult for developers. This technique uses Go’s well-designed JSON package to allow for custom object marshalling/unmarshalling with just a few lines of code, and without polluting your primary business objects with JSON format-specific fields or code. Hopefully someone finds this useful, and hopefully someone else finds this before they design their API to have dynamically typed fields. :)

Check out the Github repo for a working example of unmarshalling and marshalling the different types of data.

Show your support

Clapping shows how much you appreciated Nathan Smith’s story.