Open in app

Sign in

Write

Sign in

Decoding Dynamic JSON Data

Peter Bi
7 min readMay 29, 2023

1. Introduction

In previous articles, we explored how to encode and decode HCL data using the Golang package determined. Interestingly, determined was initially developed for decoding JSON data of the interface type.

The core Golang package encoding/json is an exceptional library for managing JSON data. However, to decode the interface type, it necessitates writing a customized Unmarshaler for the target object. While this isn’t typically a challenging task, it often results in repetitive code for different types of objects and packages.

Therefore, determined was created to streamline the coding process and enhance productivity.

2. Protobuf

Following the idea in reflect, we use protobuf to implement Unmarshaler.

The following proto interprets an interface type in dynamic JSON data:

syntax = "proto3";

package det;

option go_package = "./det";

message Struct {
 string class_name = 1;
 map<string, Value> fields = 2;
}

message Value {
  // The kind of value.
  oneof kind {
    Struct single_struct   = 1;
    ListStruct list_struct = 2;
    MapStruct map_struct   = 3;
  }
}

message ListStruct {
  repeated Struct list_fields = 1;
}

message MapStruct {
  map<string, Struct> map_fields = 1;
}

where class_name is the go struct type name at run-time.

The CLI, protoc will generate the following Golang code:

type Struct struct {
 ClassName string            `protobuf:"bytes,1,opt,name=ClassName,proto3" json:"ClassName,omitempty"`
 Fields    map[string]*Value `protobuf:"bytes,2,rep,name=fields,proto3" json:"fields,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
}

type Value struct {
 // The kind of value.
 //
 // Types that are assignable to Kind:
 // *Value_SingleStruct
 // *Value_ListStruct
 // *Value_MapStruct
 Kind isValue_Kind `protobuf_oneof:"kind"`
}
....

3. NewStruct

There’s no need for users to interact directly with the aforementioned proto-generated package. Instead, the required step involves creating a Golang map to interpret the interface. This map is then passed to the NewStruct function to generate a new Struct:

// NewStruct constructs a Struct from a generic Go map.
// The map keys of v must be valid UTF-8.
// The map values of v are converted using NewValue.
func NewStruct(name string, v ...map[string]interface{}) (*Struct, error)
// 
// NewValue conversion:
// ╔═══════════════════════════╤══════════════════════════════╗
// ║ Go type                   │ Conversion                   ║
// ╠═══════════════════════════╪══════════════════════════════╣
// ║ string                    │ ending SingleStruct value    ║
// ║ []string                  │ ending ListStruct value      ║
// ║ map[string]string         │ ending MapStruct value       ║
// ║                           │                              ║
// ║ [2]interface{}            │ SingleStruct value           ║
// ║ [][2]interface{}          │ ListStruct value             ║
// ║ map[string][2]interface{} │ MapStruct value              ║
// ║                           │                              ║
// ║ *Struct                   │ SingleStruct                 ║
// ║ []*Struct                 │ ListStruct                   ║
// ║ map[string]*Struct        │ MapStruct                    ║
// ╚═══════════════════════════╧══════════════════════════════╝

Fields of primitive data types, or with defined go struct, should be ignored, since they will be decoded automatically by encoding/json.

Here are example usages of NewStruct.

Single Interface:

Here geo contains interface field Shape.

type geo struct {
    Name  string `json:"name" hcl:"name"`
    Shape inter  `json:"shape" hcl:"shape,block"`
}

type inter interface {
    Area() float32
}

type square struct {
    SX int `json:"sx" hcl:"sx"`
    SY int `json:"sy" hcl:"sy"`
}

func (self *square) Area() float32 {
    return float32(self.SX * self.SY)
}

type circle struct {
    Radius float32 `json:"radius" hcl:"radius"`
}

func (self *circle) Area() float32 {
    return 3.14159 * self.Radius
}

Assume a serialized JSON of geo is received, where field Shape is known to be circle at run-time. To decode Shape of type circle, build the following spec:

spec, err := NewStruct(
  "geo", map[string]interface{}{"Shape": "circle"})

If Shape is type square, build this spec:

spec, err = NewStruct(
  "geo", map[string]interface{}{"Shape": "square"})

List of interface:

Here picture contains Drawings, which is a slice of interface. The spec for serialized slice of square, circle or combination of square and circle, are

type picture struct {
    Name     string   `json:"name" hcl:"name"`
    Drawings []inter  `json:"drawings" hcl:"drawings,block"`
}

# incoming data is slice of square, size 2
spec, err := NewStruct(
  "Picture", map[string]interface{}{
    "Drawings": []string{"square", "square"}})

# the first element is square and the second circle
spec, err := NewStruct(
  "Picture", map[string]interface{}{
    "Drawings": []string{"square", "circle"}})

# if all elements of interface slice is type square,
spec, err := NewStruct(
  "Picture", map[string]interface{}{
    "Drawings": []string{"square"}})

Note that if all elements are of the same type square in Drawing, just pass 1-element array []string{“square”}.

Map of interface:

Here Shapes is a map of interface:

type geometry struct {
    Name   string           `json:"name" hcl:"name"`
    Shapes map[string]inter `json:"shapes" hcl:"shapes,block"`
}

spec, err := NewStruct(
  "geometry", map[string]interface{}{
    "Shapes": map[string]string{"k1":"square", "k2":"square"}})

# if all values of interface map is type square
spec, err := NewStruct(
  "Picture", map[string]interface{}{
    "Shapes": []string{"square"}})

Nested:

In toy, Geo is of type geo which contains interface Shape:

type toy struct {
    Geo     geo     `json:"geo" hcl:"geo,block"`
    ToyName string  `json:"toy_name" hcl:"toy_name"`
    Price   float32 `json:"price" hcl:"price"`
}

spec, err = NewStruct(
  "toy", map[string]interface{}{
    "Geo": [2]interface{}{
      "geo", map[string]interface{}{"Shape": "square"}}})

Nested of nested:

Here child has field Brand which is a map of nested toy:

type child struct {
    Brand map[string]*toy `json:"brand" hcl:"brand,block"`
    Age   int  `json:"age" hcl:"age"`
}

spec, err = NewStruct(
  "child", map[string]interface{}{
    "Brand": [][2]interface{}{
      "k1":[2]interface{}{"toy", map[string]interface{}{
        "Geo": [2]interface{}{
          "geo", map[string]interface{}{"Shape": "circle"}}}},
      "k2":[2]interface{}{"toy", map[string]interface{}{
        "Geo": [2]interface{}{
          "geo", map[string]interface{}{"Shape": "square"}}}},
    },
  },
)

4. Use JsonUnmarshal to Decode JSON

To decode JSON to object containing interface types, use JsonUnmarshal:

// JsonUnmarshal unmarshals JSON data with interfaces determined by spec.
//
//   - dat: JSON data
//   - current: object as pointer
//   - spec: *Struct
//   - ref: struct map, with key being string name and value reference to struct
func JsonUnmarshal(dat []byte, current interface{}, spec *Struct, ref map[string]interface{}) error

The following program decodes JSON data1 into object child:

package main

import (
    "fmt"

    "github.com/genelet/determined/det"
)

type geo struct {
    Name  string `json:"name" hcl:"name"`
    Shape inter  `json:"shape" hcl:"shape,block"`
}

type inter interface {
    Area() float32
}

type square struct {
    SX int `json:"sx" hcl:"sx"`
    SY int `json:"sy" hcl:"sy"`
}

func (self *square) Area() float32 {
    return float32(self.SX * self.SY)
}

type circle struct {
    Radius float32 `json:"radius" hcl:"radius"`
}

func (self *circle) Area() float32 {
    return 3.14159 * self.Radius
}

type toy struct {
    Geo     geo     `json:"geo" hcl:"geo,block"`
    ToyName string  `json:"toy_name" hcl:"toy_name"`
    Price   float32 `json:"price" hcl:"price"`
}

type child struct {
    Brand map[string]*toy `json:"brand" hcl:"brand,block"`
    Age   int  `json:"age" hcl:"age"`
}

func main() {
    data1 := `{
"age" : 5,
"brand" : {
"abc1" : {
    "toy_name" : "roblox",
    "price" : 99.9,
    "geo" : {
        "name" : "medium shape",
        "shape" : { "radius" : 1.234 }
    }
},
"def2" : {
    "toy_name" : "minecraft",
    "price" : 9.9,
    "geo" : {
        "name" : "quare shape",
        "shape" : { "sx" : 5, "sy" : 6 }
    }
}
}
}`

    spec, err := det.NewStruct(
    "child", map[string]interface{}{
        "Brand": map[string][2]interface{}{
            "abc1":[2]interface{}{"toy", map[string]interface{}{
                "Geo": [2]interface{}{
                    "geo", map[string]interface{}{"Shape": "circle"}}}},
            "def2":[2]interface{}{"toy", map[string]interface{}{
                "Geo": [2]interface{}{
                    "geo", map[string]interface{}{"Shape": "square"}}}},
        },
    },
    )
    ref := map[string]interface{}{"toy": &toy{}, "geo": &geo{}, "circle": &circle{}, "square": &square{}}

    c := new(child)
    err = det.JsonUnmarshal([]byte(data1), c, spec, ref)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%v\n", c.Age)
    fmt.Printf("%#v\n", c.Brand["abc1"])
    fmt.Printf("%#v\n", c.Brand["abc1"].Geo.Shape)
    fmt.Printf("%#v\n", c.Brand["def2"])
    fmt.Printf("%#v\n", c.Brand["def2"].Geo.Shape)
}
# the program outputs:5
&main.toy{Geo:main.geo{Name:"medium shape", Shape:(*main.circle)(0xc0000b6468)}, ToyName:"roblox", Price:99.9}
&main.circle{Radius:1.234}
&main.toy{Geo:main.geo{Name:"square shape", Shape:(*main.square)(0xc0000b6350)}, ToyName:"minecraft", Price:9.9}
&main.square{SX:5, SY:6}

5. Customized Unmarshaler of encoding/json

If UnmarshalJSON is implemented on go struct, it is said to have a customized unmarshaler and so Golang core package encoding/json will automatically decode it.

With JsonUnmarshal, we can easily write a customized unmarshaler for child:

type child struct {
    Brand map[string]*toy `json:"brand" hcl:"brand,block"`
    Age   int  `json:"age" hcl:"age"`
    spec  *det.Struct
    ref map[string]interface{}
}
func (self *child) Assign(spec *det.Struct, ref map[string]interface{}) {
    self.spec = spec
    self.ref = ref
}
func (self *child) UnmarshalJSON(dat []byte) error {
    return det.JsonUnmarshal(dat, self, self.spec, self.ref)
}

Now the sample code in Chapter 4 can use encoding/json to decode:

package main

import (
    "encoding/json"
    "fmt"

    "github.com/genelet/determined/det"
)

type geo struct {
    Name  string `json:"name" hcl:"name"`
    Shape inter  `json:"shape" hcl:"shape,block"`
}

type inter interface {
    Area() float32
}

type square struct {
    SX int `json:"sx" hcl:"sx"`
    SY int `json:"sy" hcl:"sy"`
}

func (self *square) Area() float32 {
    return float32(self.SX * self.SY)
}

type circle struct {
    Radius float32 `json:"radius" hcl:"radius"`
}

func (self *circle) Area() float32 {
    return 3.14159 * self.Radius
}

type toy struct {
    Geo     geo     `json:"geo" hcl:"geo,block"`
    ToyName string  `json:"toy_name" hcl:"toy_name"`
    Price   float32 `json:"price" hcl:"price"`
}

type child struct {
    Brand map[string]*toy `json:"brand" hcl:"brand,block"`
    Age   int  `json:"age" hcl:"age"`
    spec  *det.Struct
    ref map[string]interface{}
}

func (self *child) Assign(spec *det.Struct, ref map[string]interface{}) {
    self.spec = spec
    self.ref = ref
}

func (self *child) UnmarshalJSON(dat []byte) error {
    return det.JsonUnmarshal(dat, self, self.spec, self.ref)
}

func main() {
    data1 := `{
"age" : 5,
"brand" : {
"abc1" : {
    "toy_name" : "roblox",
    "price" : 99.9,
    "geo" : {
        "name" : "medium shape",
        "shape" : { "radius" : 1.234 }
    }
},
"def2" : {
    "toy_name" : "minecraft",
    "price" : 9.9,
    "geo" : {
        "name" : "quare shape",
        "shape" : { "sx" : 5, "sy" : 6 }
    }
}
}
}`

    spec, err := det.NewStruct(
    "child", map[string]interface{}{
        "Brand": map[string][2]interface{}{
            "abc1":[2]interface{}{"toy", map[string]interface{}{
                "Geo": [2]interface{}{
                    "geo", map[string]interface{}{"Shape": "circle"}}}},
            "def2":[2]interface{}{"toy", map[string]interface{}{
                "Geo": [2]interface{}{
                    "geo", map[string]interface{}{"Shape": "square"}}}},
        },
    },
    )
    ref := map[string]interface{}{"toy": &toy{}, "geo": &geo{}, "circle": &circle{}, "square": &square{}}

    c := new(child)
    c.Assign(spec, ref)
    err = json.Unmarshal([]byte(data1), c)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%v\n", c.Age)
    fmt.Printf("%#v\n", c.Brand["abc1"])
    fmt.Printf("%#v\n", c.Brand["abc1"].Geo.Shape)
    fmt.Printf("%#v\n", c.Brand["def2"])
    fmt.Printf("%#v\n", c.Brand["def2"].Geo.Shape)
}

The advantage of using a customized unmarshaler is that any Go struct, which encapsulates a child, can directly use encoding/json without worrying about interface fields in the child.

6. Conclusion

The determined package handles dynamic data types in JSON and HCL. It can be downloaded from

https://github.com/genelet/determined

Json
Go
Unmarshal
Json Schema
Golang

--

--

Peter Bi

Written by Peter Bi

2 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams