Unmarshal HCL Data to GO Object
1. Introduction
In the first part of the article, we discussed how to encode a Go object into HCL data. In this section, we will explore how to convert HCL data back into a Go object using the determined package.
The Unmarshal function in determined can
- support a wider range of data types, including map and labels
- provide a powerful yet easy-to-use Struct specification to decode data with a dynamic schema
Similar to JSON, HCL data cannot be decoded into an object if the latter contains an interface field. We need a specification for the actual data structure of the interface at runtime. HCL has the hcldec package to handle this issue.
However, hcldec is not straightforward to use. For instance, describing the following data structure can be challenging:
{
"io_mode": "async",
"services": {
"http": {
"web_proxy": {
"listen_addr": "127.0.0.1:8080",
"processes": {
"main": {
"command": ["/usr/local/bin/awesome-app", "server"]
},
"mgmt": {
"command": ["/usr/local/bin/awesome-app", "mgmt"]
}
}
}
}
}
}hcldec needs a long description:
spec := hcldec.ObjectSpec{
"io_mode": &hcldec.AttrSpec{
Name: "io_mode",
Type: cty.String,
},
"services": &hcldec.BlockMapSpec{
TypeName: "service",
LabelNames: []string{"type", "name"},
Nested: hcldec.ObjectSpec{
"listen_addr": &hcldec.AttrSpec{
Name: "listen_addr",
Type: cty.String,
Required: true,
},
"processes": &hcldec.BlockMapSpec{
TypeName: "process",
LabelNames: []string{"name"},
Nested: hcldec.ObjectSpec{
"command": &hcldec.AttrSpec{
Name: "command",
Type: cty.List(cty.String),
Required: true,
},
},
},
},
},
}
val, moreDiags := hcldec.Decode(f.Body, spec, nil)
diags = append(diags, moreDiags...)Note that hcldec also parses variables, functions and expression evaluations, as we see in Terraform. Those features have only been implemented partially in determined.
In determined, the specification could be written simply as:
spec, err := NewStruct("Terraform", map[string]interface{}{
"services": [][2]interface{}{
{"service", map[string]interface{}{
"processes": [2]interface{}{
"process", map[string]interface{}{
"command": "commandName",
}},
},
}},
},
} which says that service is the only item in list field services; within service, there is field processes, defined to be scalar of process, which contains interface field command and its runtime implementation is commandName. Fields of primitive data type or defined go struct should be ignored in spec, because they will be decoded automatically.
2. Struct and Value
Beneath the surface, we have followed Go’s reflect package to define data Struct and Value in proto message,
syntax = "proto3";
package dethcl;
option go_package = "./dethcl";
message Struct {
string className = 1;
map<string, Value> fields = 2;
}
message Value {
// The kind of value.
oneof kind {
Struct single_struct = 1;
ListStruct list_struct = 2;
}
}
message ListStruct {
repeated Struct list_fields = 1;
}which is auto generated into the Go code:
type Struct struct {
ClassName string `protobuf:"bytes,1,opt,name=className,proto3" json:"className,omitempty"`
Fields map[string]*Value `protobuf:"bytes,2,rep,name=fields,proto3" json:"fields,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
}
type Value struct {
// The kind of value.
//
// Types that are assignable to Kind:
//
// *Value_SingleStruct
// *Value_ListStruct
Kind isValue_Kind `protobuf_oneof:"kind"`
}
type ListStruct struct {
ListFields []*Struct `protobuf:"bytes,1,rep,name=list_fields,json=listFields,proto3" json:"list_fields,omitempty"`
}
...To build a new Struct, use function NewStruct:
func NewStruct(class_name string, v …map[string]interface{}) (*Struct, error)
//
// where v is a nested primative map with
// - key being parsing tag of field name
// - value being the following Struct conversions:
//
// ╔══════════════════╤═══════════════════╗
// ║ Go type │ Conversion ║
// ╠══════════════════╪═══════════════════╣
// ║ string │ ending Struct ║
// ║ [2]interface{} │ SingleStruct ║
// ║ │ ║
// ║ []string │ ending ListStruct ║
// ║ [][2]interface{} │ ListStruct ║
// ║ │ ║
// ║ *Struct │ SingleStruct ║
// ║ []*Struct │ ListStruct ║
// ╚══════════════════╧═══════════════════╝In the following example, the geo type contains interface Shape which is implemented as either circle or square:
type geo struct {
Name string `json:"name" hcl:"name"`
Shape inter `json:"shape" hcl:"shape,block"`
}
type inter interface {
Area() float32
}
type square struct {
SX int `json:"sx" hcl:"sx"`
SY int `json:"sy" hcl:"sy"`
}
func (self *square) Area() float32 {
return float32(self.SX * self.SY)
}
type circle struct {
Radius float32 `json:"radius" hcl:"radius"`
}
func (self *circle) Area() float32 {
return 3.14159 * self.Radius
}At run time, we know the data instance of geo is using type Shape = cirle, so our Struct is:
spec, err := dethcl.NewStruct(
"geo", map[string]interface{}{"Shape": "circle"})and for Shape of square:
spec, err = NewStruct(
"geo", map[string]interface{}{"Shape": "square"})We have ignored field Name because it is a primitive type.
3. More Examples
Type picture has field Drawings which is a list of Shape of size 2:
type picture struct {
Name string `json:"name" hcl:"name"`
Drawings []inter `json:"drawings" hcl:"drawings,block"`
}
# incoming data is slice of square, size 2
spec, err := NewStruct(
"Picture", map[string]interface{}{
"Drawings": []string{"square", "square"}})Type geometry has field Shapes as a map of Shape of size 2:
type geometry struct {
Name string `json:"name" hcl:"name"`
Shapes map[string]inter `json:"shapes" hcl:"shapes,block"`
}
# incoming HCL data is map but MUST be expressed as slice of one label! e.g.
# name = "medium shapes"
# shapes obj5 {
# sx = 5
# sy = 6
# }
# shapes obj7 {
# sx = 7
# sy = 8
# }
spec, err := NewStruct(
"geometry", map[string]interface{}{
"Shapes": []string{"square", "square"}})Type toy has fieldGeo which contains Shape:
type toy struct {
Geo geo `json:"geo" hcl:"geo,block"`
ToyName string `json:"toy_name" hcl:"toy_name"`
Price float32 `json:"price" hcl:"price"`
}
spec, err = NewStruct(
"toy", map[string]interface{}{
"Geo": [2]interface{}{
"geo", map[string]interface{}{"Shape": "square"}}})Type child has field Brand which is a map of the above Nested of nested toy:
type child struct {
Brand map[string]*toy `json:"brand" hcl:"brand,block"`
Age int `json:"age" hcl:"age"`
}
spec, err = NewStruct(
"child", map[string]interface{}{
"Brand": [][2]interface{}{
[2]interface{}{"toy", map[string]interface{}{
"Geo": [2]interface{}{
"geo", map[string]interface{}{"Shape": "circle"}}}},
[2]interface{}{"toy", map[string]interface{}{
"Geo": [2]interface{}{
"geo", map[string]interface{}{"Shape": "square"}}}},
},
},
)4. Unmarshal HCL Data to Object
The decoding function Unmarshal can be used in 4 cases.
- Decode HCL data to object without dynamic schema.
func Unmarshal(dat []byte, object interface{}) error2. Decode data to object without dynamic schema but with label. The labels will be assigned to the label fields in object.
func Unmarshal(dat []byte, object interface{}, labels ...string) error3. Decode data to object with dynamic schema specified by spec and ref.
func UnmarshalSpec(dat []byte, current interface{}, spec *Struct, ref map[string]interface{}) error
//
// spec: describe how the interface fields are interprested
// ref: a reference map to map class names in spec, to objects of empty value.
// e.g.
// ref := map[string]interface{}{"cirle": new(Circle), "geo": new(Geo)}4. Decode data to object with dynamic schema specified by spec and ref , and with label. The labels will be assigned to the label fields in object.
func UnmarshalSpec(dat []byte, current interface{}, spec *Struct, ref map[string]interface{}, label_values ...string) error
//
// spec: describe how the interface fields are interprested
// ref: a reference map to map class names in spec, to objects of empty value.
// e.g.
// ref := map[string]interface{}{"cirle": new(Circle), "geo": new(Geo)}
In the following example, we decode data to child of type Nested of nested, which contains multiple interfaces and maps,
package main
import (
"fmt"
"github.com/genelet/determined/dethcl"
)
func main() {
data1 := `
age = 5
brand "abc1" {
toy_name = "roblox"
price = 99.9
geo {
name = "medium shape"
shape {
radius = 1.234
}
}
}
brand "def2" {
toy_name = "minecraft"
price = 9.9
geo {
name = "quare shape"
shape {
sx = 5
sy = 6
}
}
}
`
spec, err := dethcl.NewStruct("child", map[string]interface{}{
"Brand": [][2]interface{}{
[2]interface{}{"toy", map[string]interface{}{
"Geo": [2]interface{}{
"geo", map[string]interface{}{"Shape": "circle"}}}},
[2]interface{}{"toy", map[string]interface{}{
"Geo": [2]interface{}{
"geo", map[string]interface{}{"Shape": "square"}}}},
},
})
ref := map[string]interface{}{"toy": &toy{}, "geo": &geo{}, "circle": &circle{}, "square": &square{}}
c := new(child)
err = dethcl.UnmarshalSpec([]byte(data1), c, spec, ref)
if err != nil {
panic(err)
}
fmt.Printf("%v\n", c.Age)
fmt.Printf("%#v\n", c.Brand["abc1"])
fmt.Printf("%#v\n", c.Brand["abc1"].Geo.Shape)
fmt.Printf("%#v\n", c.Brand["def2"])
fmt.Printf("%#v\n", c.Brand["def2"].Geo.Shape)
}
# the program outputs:
5
&main.toy{Geo:main.geo{Name:"medium shape", Shape:(*main.circle)(0xc000018650)}, ToyName:"roblox", Price:99.9}
&main.circle{Radius:1.234}
&main.toy{Geo:main.geo{Name:"quare shape", Shape:(*main.square)(0xc000018890)}, ToyName:"minecraft", Price:9.9}
&main.square{SX:5, SY:6}The output is populated properly into specified objects.
5. Summary
Determined is a robust and innovative package for encoding and decoding HCL. It enhances the capabilities of the existing Go package gohcl for static data processing, and hcldec for dynamic data processing. This brings HCL one step closer to becoming a universal data interchange format, akin to JSON and YAML.
Determined is open-sourced. You are welcome to send issues to the author.