Data Format Conversions Among HCL, JSON and YAML

Introduction

Hashicorp Configuration Language (HCL) is a user-friendly data format for structured configuration. It combines parameters and declarative logic in a way that is easily understood by both humans and machines. HCL is integral to Hashicorp’s cloud infrastructure automation tools, such as Terraform and Nomad. With its robust support for expression syntax, HCL has the potential to serve as a general data format with programming capabilities, making it suitable for use in no-code platforms.

However, in many scenarios, we still need to use popular data formats like JSON and YAML alongside HCL. For instance, Hashicorp products use JSON for data communication via REST APIs, while Docker or Kubernetes management in Terraform requires YAML.

Question

An intriguing question arises: Is it possible to convert HCL to JSON or YAML, and vice versa? Could we use HCL as the universal configuration language in projects and generate YAML or JSON with CLI or within Terraform on the fly?

Unfortunately, the answer is generally no. The expressive power of HCL surpasses that of JSON and YAML. In particular, HCL uses multiple labels to express maps, while JSON and YAML use only single key in maps. Most importantly, HCL allows variables and logic expressions, while JSON and YAML are purely data declarative. Therefore, some features in HCL can never be accurately represented in JSON.

However, in cases there are no variables or logical expressions, but only generic maps, lists, and scalars, then the answer is yes. This type of HCL can be accurately converted to JSON, and vice versa.

There is a practical advantage of HCL over YAML: HCL is very readable and less prone to errors, while YAML is sensitive to markers like white-space. One can write a configuration in HCL and let a program handle conversion.

The Package

determined is a GO package to marshal and unmarshal dynamic JSON and HCL contents with interface types. See article here for JSON usage and here for HCL usage. It has a convert library for conversions among different data formats.

Technically, a JSON or YAML string can be unmarshalled into an anonymous map of map[string]interface{}. For seamless conversion, determined has implemented methods to unmarshal any HCL string into an anonymous map, and marshal an anonymous map into a properly formatted HCL string.

Download determined from:

$ go get github.com/genelet/determined

The following functions in determined/convert can be used for conversion:

• hcl to json: HCLToJSON(raw []byte) ([]byte, error)
• hcl to yaml: HCLToYAML(raw []byte) ([]byte, error)
• json to hcl: JSONToHCL(raw []byte) ([]byte, error)
• json to yaml: JSONToYAML(raw []byte) ([]byte, error)
• yaml to hcl: YAMLToHCL(raw []byte) ([]byte, error)
• yaml to json: YAMLToJSON(raw []byte) ([]byte, error)

If you start with HCL, make sure it contains only primitive data types of maps, lists and scalars.

In HCL, square brackets are lists and curly brackets are maps. Use equal sign = and comma to separate values for list assignment. But no equal sign nor comma for map.

Here is the example to convert HCL to YAML:

package main

import (
    "fmt"
    "github.com/genelet/determined/convert"
)

func main() {
    bs := []byte(`parties = [
  "one",
  "two",
  [
    "three",
    "four"
  ],
  {
    five = "51"
    six = 61
  }
]
roads {
  y = "b"
  z {
    za = "aa"
    zb = 3.14
  }
  x = "a"
  xy = [
    "ab",
    true
  ]
}
name = "marcus"
num = 2
radius = 1
`)
    yml, err := convert.HCLToYAML(bs)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%s\n", yml)
}

Note that HCL is enclosed internally in curly bracket. But the top-level curly bracket should be removed, so it can be accepted by the HCL parser.

Run the program to get YAML:

$ go run x.go
# output in YAML
name: marcus
num: 2
parties:
    - one
    - two
    - - three
      - four
    - five: "51"
      six: 61
radius: 1
roads:
    x: a
    xy:
        - ab
        - true
    "y": b
    z:
        za: aa
        zb: 3.14

The CLI

In directory cmd, there is a CLI program fmtconvert/main.go. Its usage is

# hcl, json and yaml are choices of the formats

$ fmtconvert
# fmtconvert [options] <filename>
  -from string
     from format (default "json")
  -to string
     to format (default "hcl")

This is a HCL:

version = "3.7"
services "db" {
  image = "hashicorpdemoapp/product-api-db:v0.0.22"
  ports = [
    "15432:5432"
  ]
  environment {
    POSTGRES_DB = "products"
    POSTGRES_USER = "postgres"
    POSTGRES_PASSWORD = "password"
  }
}
services "api" {
  environment {
    CONFIG_FILE = "/config/config.json"
  }
  depends_on = [
    "db"
  ]
  image = "hashicorpdemoapp/product-api:v0.0.22"
  ports = [
    "19090:9090"
  ]
  volumes = [
    "./conf.json:/config/config.json"
  ]
}

Convert it to JSON:

$ go run convert.go -from hcl -to json the_above.hcl 
# output in JSON
{"services":{"api":{"depends_on":["db"],"environment":{"CONFIG_FILE":"/config/config.json"},"image":"hashicorpdemoapp/product-api:v0.0.22","ports":["19090:9090"],"volumes":["./conf.json:/config/config.json"]},"db":{"environment":{"POSTGRES_DB":"products","POSTGRES_PASSWORD":"password","POSTGRES_USER":"postgres"},"image":"hashicorpdemoapp/product-api-db:v0.0.22","ports":["15432:5432"]}},"version":"3.7"}

Convert it to YAML:

$ go run convert.go -from hcl -to yaml the_above.hcl
# output in YAML:
services:
    api:
        depends_on:
            - db
        environment:
            CONFIG_FILE: /config/config.json
        image: hashicorpdemoapp/product-api:v0.0.22
        ports:
            - 19090:9090
        volumes:
            - ./conf.json:/config/config.json
    db:
        environment:
            POSTGRES_DB: products
            POSTGRES_PASSWORD: password
            POSTGRES_USER: postgres
        image: hashicorpdemoapp/product-api-db:v0.0.22
        ports:
            - 15432:5432
version: "3.7"

We see that HCL’s syntax is more readable, and less error-prone compared to JSON and YAML.

Summary

HCL is a novel data format that offers advantages over JSON and YAML. In this article, we have demonstrated how to convert data among these three formats.