Serializing to JSON in SML/NJ

Say you’re hoping to work with some data structures in Standard ML, and you find yourself wanting to serialize some data to a JSON file. Luckily for you, the SML/NJ compiler ships with a snazzy JSON library! In this post, we’ll explore what it’s capable of and build our own simple serialization utility.

Excerpt of serialization code

Importing the Library

The library itself can be loaded using the json-lib.cm file under the $json-lib.cm anchor — using a slick shorthand, just $/json-lib.cm . To import, run smlnj -m $/json-lib.cm , or add $/json-lib.cm to your CM file.

First, let’s take a look at the actual JSON structure:

structure JSON =
  struct
    datatype value
      = OBJECT of (string * value) list
      | ARRAY of value list
      | NULL
      | BOOL of bool
      | INT of IntInf.int
      | FLOAT of real
      | STRING of string
  end

Not particularly interesting — just a single datatype! Most of the constructors should be pretty intuitive; the two worth noting are OBJECT , which is an ordered list of key-value pairs (not a dictionary), and INT , which stores an infinite-precision integer (not just a regular 31-bit int ).

Working with the datatype on its own is straightforward, albeit tedious. For example:

local
  val rec aux = fn key => fn
    nil           => NONE
  | (k,v) :: rest => if k = key then SOME v else aux key rest
in
  val findField = fn key => fn
    JSON.OBJECT l => aux key l
  | _ => raise Fail "Unexpected type"
end

Luckily, many similar utilities come with the included JSONUtil structure. Some examples:

(* extract `b` from `BOOL b` *)
exception NotBool of value
val asBool : value -> bool
(* extract `i` from `INT i`, in 31-bit `int` or `IntInf.int` form *)
exception NotInt of value
val asInt : value -> int
val asIntInf : value -> IntInf.int
(* look up an object field *)
exception FieldNotFound of value * string
val findField : value -> string -> value option
(* map an `ARRAY` into a list *)
exception NotArray
val arrayMap : (value -> 'a) -> value -> 'a list
(* work with nested key/index paths *)
datatype edge
  = SEL of string  (* select by a key *)
  | SUB of int     (* subscript by an index *)
type path = JSONUtil.edge list
val get : value * path -> value
val replace : value * path * value -> value
val insert : value * path * string * value -> value
val append : value * path * value list -> value

This solves much of our unwieldiness woes in just a few characters. For example, here’s how we could turn a list of JSON integers into an actual int list:

> smlnj -m $/json-lib.cm
λ> structure J = JSONUtil;
λ> val arr = JSON.ARRAY [JSON.INT 1, JSON.INT 5, JSON.INT 0];
val arr = ARRAY [INT 1,INT 5,INT 0] : value
λ> J.arrayMap J.asInt arr;
val it = [1,5,0] : int list

And here’s indexing into a slightly messier OBJECT :

λ> val obj = JSON.OBJECT [("foo",JSON.INT 3),("bar", arr)];
val obj = OBJECT [("foo",INT 3),("bar",ARRAY [INT 1,INT 5,INT 0])] : JSON.value
λ> J.get (obj,[J.SEL "bar",J.SUB 1]);
val it = INT 5 : JSON.value

Parsing and Printing

Interacting with JSON files is key (pun intended) — luckily, the JSONParser and JSONPrinter structures have us covered.

Parsing a file is easy; just use JSONParser.parseFile :

(* foo.json *)
{
  "yes": 3,
  "no": [4, 5.0, null]
}
λ> JSONParser.parseFile "foo.json";
val it = OBJECT [("yes",INT 3),("no",ARRAY [INT 4,FLOAT 5.0,NULL])]
  : JSON.value

Printing is a bit trickier, since we have to deal with the file ourselves, but still not bad:

λ> JSONPrinter.print (TextIO.stdOut, obj);
{"foo":3,"bar":[1,5,0]}
val it = () : unit
λ> let
.    val file = TextIO.openOut "bar.json"
.  in
.    JSONPrinter.print (file, obj) before TextIO.closeOut file
.  end;
val it = () : unit
(* bar.json *)
{"foo":3,"bar":[1,5,0]}

Serialization

While JSON is nice for storage, we lose a lot of the type restrictiveness when encoding our data, forcing us to “deal with” lots of extra ultimately-unused cases in our code. As we’ve seen, JSONUtil can help us with the conversion process from JSON to more restrictive types. Now, let’s consider how we could use the notion of a typeclass to describe what a “serializable” type looks like.

A typeclass is a signature with a single type (typically named t) and some operations on the type, all under the assumption that the type is known to the client. (It lets us describe a class of types which share a given property.)

We can write a fairly obvious signature to describe what it means to be “serializable”:

signature SERIALIZABLE =
  sig
    type t
    val toJSON   : t -> JSON.value
    val fromJSON : JSON.value -> t
    (* invariant: fromJSON o toJSON ≅ Fn.id *)
  end

Writing instances of the typeclass isn’t bad, either:

structure SerializeInt :> SERIALIZABLE where type t = int =
  struct
    type t = int
    val toJSON = JSON.INT o IntInf.fromInt
    val fromJSON = JSONUtil.asInt
  end
functor SerializePair (
  structure A : SERIALIZABLE
  structure B : SERIALIZABLE
) :> SERIALIZABLE where type t = A.t * B.t =
  struct
    type t = A.t * B.t
    val toJSON = fn (a,b) => JSON.ARRAY [A.toJSON a, B.toJSON b]
    val fromJSON = fn JSON.ARRAY [a,b] => (A.fromJSON a, B.fromJSON b)
    | _ => raise Fail "invalid format"
  end

Although it’s a bit trickier, we can come up with our own encodings of inductive types, such as trees:

structure Tree =
  struct
    datatype tree = Empty | Node of tree * string * tree
  end
structure SerializeTree :> SERIALIZABLE where type t = Tree.tree =
  struct
    type t = Tree.tree
    val rec toJSON = fn
      Tree.Empty        => JSON.NULL  (* easy enough choice here *)
    | Tree.Node (l,x,r) => JSON.ARRAY [
        toJSON l,
        JSON.STRING x,
        toJSON r
      ]
    val rec fromJSON = fn
      JSON.NULL => Tree.Empty
    | JSON.ARRAY [
        l,
        JSON.STRING x,
        r
      ] => Tree.Node (fromJSON l,x,fromJSON r)
    | _ => raise Fail "invalid format"
  end

Notice: The toJSON and fromJSON functions mirror each other structurally! After deciding on the (injective) encoding scheme in toJSON, writing fromJSON is a breeze.

Using the parsing and printing logic shown earlier, we can easily transform a SERIALIZABLE instance into a structure for saving and loading data:

functor Serialize (S : SERIALIZABLE) =
  struct
    val load = S.fromJSON o JSONParser.parseFile
    val save = fn filename => fn data => (
      let
        val file = TextIO.openOut filename
        val json = S.toJSON data
      in
        JSONPrinter.print (file, json) before TextIO.closeOut file
      end
    )
  end

Here’s a trace of the full serialization process on a sample tree:

Example serialized tree

Summary

We used the $/json-lib.cm library from SML/NJ to manipulate, load, and save JSON data. Using the utilities, we were able to implement a simple serialization technique.

There are some obvious drawbacks to this style of serialization; for example, serializing values involving functions or exns doesn’t sound promising. That said, using JSON is a great way to store inductive data in a human-readable, language-independent format.