JSON decoding in Elm is still difficult

Pretext

These are a series of posts talking about solving some of the problems with Elm that I see a lot in the community, and some discussion points around those problems. If you don’t come away thinking about a new way to solve a problem, then my post has failed you!

In this post, we’ll talk about JSON. We’ll talk about how to decode JSON, some of the problems with the current approach, and take an examination of an alternative approach that I think is interesting. Note, to me, interesting isn’t always good! It’s just different.

Decoding

The state of JSON decoding/encoding has been a bit of a problem for a while. Let’s talk about that. If we go to the docs page, we’ll see that the Json decoder examples look nice and simple, right? No big deal. Here’s one for a point below.

Unfortunately, that’s not what decoders look like in production. Here’s a good example of one:

It’s a lot of boilerplate.

Wanna know a secret? All of that can be generated for you. In fact, I actually generated that example from this json using json-to-elm.

{ "name" : "hello"
, "age": 5
, "location" : {
    "name" : "London",
    "zip": "sa18"
  }
, "friends" : [ "flip" ]
, "languages" : ["language"]
, "national": "country"
, "isOnline" : false
}

Boilerplate’s not a big deal, right? We have elm-decode-pipeline to deal with that! Well, what if I introduce a subtle bug?

type alias Person =
    { name : String
    , location : String
    }
decodePerson : Json.Decode.Decoder Person
decodePerson =
    Json.Decode.map2 Person
        (field "location" Json.Decode.string)
        (field "name" Json.Decode.string)

Can you spot it? What happens if I give it the json blob below

{ "name": "hello"
, "location": "somewhere"
}

Will this decode correctly? The answer is yes, it will decode. It won’t decode correctly, however. It will create the record

{ name = "somewhere", location = "hello" }

That’s because of how decoders work. When you create a solid type alias in Elm, a constructor will be generated of the same name. In the above example, Person is a function with the type signature Person : String -> String -> Person . The way decoding works is that each field is partially applied to the function, before passing the new partial onwards. This means that for Person , it first gets given a string which it uses as a name, then a second that is used as the location. So, if your decode fields match the type, but not the order of the fields in the records, then you can easily get a runtime logical error that is not your fault, and you may not even catch it. elm-test fuzzers might not even catch it properly.

Our alternative

Okay, so in terms of implementing Json without depending on order
Let’s start by defining a simple model:

type alias Model =
 { name : String
 , age : Int 
 }

Nothing fancy really. Now let’s describe how we’d like our decoders to look:

decodeModel : Json.Value -> Result String Model
decodeModel value =
  Ok { name = "", age = 0 } 
    |> decodeField "age" Json.int setAge value
    |> decodeField "name" Json.string setName value

Notice how I’ve gotten rid of the confusing Decoder abstraction for now, we’re just taking about things that take a Json.Valueand return a Result. This is because so many people come to Elm, and get confused right away by Decoder, since it’s an opaque type. Also notice that name and age are out of order! Also notice that I use something called setAge and setName.These are setters, which just take the model and set the attribute. Here’s an example implementation:

setName : Model -> String -> Model
setName model name = { model | name = name }

They’re mostly boilerplate.

We use this decoder like this:

exampleObject = 
  E.object 
    [ ("name", E.string "noah")
    , ("age", E.int 10) 
    ]
main = 
  exampleObject
    |> decodeModel
    |> toString
    |> Html.text

The first line is just encoding a JSON object for an example. Then we just call the decoder, since it’s a a function expecting a Json.Value. No more need for decodeValue or decodeString. decodeModel will return a result of Ok { name = “noah”, age = 10 }.

decodeField is a scary looking thing which is actually pretty simple:

decodeField :
    String
    -> Json.Decoder a
    -> (model -> a -> model)
    -> Json.Value
    -> Result String model
    -> Result String model
decodeField fieldName decoder setter value model =
    let 
        decoded = 
            Json.decodeValue (Json.field fieldName decoder) value
    in 
    case model of
        Err s ->
            case decoded of
                Err newMessage ->
                    Err (s ++ "\nAnd " ++ newMessage)
                Ok _ ->
                    model
         Ok v ->
            Result.map (setter v) decoded

There are various ways of cleaning this up some more, but this is how you do out-of-order decoding in Elm. Notice that we gather up the messages throughout the chain — meaning if your decoding fails at some point, then you will get the full stack trace! Pretty handy. Here’s an example:

Expecting an object with a field named `name` but instead got: null
And Expecting an object with a field named `age` but instead got: null
And Expecting an object with a field named `location` but instead got: null

Conclusion

Decoding in Elm is still not a solved problem, with a lot of gotcha. If you’ve used Elm for a while, then you’ve become familiar with the sensible defaults and approaches it takes to things. It would be great if Json decoding was a little easier to follow.

It could be further simplified if Elm provided a couple of things. Namely, things that were taken out of the type system that didn’t need to be (e.g adding fields to a record — then we wouldn’t need to provide a default). With this change, you can make refactoring changes to your model without worrying about the order. If you have the right tooling, then generating setters/getters is trivial, but it’s still a bit of legwork. It would also be great if setters could be generated as getters currently are. There’s some other places where this is a pain point, but that’s for another post.

I’ve published this as a package called elm-alternative-json. I’d love to discuss this on Slack with anyone interested, or on Github, or twitter.