API Design — Deriving Lift

There have been many attempts to build Swift libraries for both parsing and generating JSON. The reason is probably that there is no obvious Swift solution and JSON parsing is something most developers have to frequently confront. Swift 4 introduced Codable with the promise to solve this once and for all, and for many scenarios, it is just magic. But as soon as your models start to diverge from the simple one-to-one mapping between model and JSON, you have to implement everything yourself using a quite verbose API. Swift also currently lacks APIs for building and parsing JSON on the fly, not going through model objects, which is common when building and parsing network requests. So it seems there is still a need for third-party JSON libraries, explaining why many developers search GitHub or the web for better solutions to the problem.

But a few of us also takes on the challenge to create our own library. As you probably have guessed by now, that is exactly what we did. Soon after the introduction of Swift, we started to build our own version of a Swift JSON library at iZettle. Over the years it has matured significantly to support the different models and micro-services used by our iZettle app. With the recent open-sourcing of Lift, we thought it would be interesting to see how Lift became what it is today.

It is often useful to sketch out new APIs in a playground. Though great for initial prototyping, it is usually not until you apply your ideas and implementation to real production code that you will find many of the problems that have to be addressed. Of course, everyone’s needs are different, and in-house solutions tend to favor your specific challenges. But it is a good idea to plan for an eventual open-sourcing of your library from the get-go as it forces you to think in more general terms. But of course, you will never know for sure if your assumptions were right until you release your library into the wild.

So what makes a good JSON library?

To help out with the design work, we first tried to identify a list of requirements of what makes a good JSON library. Primarily we looked at our internal requirements, but we also looked at some of the existing JSON libraries to see how they approached the problem. In this blog we will, step by step, go through how we came up with the current design of the Lift library.

Intuitive and lightweight syntax

Ideally we would like to generate and parse JSON with a syntax like this:

var json = ["number": 5]
json["number"] = 4711
let val = json["number"] // val is inferred to 'Int?'

This is valid Swift code, but as JSON can hold a mix of heterogenous types such as arrays, dictionaries, numbers and strings, we run into problems when we start to mix types:

var json: [String: Any] = ["number": 5, "string": "hello"]
json["string"] = "hello world"
let val = json["string"] // val is inferred to 'Any?'

Static typing

Being used to Swift’s strong typing, working with the Any type is just not good enough as this forces us to use casting to get back to our wanted types:

let val = json["string"] as? String // val is inferred to 'String?'

But often the compiler already knows what type to use. This is typically the case when deserializing into model types:

struct Person { 
  var name: String?
  var age: Int?
    
  init(json: [String: Any]) {
    name = json["name"] as? String
    age = json["age"] as? Int
  }
}

Here we can see that we have to repeat the types String? and Int?

To solve this, the first revision of our JSON library, wrapped the JSON dictionary in a struct and added subscripts overloads for different types:

struct Jar {
  var dict: [String: Any]
    
  subscript(key: String) -> String? { 
    return dict[key].flatMap { $0 as? String }
  }
    
  subscript(key: String) -> Int? { 
    return dict[key].flatMap { $0 as? Int }
  }
}

The type signature of the different subscripts differs only on the return type. In comparison to many other languages, Swift allows overloading of methods not only based on their arguments but also their return types. This together with Swift’s powerful type inference, the compiler will now pick the correct subscript version based on the receiving type. This removes the need to repeat the type information through casting:

init(jar: Jar) {
  name = jar["name"]
  age = jar["age"]
}

Now we are back to the ideal syntax. But we are restricted to work only with optionals.

Handling of non-optionals

Very seldom do we have models that consist of only optionals. We really do not want our JSON library to enforce optionality. One way forward is to force unwrap the result:

struct Person { 
  var name: String // No longer optional
  var age: Int?
    
  init(jar: Jar) {
    name = jar["name"]! // force unwrap will halt the app if failing
    age = jar["age"]
  }
}

But if the JSON does not contain the key “name” or if the value of “name” is not a String, the app will halt. This might be ok if you can guarantee the format of your JSON. But that is seldom the case.

So now you start to see the guard/if let dance:

init?(jar: Jar) { // init is now optional
  guard let name = jar["name"] else { return nil }
  self.name = name
  age = jar["age"]
}

This requires returning an optional Person. Unfortunately, we will not know what caused the error if nil is returned. Was the value absent, or of the wrong type? What we really would like is some proper error handling.

Error handling

Swift has a great model for error handling, so why not use it? An improvement over using optional initializers is to let the initializer throw instead:

init(jar: Jar) throws { // init is now throwing
  guard let name = jar["name"] else { 
    throw Error("The key 'name' is missing or it's value is not a String") 
  }
  self.name = name
  age = jar["age"]
}

A more convenient way could be to extend Optional with an unwrap helper method that throws if the value is missing:

extension Optional {
  func unwrap(description: String) throws -> Wrapped {
    switch self {
      case let val?: return val
      case nil: throw UnwrapError(description)
    }
  }
}

This allows rewriting the string extraction as:

init(jar: Jar) throws {
  name = try jar["name"].unwrap(...) 
  age = jar["age"]
}

This improves the error handling but the error lacks details of what exactly went wrong. Also, some errors are silently ignored:

init(jar: Jar) throws {
  name = try jar["name"].unwrap(...)  // Was the key "name" missing or the value not a String?
  age = jar["age"] // If the value of "age" is not an Int, this should cause an error as well.
}

If Swift had support for generic(1) and throwing subscripts we might try something like:

subscript<T>(key: String) throws -> T? {
  return try dict[key].map { val in
    try (val as? T).unwrap(...)
  }
}

subscript<T>(key: String) throws -> T {
  let val = try dict[key].unwrap(...)
  return try (val as? T).unwrap(...)
}

Unfortunately we have to wait for a future version of Swift for that, and instead, work with what we have today. But as it happens, operators can both throw and be generic:

postfix func ^<T>(value: Any?) throws -> T? {
  return try value.map {
    try ($0 as? T).unwrap(...)
  }
}

postfix func ^<T>(value: Any?) throws -> T {
  let val = try value.unwrap(...)
  return try (val? T).unwrap(...)
}

In Lift this is called the “lift operator” as it is used to lift values out of Jar containers(2).

We have now derived the syntax used by the Lift library:

init(jar: Jar) throws {
  name = try jar["name"]^
  age = try jar["age"]^
}

Extensible

Unfortunately, the above implementation of the lift operator will only allow us to work with the limited set of known JSON types. If we like to work other types than JSON’s string, number, bool, null, array, and dictionaries, we are out of luck. What we need is a way to go from JSON values to other types as well. This is a perfect place to introduce a new protocol:

protocol JarConvertible {
  init(jar: Jar) throws
}

Similarly, we need to be able to generate JSON from these other types as well:

protocol JarRepresentable {
  var jar: Jar { get }
}

And for many types we want to implement both protocols:

typealias JarElement = JarRepresentable & JarConvertible

Applying this to Int could look like:

extension Int: JarElement {
  init(jar: Jar) throws {
    self = try (jar.asAny() as? Int).unwrap("Object '\(value)' is not convertible to Int")
  }
    
  var jar: Jar { return Jar(unchecked: self) }
}

Custom type handling

This is all and well for common types already known to Lift. But the use of a protocol will also open up Lift for our custom types (types not known to Lift). This works for both simple types such as:

struct Money {
  var fractionized: Int
}

Where we now can reuse the already added support for Int:

extension Money: JarElement {
  init(jar: Jar) throws {
    fractionized = try jar^
  }
    
  var jar: Jar { return Jar(fractionized) }
}

let jar: Jar = ["amount": Money(fractionized: 2000)]
let amount: Money = try jar["amount"]^

As well as for more complex and record like types:

struct Payment {
  var amount: Money
  var date: Date
}

extension Payment: JarElement {
  init(jar: Jar) throws {
    amount = try jar["amount"]^
    date = try jar["date"]^
  }
    
  var jar: Jar { 
    return ["amount": amount, "date": date] 
  }
}

let jar: Jar = ["payment": Payment(...)]
let payment: Payment = try jar["payment"]^

We have now come to a point where the API is mature enough to work both with built-in types as well as our custom types. Due to limits of the current version of Swift, we did not reach the ideal syntax that we aimed for, but we came fairly close and gained a lot of other benefits along the way.

Summary

We started with an ideal syntax to work with JSON like data. Being a good Swift citizen we soon added requirements such as static typing, error handling and support to extend our custom types. This led us towards the syntax and protocols being used by the Lift library today.

Of course, there is a lot more to a fully featured JSON library than what was covered in this article. If you want to read more about Lift and try it out yourself you can find it at GitHub. And as Lift is open source you are more than welcome to look at its implementation, and why not contribute to it as well?

1. Swift 4.0 added support for generic subscripts, but they still can not throw.
2. In Lift the lift operator will work on Jar containers instead of Any values and the Jar subscripts will wrap returned values in new Jar containers.