Implicit resolution in Scala: an example with spray-json

Stijn Vermeeren
Nov 27, 2017 · 8 min read

Introducing spray-json

At Starmind, we use the spray-json library to parse and generate JSON. Defining how JSON is generated for our classes can look like this:

import spray.json._

case class Person(firstName: String, lastName: String)

implicit val personJsonWriter = new JsonWriter[Person] {
def write(person: Person): JsValue = {
JsObject(
"first_name" -> JsString(person.firstName),
"last_name" -> JsString(person.lastName)
)
}
}

Person(firstName = "David", lastName = "Bowie").toJson

Run this code on Scastie.

The JsonWriter is used as an implicit parameter for the .toJson method. This produces a JsValue object (from the DSL that spray-json uses to represent JSON in Scala) on which we can for example call .prettyPrint to turn it into a familiar JSON string:

{
"first_name": "David",
"last_name": "Bowie"
}

There are two more things that we need to understand about spray-json for this article.

First, besides the JsonWriter[T] trait with a write method to convert something of type T to a JsValue, there is also a JsonReader[T] trait with a read method to do the inverse conversion (from a JsValue to type T). When we want to support both reading and writing, we can implement a JsonFormat[T] which is simply defined as

trait JsonFormat[T] extends JsonReader[T] with JsonWriter[T]

Second, there are traits RootJsonWriter, RootJsonReader and RootJsonFormat, which extend from JsonWriter, JsonReader and JsonFormat respectively, without adding any additional functionality. The spray-json documentation explains:

According to the JSON specification not all of the defined JSON value types are allowed at the root level of a JSON document. A JSON string for example (like "foo") does not constitute a legal JSON document by itself. Only JSON objects or JSON arrays are allowed as JSON document roots.

In order to distinguish, on the type-level, “regular” JsonFormats from the ones producing root-level JSON objects or arrays spray-json defines the RootJsonFormat type, which is nothing but a marker specialization of JsonFormat. Libraries supporting spray-json as a means of document serialization might choose to depend on a RootJsonFormat[T] for a custom type T (rather than a "plain" JsonFormat[T]), so as to not allow the rendering of illegal document roots.

In fact, what is or isn’t valid top-level JSON doesn’t seem to be so simple, with different specifications using different definitions. However, that’s a completely different story…

The missing JsonWriters

Now spray.json.DefaultJsonProtocol (available both as a trait and as an object) contains a number of predefined writers, readers and formats, as well as implicit methods that can e.g. automatically provide a JsonFormat[List[T]] for any type T which has an implicit JsonFormat[T]. (Haoyi’s Programming Blog has a good explanation on how such derived implicits work.) This enables us to do something like this:

import spray.json._
import spray.json.DefaultJsonProtocol._

case class Person(firstName: String, lastName: String)

implicit val personJsonFormat = new RootJsonFormat[Person] {
def write(person: Person): JsValue = {
JsObject(
"first_name" -> person.firstName.toJson,
"last_name" -> person.lastName.toJson
)
}

def read(value: JsValue): Person = ???
}

val davidBowie = Person(firstName = "David", lastName = "Bowie")
val iggyPop = Person(firstName = "Iggy", lastName = "Pop")
List(davidBowie, iggyPop).toJson

Run this code on Scastie.

The .toJson method looks for an implicit JsonWriter[List[Person]], which is provided by the method

implicit def listFormat[T :JsonFormat]: RootJsonFormat[List[T]]

from the DefaultJsonProtocol with type parameter T set to Person (and the :JsonFormat context bound is satisfied by our own implicit JsonFormat[Person]). In other words, writing more explicitly

List(davidBowie, iggyPop).toJson(listFormat(personJsonFormat))

is equivalent to List(davidBowie, iggyPop).toJson.

I used a JsonFormat here instead of a JsonWriter. If we are only interested in writing, it would be cleaner to only implement a JsonWriter, instead of leaving the read method unimplemented, or having it throw an exception. Annoyingly, spray-json seems to encourage exactly the latter hack, by proving a method

def lift[T](writer :JsonWriter[T]) = new JsonFormat[T] {
def write(obj: T): JsValue = writer.write(obj)
def read(value: JsValue) =
throw new UnsupportedOperationException("JsonReader implementation missing")
}

in the DefaultJsonProtocol, while neglecting to provide any implicit JsonWriters for lists or other collections.

Thankfully, this omission in the spray-json library provides us with the chance to learn some important lessons about implicit resolution, by trying to implement the missing JsonWriters ourselves…

Extending the DefaultJsonProtocol

We can basically just copy-paste a few lines from the spray-json source code to create implicit JsonWriters for lists.

import spray.json._

object ExtendedJsonProtocol extends DefaultJsonProtocol {
implicit def listJsonWriter[T : JsonWriter]: RootJsonWriter[List[T]] = new RootJsonWriter[List[T]] {
def write(list: List[T]): JsArray = JsArray(list.map(_.toJson).toVector)
}
}
import ExtendedJsonProtocol._

case class Person(firstName: String, lastName: String)

implicit val personJsonWriter = new RootJsonWriter[Person] {
def write(person: Person): JsValue = {
JsObject(
"first_name" -> person.firstName.toJson,
"last_name" -> person.lastName.toJson
)
}
}

val davidBowie = Person(firstName = "David", lastName = "Bowie")
val iggyPop = Person(firstName = "Iggy", lastName = "Pop")
List(davidBowie, iggyPop).toJson

Run this code on Scastie.

Easy.

But what if we, after having extended the default JSON protocol like this, decide that we want to support both reading and writing for our Person case class? We can replace our implicit JsonWriter[Person] with an implicitJsonFormat[Person]. Let’s try with the format that spray-json can automatically generate for case classes:

import spray.json._

object ExtendedJsonProtocol extends DefaultJsonProtocol {
implicit def listJsonWriter[T : JsonWriter]: RootJsonWriter[List[T]] = new RootJsonWriter[List[T]] {
def write(list: List[T]): JsArray = JsArray(list.map(_.toJson).toVector)
}
}
import ExtendedJsonProtocol._

case class Person(firstName: String, lastName: String)

implicit val personJsonFormat = jsonFormat2(Person)

val davidBowie = Person(firstName = "David", lastName = "Bowie")
val iggyPop = Person(firstName = "Iggy", lastName = "Pop")
List(davidBowie, iggyPop).toJson

Run this code on Scastie.

Suddenly the compiler will not accept the code anymore…

Error:(17, 46) ambiguous implicit values:
both method listFormat in trait CollectionFormats of type [T](implicit evidence$1: spray.json.JsonFormat[T])spray.json.RootJsonFormat[List[T]]{def write(list: List[T]): spray.json.JsArray}
and method listJsonWriter in object ExtendedJsonProtocol of type [T](implicit evidence$1: spray.json.JsonWriter[T])spray.json.RootJsonWriter[List[T]]
match expected type spray.json.JsonWriter[List[A$A10.this.Person]]
lazy val result = List(davidBowie, iggyPop).toJson
^

From the “ambiguous implicit values” error message, we can understand that the compiler has two ways of implicitly generating a JsonWriter[List]: using the DefaultJsonProtocol to create a RootJsonFormat[List], i.e.

List(davidBowie, iggyPop).toJson(listFormat(personJsonFormat))

or using our listJsonWriter method to convert the JsonFormat[Person] into a RootJsonWriter[List] :

List(davidBowie, iggyPop).toJson(listJsonWriter(personJsonFormat))
When we start from a RootJsonWriter[T], we only have one implicit conversion that can take us to a JsonWriter[List[T]]. However, when we start with a RootJsonFormat, we now have two possible implicit conversions on the way to a JsonWriter[List[T]]…

Scala would be pretty crippled if it always threw an error whenever more than one implicit conversion is applicable. Of course Scala has some mechanism to rank the available implicit conversions and pick the most appropriate one. But how does this work exactly and why did it fail in the example above? Time to dive into dreaded Scala Language Specification!

Understanding implicit resolution

The Scala Language Specification has a chapter which is dedicated to implicits, so let’s start looking there. In section “7.2 Implicit parameters”, we find a definition of implicit scope (which could be subject of several essays by itself; the Scala FAQ is a good starting point) as well as the sentence:

If there are several eligible arguments which match the implicit parameter’s type, a most specific one will be chosen using the rules of static overloading resolution.

This takes us to section “6.26.3 Overloading resolution”, which is quite daunting to read. However, if we strip away all the technical details, the main ideas are not that complex. Scala looks for the “most specific” alternative. Here, being “more specific” is a partial relation which takes two different aspects into account. To decide whether an alternative X is more specific than an alternative Y, we can imagine a competition in two rounds (one round for each of the two aspects), with the final result being the sum of the scores of the individual rounds.

  • Round 1: Where is the alternative defined?
    X scores one point over Y when X is defined in an class or object that is derived from (for example extends from) the class or object where Y is defined (and vice versa).
  • Round 2: What is the type of the alternative?
    X scores one point over Y when the type of X conforms to (i.e. is a subtype of) the type of Y (and vice versa).
    (It is not obvious to extract this from the very technical definition given in the Scala Language Specification, which confusingly also uses the words “as specific as”. However, most of the technical details are concerned with function application and do not affect implicit resolution.)

If there is a winner after adding up the scores of the two rounds, then this alternative is the more specific one.

However, the score can also be a draw, which causes the “ambiguous implicit values” error that we encountered before:

  • The score is a draw in both rounds. For example, both alternatives have the same type and are defined in the same class/object (or in unrelated classes/objects).
  • One alternative wins one round and the other alternative wins the other round. For example, alternative X is defined in a subclass of where alternative Y is defined, but at the same time the type of Y is a subtype of the type of X.

The second situation occurred in our example. Our own listJsonWriter is defined in an object that extends from the DefaultJsonProtocol where spray-json’s listFormat is defined. So round 1 is a 1–0 victory for our listJsonWriter. However, JsonFormat[List[T]], which is the type returned by the listFormat method, is a subtype of JsonWriter[List[T]], the type returned by our listJsonWriter method. So round 2 is a 0–1 victory for spray-json’s listFormat. Overall it is a draw, so that’s why the Scala compiler throws an “ambiguous implicit values” error.

Finding the correct implicits

With this knowledge about how implicit resolution works, the solution to our “ambigious implicit values” problem is not hard to find. Since we can’t do much about the type of the listJsonWriter method, we must make this method less specific than spray-json’s listFormat by changing where it is defined. If our ExtendedJsonProtocol no longer extends from the DefaultJsonProtocol, then we should be fine. One way to achieve this in our worksheet is to change the extends DefaultJsonProtocol into an import DefaultJsonProtocol._.

import spray.json._
import DefaultJsonProtocol._

object ExtendedJsonProtocol {
implicit def listJsonWriter[T : JsonWriter]: RootJsonWriter[List[T]] = new RootJsonWriter[List[T]] {
def write(list: List[T]): JsArray = JsArray(list.map(_.toJson).toVector)
}
}
import ExtendedJsonProtocol._

case class Person(firstName: String, lastName: String)

implicit val personJsonFormat = jsonFormat2(Person)

val davidBowie = Person(firstName = "David", lastName = "Bowie")
val iggyPop = Person(firstName = "Iggy", lastName = "Pop")
List(davidBowie, iggyPop).toJson

Run this code on Scastie.

Et voilà, it works as expected again! Our ExtendedJsonProtocol no longer gets in the way of the DefaultJsonProtocol.

Note that in the above example, the ExtendedJsonProtocol is not used at all. Only if a type T has a JsonWriter but no JsonFormat, then our implicitlistJsonWriter method will be used when a JsonWriter[List[T]] is needed. Whenever a JsonFormat is available, the more specific listFormat method from spray-json will be used instead.


I hope you found this article useful. I’d be glad to hear any feedback. I hope to publish more articles like this on advanced Scala programming in the future.

Stijn Vermeeren

Written by

Mathematician, Data Scientist and Scala Software Engineer at Starmind in Zürich, Switzerland. Polyglot, Hiker and Mountaineer, 2-kyu Go Player.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade