JSON Serialization in Ktor
Introduction
As part of my series “An opinionated Kotlin backend service”, I was checking out different ways to serialize/deserialize JSON payloads.
According to https://ktor.io/docs/serialization.html these are the supported converters:
I tested #1 and #3 and because I like challenges from time to time I threw Moshi into the mix. I left out Jackson because… well why not ;-).
My test was simple. I wanted to serialize/deserialize this data structure(s):
data class Customer(
var customerUUID: String,
var createdAt: Instant,
var modified: Instant,
var firstName: String,
var lastName: String,
var language: Language,
var account: Account,
)data class Account(
var accountUUID: String,
var createdAt: Instant,
var modified: Instant,
var status: AccountStatus
)
The serialization part retrieved a list of customers (ArrayList<Customer>), each one referencing an account -> GET.
The deserialization part created an account first then a customer referencing that account -> POST.
Intermezzo
Custom converters can be registered with Ktor easily. They are just another Ktor Feature like routing, logging or error handling:
(in other frameworks a feature would be called middleware, filter or delegate handler)
A feature can be installed using the install function. For our purposes we install a ContentNegotiation feature like so:
install(ContentNegotiation) {
register(ContentType.Application.Json, CustomJsonConverter())
register(ContentType.Application.Xml, CustomXmlConverter())
}All three tested converters have extension functions to simplify the registration:
install(ContentNegotiation) {
gson() // Gson converter
json() // kotlinx.serialization converter
moshi { } // Moshi converter
}Of course you only install one of them at a time.
Gson
Dependencies
Add this dependency in your Gradle build file:
implementation(“io.ktor:ktor-gson:$ktor_version”)Installation
As mentioned above this is all you need to install the GSON converter:
install(ContentNegotiation) {
gson()
}The gson function gives you access to the GsonBuilder to customize the converter:
install(ContentNegotiation) {
gson {
setPrettyPrinting()
disableHtmlEscaping()
registerTypeAdapter(yourClass, yourCustomAdapter)
}
}Customization
Running my tests out of the box. I got this for the customer/account data structure:
[
{
"customerUUID": "bbbe4f11-c36d-49cd-a031-787d6559d3ea",
"createdAt": {
"seconds": 1617802016,
"nanos": 609375000
},
"modified": {
"seconds": 1617802016,
"nanos": 609375000
},
"firstName": "Emanuel",
"lastName": "Moecklin",
"language": "en",
"account": {
"accountUUID": "20836570-d4ef-420e-b500-7b7f6516e1a8",
"createdAt": {
"seconds": 1617802016,
"nanos": 415274000
},
"modified": {
"seconds": 1617903421,
"nanos": 798094000
},
"status": "Created"
}
},As you can see the Instant object was serialized but in a rather unconventional way. To remedy that I added a custom adapter:
object GsonInstantAdapter: TypeAdapter<Instant>() {
override fun write(writer: JsonWriter, value: Instant?) {
runCatching {
writer.value(value.toString())
}.onFailure {
throw IllegalArgumentException(it.message)
}
}
override fun read(reader: JsonReader): Instant? {
return runCatching {
Instant.parse(reader.nextString())
}.throwOnFailure()
}
}Please note that I used https://github.com/michaelbull/kotlin-result to handle the results of the serialization/deserialization (runCatching, onFailure).
throwOnFailure is an extension function I use in combination with Valiktor (see my article about object validation: https://medium.com/nerd-for-tech/object-validation-in-kotlin-c7e02b5dabc). The following code leaves out the Valiktor part, the full code will be in my main article “An opinionated Kotlin backend service”.
fun <V> Result<V, Throwable>.throwOnFailure(): V? {
onFailure {
val message = it.message ?: it.javaClass.simpleName
throw IllegalArgumentException(message)
}
return component1()
}Now we need to register the adapter with the GsonBuilder:
gson {
setPrettyPrinting()
disableHtmlEscaping()
registerTypeAdapter(Instant::class.java, GsonInstantAdapter)
}The result:
"createdAt": "2021-04-09T19:55:13.571019Z",
"modified": "2021-04-09T20:22:58.167671Z",kotlinx.serialization
Dependencies
Kotlin’s own serialization plugin needs to be configured like so:
plugins {
kotlin("plugin.serialization") version "1.4.32"
}It also needs this dependency:
implementation("io.ktor:ktor-serialization:$ktor_version")Installation
install(ContentNegotiation) {
json(Json {
prettyPrint = true
isLenient = true
})
}Customization
Running this will throw an exception:
kotlinx.serialization.SerializationException: Serializer for class 'Customer' is not found. Mark the class as @Serializable or provide the serializer explicitly.
So we add a @Serializable annotation to each data class:
@Serializable
data class Account(@Serializable
data class Customer(
Now we get a compile error:
Serializer has not been found for type ‘Instant?’. To use context serializer as fallback, explicitly annotate type or property with @Contextual
To remedy this I wrote a serializer for the Instant class (analogous Gson):
@OptIn(ExperimentalSerializationApi::class)
@Serializer(forClass = Instant::class)
object InstantSerializer : KSerializer<Instant> {
override fun deserialize(decoder: Decoder): Instant
= Instant.parse(decoder.decodeString())
override fun serialize(encoder: Encoder, value: Instant) {
encoder.encodeString(value.toString())
}
}To register the serializer each property needs to be annotated like this:
@Serializable(with = InstantSerializer::class)
var createdAt: Instant?,@Serializable(with = InstantSerializer::class)
var modifiedAt: Instant?,
Alternatively you can add this at the top of the file (more convenient if you have multiple properties):
@file:UseSerializers(InstantSerializer::class)The second option can be used to register multiple adapters:
@file:UseSerializers(InstantSerializer::class, AnotherSerializer::class)Moshi
Moshi isn’t officially supported by Ktor but Jackson is, so why not Jackson but Moshi? The answer is simple: I wanted a library that is modern with a concise and fluent API and supporting Kotlin explicitly. Both libraries are foremost Java libraries but Moshi certainly has the better Kotlin support.
Jackson has been known as “the Java JSON library” or “the best JSON parser for Java”. Or simply as “JSON for Java”.
(https://github.com/FasterXML/jackson)
Moshi is a great JSON library for Kotlin. It understands Kotlin’s non-nullable types and default parameter values. When you use Kotlin with Moshi you may use reflection, codegen, or both.
(https://github.com/square/moshi)
Dependencies
Setting up Moshi wasn’t as straight forward as the other two libraries. TBH it as quite an ordeal to make this work so let me walk you through the process.
First you add the standard dependencies:
implementation("com.squareup.moshi:moshi:1.12.0")
implementation("com.squareup.moshi:moshi-kotlin:1.12.0")Then you need to install Moshi as a Ktor feature. I found this library that bridges the gap between Moshi and Ktor: https://github.com/rharter/ktor-moshi. Unfortunately after adding it I got an exception:
Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
After some research I fixed it by running some of the conversion code off the calling thread:
After adding this Moshi can be installed as Ktor feature:
install(ContentNegotiation) {
moshi { }
}When running this and trying to serialize I got:
Cannot serialize Kotlin type Customer. Reflective serialization of Kotlin classes without using kotlin-reflect has undefined and unexpected behavior. Please use KotlinJsonAdapterFactory from the moshi-kotlin artifact or use code gen from the moshi-kotlin-codegen artifact.
After some digging I found that there are two ways to have Moshi create adapters automatically:
- Reflection
- Codegen
Moshi Reflection
As the name implies, it’s using reflection to generate adapters. All you need to do is add the KotlinJsonAdapterFactory:
moshi {
addLast(KotlinJsonAdapterFactory())
}Note that we use addLast instead of add because Moshi adapters are ordered by precedence and you want your own adapters to “be first”.
Of course now we also need an adapter for the Instant class:
object MoshiInstantAdapter : JsonAdapter<Instant>() {
@FromJson
override fun fromJson(reader: JsonReader): Instant =
Instant.parse(reader.nextString())
@ToJson
override fun toJson(writer: JsonWriter, value: Instant?) {
writer.value(value.toString())
}
}The adapter needs to be registered:
moshi {
add(MoshiInstantAdapter)
addLast(KotlinJsonAdapterFactory())
}And we’re good to go, you’d think… but no:
No JsonAdapter for class java.util.ArrayList, you should probably use List instead of ArrayList (Moshi only supports the collection interfaces by default) or else register a custom JsonAdapter.
After a lot of digging I found that Moshi simply doesn’t support ArrayLists out of the box and I found this answer on StackOverflow https://stackoverflow.com/a/61272734/534471:
Registering the ArrayListAdapter and we’re finally and truly done:
moshi {
add(MoshiInstantAdapter)
add(MoshiArrayListJsonAdapter.FACTORY)
addLast(KotlinJsonAdapterFactory())
}Moshi Codegen
Moshi reflection is slow and adds a good amount of extra code (kotlin-reflect library) -> important for mobile apps. Codegen on the other hand uses an annotation processor. It generates a small and fast adapter for each of your Kotlin classes at compile time.
To use Codegen we need to add
To use Codegen we need to add the Kotlin Annotation Processing Tool (kapt) and then the Codegen annotation processor:
plugins {
kotlin("jvm") version "1.4.32" apply false
kotlin("kapt") version "1.4.32" apply false
}kapt("com.squareup.moshi:moshi-kotlin-codegen:1.12.0")
Please note that kapt needs to come after the jvm plugin. Also note that it’s kapt(“…”) not implementation(“…”). I wasted time because I used implementation (copy & paste error) and wondered why it didn’t generate the adapter classes (got “Failed to find the generated JsonAdapter class”).
We also need to annotate each data class with @JsonClass(generateAdapter = true):
@JsonClass(generateAdapter = true)
data class Account(@JsonClass(generateAdapter = true)
data class Customer(
That’s it.
Conclusion
- Gson (19.4k stars, 3.8k forks, last commit May 13, 2020):
easy to setup, the only lib that worked out of the box (kind of). I definitely spent the least amount of time, getting Gson serialization up and running. If you don’t want to waste time, getting the bleeding edge tech to work and don’t care about maximum performance, Gson is the way to go. - kotlinx.serialization (3k stars, 320 forks, last commit April 5, 2021):
the new sheriff in town. It took me longer to set it up than Gson but it’s obviously made for Kotlin and as Kotlin plugin does its magic at compile not runtime. It also supports Kotlin multiplatform, you can use it for Android, Ktor, Gradle/Maven, Kotlin/JS, Kotlin/Native etc..
The only “beef” I had with kotlinx.serialization is the fact that it supports classes with @Serializable annotation but doesn’t support Serializable classes (the ones extending the Serializable interface). I understand that creating serializers at compile time is impossible for interfaces “@Serializable annotation is ignored because it is impossible to serialize automatically interfaces or enums. Provide serializer manually via e.g. companion object)” but there could be a reflection based solution for interfaces at runtime. - Moshi (7.2k stars, 598 forks, last commit April 10, 2021):
was hard to set up, the documentation is confusing if you’re not targeting its default platforms (Android and Java). Right now I don’t see a reason why I would take Moshi over any of the other two libraries. The project is quite active though so maybe it will evolve to offer better/easier support for Ktor and Kotlin multiplatform.
I personally will use kotlinx.serialization. I didn’t find any show stoppers so far and it seems to be a very active active project although I will definitely keep an eye on Moshi and where it’s going.
Note: while researching https://github.com/papsign/Ktor-OpenAPI-Generator (article to follow), I got the following error with kotlinx.serialization
Exception occurred: Serializing collections of different element types is not yet supported
I switched back to using Gson and everything works as expected. Using older but battle proven libraries seems to have its advantages.
Addendum: performance
Comparing the performance of serialization/deserialization for the different libraries seems is a challenging task. I noticed major differences between different benchmarks. Some see kotlinx.serialization as the winner, some Moshi and others Jackson:
- https://medium.com/stanwood/save-my-ass-benchmark-of-json-deserializers-on-android-28341c1e82df
-> kotlinx.serialization > Gson > Moshi >>> Jackson
(compares older versions of the libs, benchmark from 2018) - https://proandroiddev.com/goodbye-gson-hello-moshi-4e591116231e
-> Moshi > Gson
(doesn’t run benchmarks of its own just references the next benchmark) - https://github.com/ZacSweers/json-serialization-benchmarking/
https://zacsweers.github.io/json-serialization-benchmarking/
Moshi > kotlinx.serialization / Gson - https://tech.olx.com/android-jetpack-benchmark-json-parsers-performance-1e76031a296b
Jackson > kotlinx.serialization > Gson > Moshi
(compares older versions of the libs, benchmark from 2019)
If performance is important to you, you might want to run your own benchmarks for your specific environment and for your specific use cases. Surprisingly my own benchmarks put Gson ahead of Moshi (with reflection being slower than codegen) while kotlinx.serialization came in last but since I spent about 15 minutes on those benchmarks I don’t really trust them.
My experience with complex backend services is that network performance / latency is usually the most time consuming aspect (and sometimes database performance) while serialization is really negligible so I won’t have sleepless nights pondering the performance of different serialization libraries.
Feel free to provide feedback to the article and happy coding!
