Moshi Kotlin Code Gen: An Open Source Story
For context — see the previous post I wrote about Moshi’s new Kotlin code gen support. This blog is a somewhat technical, mostly personal account of how it came to be.
Origins
Moshi’s Kotlin code gen has an interesting backstory. It started as a prototype in my personal CatchUp app as a tooling subproject called MoshKt (pronounced “Mosh-kit”, like mosh pit. It was a play on Kotlin’s .kt extension, Moshi’s name, and a tendency to end developer tools in kit. ¯\_(ツ)_/¯ don’t @ me).
There were existing solutions out there for Json serialization in Kotlin, but none that I was happy with. Up to that point I was using auto-value-moshi, a library I help maintain with its author Ryan Harter. This worked with Kotlin, but came at an annoying cost of having to use AutoValue despite the fact that I was writing Kotlin code. data classes do the same thing in a more concise, idiomatic way! AutoValue also generates Java code, imposing a nontrivial build cost due to mixed source sets. This seemed like the least annoying option to me for a long time though. I tried writing a prototype processor like this some time last year (after the initial release of KotlinPoet), but gave up after realizing how many gotchas were involved when you couldn’t understand language features since all the processor saw was the raw Java-fied version of the Kotlin code.
Fast forward to January-ish this year — Kotshi existed by then for data classes, but the added work required to share Kotlin-specific context and support things like default values seemed to confirm my early suspicion that Kotlin support would require a lot of extra work from the consumer to convey necessary information to the processor.
Metadata
On the other hand, I was aware of Kotlin’s @Metadata annotations and knew that reading them could tell me the Kotlin-specific information I was looking for.
@Metadataannotations TL;DR: these are annotations generated on to every Kotlin type with a serialized Protobuf detailing all the Kotlin language features in that class. This is how Kotlin’s compiler reads relevant information from other Kotlin types and libraries. Relevant talk.
There’s a neat little library on GitHub by Eugenio Marletti called, creatively, kotlin-metadata. This library pulls in the embeddable Kotlin compiler and uses its internal Protobuf serialization classes to read these @Metadata annotations exactly as Kotlin reads them. The protos can then tell you in a nice, modeled fashion various information about the classes and their Kotlin specifics. Default values, property names, typealiases, you name it. It doesn’t try to recreate the javax.lang.model.element APIs that annotation processors already see, rather it complements it with just enough information to tell you the relevant Kotlin bits. kotlin-metadata is also what Room uses in its Kotlin support.
So, one January afternoon, I sat down to try to explore this library and try my hand at writing a JsonAdapter-generating processor again as a clean room implementation.
… and finished it a few hours later. Mostly. It didn’t support generics, default values, or custom @JsonQualifier annotations yet, but it supported pretty much everything I needed to be functional in CatchUp. A nice bonus was that the compile times for the various service modules went down significantly with it. Instead of having to do this pathological loops of Kotlin code and generated Java code (AutoValue as well as a factory processor), it was now just a single shot Kotlin -> Kotlin pipeline.
I continued iterating on it a few days later to add some things like adapter caching and using the selectName() API. I talked with Eugenio quite a bit to better understand how to bridge the gap between what the element APIs gave and what the Metadata protos offered, and he was extremely generous with his time to help.
On to Moshi
At this point I was excited with what I had and wanted to share. I sought Ryan’s advice, which was met with a response that roughly went:
I like this, brb copying it into our repo and migrating the whole app to it
This was a good sign :). He also devised the clever default values solution leveraging copy(). Kotshi was very much considered the community standard by this point though, and I wasn’t quite sure what to do since proposing my implementation would basically be proposing shelving most of Kotshi’s current implementation. I sought Jesse Wilson’s (one of Moshi’s maintainers) thoughts on it, and he said I should definitely try to find a path to contributing to Kotshi. I agreed and said I’d look into it more. When I went back to explore Kotshi again though, I learned it actually still generated Java code too, which made it feel like these really were two different libraries.
Full disclosure: Kotshi always wanted to generate Kotlin code and I embarrassingly missed this issue about it.
I sent Jesse a link to the code I had so far in case he was interested in reading, to which he responded something to the effect of:
oh, this is awesome. Want to just go right to Moshi?
As someone that’s been an avid user of Square libraries and admirer of Jesse’s work, this was a really cool moment. I opened the initial PR on February 9th, a (massive) code dump of MoshKt in its current form from CatchUp. Mike Nakhimovich chimed in with a great idea of trying to get it to pass Kotshi’s extensive tests as a baseline, which in turn led to implementing almost all of the remaining missing functionality (generics support, partial @JsonQualifier annotations support, and some others). This was a huge step for the progress of the library, and while the Kotshi tests were not checked in to Moshi’s repo at merge (per Jesse’s request for licensing reasons), they were a huge part of hardening the library for production usage.
About a month and a lot of review from a lot of people later, the PR was merged. Jesse and I iterated on it quite a bit over the next couple of months after that to clean up the implementation (much of it was written procedurally on that afternoon after all), added full support for @JsonQualifiers, getting more robust unit testing via rudimentary compile tests, and support for non-data classes.
In Closing
Moshi Kotlin code gen is a project I’m really proud of. I learned a lot along the way too, ranging from how Kotlin works under the hood to how to better navigate the sometimes tricky waters of human dynamics in open source.
Special thanks to Eugenio, Jesse, Ryan, and all the others that helped make this happen. Open source doesn’t always take a village, but it’s so much better when the village shows up anyway.
