Pushing the limits of Kotlin annotation processing

Is annotation processing really supported in Kotlin? ☐ Yes ☐ No ☑ It’s complicated

There are times when writing code “by hand” is not enough, because the code you’re writing is mechanically derived from other code, and the result is pure boilerplate: a nightmare of copy/paste.

That’s when metaprogramming becomes a necessity, and in Java there are two ways to achieve such capabilities: reflection and annotation processing.

Reflection is perhaps the easiest solution, but has a few downsides:

• it might cause performance problems if misused (especially on constrained platforms like Android)
• it’s less “safe”, as code that compiles fine might crash at runtime
• and most importantly: it can only inspect and access existing constructs, like methods and classes, but not create new ones

Kotlin is a much richer language than Java, and as such it stores all this extra metadata in special annotations. If you use the standard reflection API in Kotlin, you’ll only be able to inspect things from the JVM perspective; that’s why Kotlin comes with an external reflection library that allows to access the extra information. One thing to note is that at the moment it’s rather large, which is another downside when used on Android.

Annotation processing is much more powerful because not only it allows developers to inspect pretty much everything about the structure of code before running it, but offers the ability to generate extra classes.

Lately I’ve been involved in a few Kotlin projects, both personally and at work, which require the use of annotation processing, and they do so in a way that requires to either/or:

• inspect Kotlin types, because as said before they carry extra information compared to their JVM equivalent
• generate Kotlin code, as that’s the only way to access some of the advanced features that Kotlin offers

The whole issue revolves around one question:

“Is annotation processing supported in Kotlin, and how well does it work?”

To answer that, let’s start with a bit of history, necessary to understand everything that follows…

At the beginning, Kotlin didn’t support any annotation processing.

Those were dark times.

Then in Kotlin M12 (before 1.0) kapt1 came in. It’s implemented in the following way:

1. The Kotlin compiler analyses the code and serialises the information about the annotations.
2. A custom annotation processor, which is passed to javac, loads the annotation data and calls other annotation processors.

Later on, stubs generation was added as a workaround to support references to generated code:

1. At the first step mentioned above, the Kotlin compiler also produces stub class files, which contain only method signatures.
2. The stubs are added to the javac classpath, which performs normal Java compilation with annotation processing.
3. Kotlin performs a normal compilation (generated code is processed as any other Java/Kotlin source code).

Stubs generation allows to refer to generated code in method bodies, however there are still some limitations. Note that the generated code does not exist at step 1, so all references to generated code in function signatures are generated as error/NonExistentClass. If some Kotlin function signature, containing any reference to generated code (as a parameter type or a return type), is processed by an annotation processor, the processor may generate a reference to a NonExistentClass in generated java code. This will cause a Java compilation error (unresolved reference) which is quite hard to diagnose, because javac does not report any additional information about errors in generated Java code.

That’s why in Kotlin 1.0.4 the experimental kapt2 was announced, as an attempt to overcome that limitation. It implements JSR 269 by wrapping the Intellij platform AST. However it became apparent that the Intellij platform was not optimised for such use-case, so annotation processing could be very slow in some cases.

kapt3 is a replacement of the kapt2 implementation. It generates javac’s Java AST from Kotlin code and reuses javac’s annotation processing directly.

If you’re declaring this in your build.gradle, you’re using either kapt2 or kapt3:

apply plugin: ‘kotlin-kapt’

To recap:

• There are two kapt from a user point of view: one generates stubs and is stable, and the other is next-gen, experimental and work-in-progress.
• kapt2 and kapt3 are different implementations of JSR 269.
• All the differences between kapt2 and kapt3 are implementation details, which are not usually documented (at least not in the user documentation).

(Huge thanks to Alexey Tsvetkov from JetBrains for providing precise information about this section, I’m pretty much quoting him!)

After reading the above paragraphs you might argue that Kotlin does indeed fully support annotation processing, or at least it’s close, and you would be right… At least, in part.

To understand my point, remember that all flavours of kapt are based on apt, which is a tool created for Java. As such, the first huge limitation is:

Annotated Kotlin code being processed will look like Java code, and there’s no official way to read the extra language metadata.

This means that existing annotation processors will work fine, but any new annotation processor designed specifically with Kotlin in mind will have a very hard time doing anything advanced.

If you’re really brave, there is one way to extract this information in kapt1, but it’s not exactly straightforward: it requires parsing the FlatBuffers inside the @Metadata annotations placed on the stubs of the Kotlin classes, and then piece together the information… But that’s material for a future post.

Historically, annotation processing in apt takes place in rounds: at each round you process the annotations of the previous one, and have the ability to generate Java sources or .class files that will be analysed and fed to the next round, until there’s no more annotations to process.

Which takes us to the second huge limitation:

Generating Kotlin code is only possible in kapt1, with no support for multiple processing rounds.

In kapt1, you’re given the (undocumented) processor option kapt.kotlin.generated which contains the path to a folder where you can manually create .kt files: these files will be compiled by the Kotlin compiler, but if you output any annotations they won’t go through the annotation processing phase!

I opened an issue on the Kotlin tracker to add the missing Kotlin code generation functionality to kapt2/kapt3, which at the moment of writing this article is set as “In Progress”. Feel free to vote on it to stay updated.

I encountered this problem recently, and the only viable workaround I found was to split the code in two Gradle modules: the first module generates and compiles the Kotlin source files, which will then be processed by the second module. Unfortunately this only works when there’s a known set of “rounds”, and forces you to change your project structure.

Getting this to work in an Android project was even more complex, so I created a sample demonstrating how to do so.

In conclusion, it’s clear that Kotlin doesn’t yet provide first-class tools for processing annotations in a “native” way. If you are like me and desperately need those tools, there are ways to work around some of the issues, despite a lot of complexity and limitations.

I can only hope that JetBrains will fill this gap in the tooling, which in my opinion is the last missing pillar to elevate the language to its rightful place as the most amazing JVM language out there.