Reaktive binary compatibility: how we achieve it
--
As part of the Bumble family — the parent company operating Badoo and Bumble apps — one of my main projects was involved in the team that created Reaktive library — reactive extensions on pure Kotlin.
Whenever possible, any library must maintain binary compatibility. Whenever various versions of a library are not compatible in terms of their dependencies, there are going to be crashes in runtime. We may encounter this problem when adding Reaktive support to MVICore.
In this article I am going to briefly explain you what binary compatibility: its peculiarities in the case of Kotlin; how it has been supported at JetBrains, and how it is now supported at Bumble as well.
The problem of binary compatibility in Kotlin
Let’s say we have a wonderful library. com.sample:lib:1.0, with the following class:
Based on this library we have created a second library, com.sample:lib-extensions:1.0. Among its dependencies is com.sample:lib:1.0. For example, it contains the factory method for A class:
Then we release a new version of our library com.sample:lib:2.0 with the following modification:
This modification is entirely compatible from the point of view of Kotlin, right? With the default parameter we can continue to use the construct val a = A(a), but only if all the dependencies are fully re-compiled. Default parameters are not part of JVM and are implemented using a special synthetic constructor of class A, which contains all the class fields in its parameters. If we get dependencies from the Maven repository, they are already assembled and we are unable to re-compile them.
When a new version, com.sample:lib, comes out, we add it to our project straightaway. After all, we want to be up-to-date! New functions, new fixes, new bugs!
In this case we get a crash in runtime. The createA function in the bytecode attempts to call the constructor of class А with one parameter, but such constructor does not exist in the bytecode. Of all the dependencies with the same group and name, Gradle will choose the one which has the most recent version and will add it to the build.
Most probably, you have already come across binary incompatibility in your projects. I personally encountered this when I migrating our applications to AndroidX.
You will find more about binary compatibility in the following articles: Evolving Java-based APIs 2 from the creators of Eclipse; and in a recent article Public API challenges in Kotlin by Jake Wharton.
Ways of achieving binary compatibility
You might think that all you need to do is to introduce changes that are compatible, for example, add constructors with a default value when adding new fields, or add new parameters to the functions by overloading the method with a new parameter, etc. But when doing either of these it is too easy to make a mistake so various tools have been created for checking whether two different versions of a library are binary-compatible, including:
- Java API Compliance Checker
- Clirr
- Revapi
- Japicmp
- Japitools
- Jour
- Japi-checker
- SigTest
They accept two JAR files and return a result indicating the extent to which they are compatible.
However, we are developing a Kotlin library which so far, it only makes sense to use from Kotlin. And this means that 100% compatibility will not always be necessary — for example, for internal classes. Despite being public in bytecode, it’s very unlikely they will be used outside Kotlin code. For this reason, in order to retain binary compatibility, JetBrains uses Binary compatibility validator for kotlin-stdlib. The main principle is that a dump of all the public API is created from the JAR file and written to a file. This file forms the baseline for all further checks, and it looks like this:
Once changes have been made to the library’s source code, the baseline is generated again and compared to the current baseline — and, should there have been any changes to the baseline, the test will terminate with an error. These changes can be rewritten, using -Doverwrite.output=true. An error results even if the changes that have occurred are binary-compatible. This is necessary to ensure the baseline gets updated on time, and that its changes get seen directly in the pull request.
Binary compatibility validator
Let’s explore how this tool works. Binary compatibility is achieved at JVM (bytecode) level and is not dependent on language. It is entirely possible to replace the implementation of the Java class with Kotlin, without compromising binary compatibility (and vice versa).
First, we need to know which classes the library contains. Even in the case of global functions and constants, a class is created with a file name and suffix Kt, for example, ContinuationKt. In order to get all the classes, we use the class JarFile from JDK, obtain pointers to each class and pass them to org.objectweb.asm.tree.ClassNode. This class allows us to determine the visibility of the class, its methods, fields and annotations.
During compilation Kotlin adds its runtime @Metadata annotation to each class, so that kotlin-reflect can restore Kotlin representation of the class before it gets transformed into bytecode. This is what it looks like:
You can obtain @Metadata annotation from ClassNode and convert it into KotlinClassHeader. This has to be done manually since kotlin-reflect does not work with ObjectWeb ASM.
Metadata is needed to correctly handle internal, since this does not exist in the bytecode. Changes to internal classes and functions cannot influence the users of the libraries, even though they are a public API in the bytecode.
From Metadata you can find out about the companion object. Even if we declare it private, it is still stored in the public static field, Companion, and that means that this field is subject to the requirement for binary compatibility.
Of the annotations necessary it is also worth drawing attention to @PublishedApi for classes and methods which are used in public inline functions. The body of these functions remains where they are called, and that means that the classes and methods in them must be binary-compatible. If an attempt is made to use non-public classes and methods in these functions, the Kotlin compiler will return an error and will offer to tag them with a @PublishedApi annotation.
The class inheritance tree and interface implementation are important for supporting binary compatibility. We cannot, for example, simply remove a given interface from the class, whereas it is quite easy to obtain a parent class and interfaces.
Object has been removed from the list, as there is absolutely no point tracking it.
Inside the validator there are a lot of different additional checks specific to Kotlin: verification of defaults methods in the interfaces via Interface$DefaultImpls; ignoring $WhenMappings classes for the work of the when operator; and others.
Then it’s necessary to go through all the ClassNodes and obtain their MethodNodes and FieldNodes. From the class signatures, and their fields and methods we obtain ClassBinarySignature, FieldBinarySignature and MethodBinarySignature, which are declared locally in the project. They implement the MemberBinarySignature interface, are able to determine their public visibility using the isEffectivelyPublic method, and provide their signature in a readable val signature: String format.
After the ClassBinarySignature list is obtained, this can be written to a file or to the memory using the dump(to: Appendable) method, and may be compared with the baseline, which is what happens in the RuntimePublicAPITest test:
Having committed a new baseline, we obtain changes in a readable format, such as, for example, in this commit:
Using a validator in your project
It is extremely simple to use. Copy binary-compatibility-validator to your project and change its build.gradle and RuntimePublicAPITest:
In our example one of the test functions of the RuntimePublicAPITest file looks like this:
Now we run ./gradlew :tools:binary-compatibility:test -Pbinary-compatibility-override=false for every pull request and make developers update baseline files on time.
Fly in the ointment
However, this approach also has its downsides.
Firstly, we ourselves have to analyse changes to the baseline files. Such changes do not always lead to binary incompatibility. For example, implementing a new interface leads to the following difference in the baseline:
Secondly, it uses tools that were never intended for this purpose. Tests should not have side-effects such that of some file gets written to disk, which, resulting in it being used by this test, nor should they ever pass it parameters via environment variables. It would be great to use this instrument as a Gradle task. But we really don’t want to change something in the validator ourselves, so making it easy for all its changes to be pulled from the Kotlin repository, because in future there might be new constructs in the language which would need to be supported.
And thirdly, only JVM is supported.
Conclusion
Binary compatibility and a time response to a change in its state can be achieved with the help of a Binary compatibility validator. Its use in this project required changing just two files and adding tests to our CI. Despite this solution having several drawback it is still quite convenient to use. Reaktive is now going to try to support binary compatibility for JVM in the same way that JetBrains does for the Kotlin Standard Library.
Thanks for reading!
Update: Last weekend Kotlin team finally released Binary compatibility validator as standalone Gradle Plugin. It uses the same principles I described in this article and now can be used much easier!
