Many of our core services at Zendesk are written in Scala. Over the years, we have amassed a growing number of internal libraries and external dependencies. Keeping our dependencies up to date is an important goal, but every once in a while a presumably simple version update turns into a nerve-wrecking odyssey. One common problem we encounter is the introduction of binary incompatibilities between versions.
What is Binary Compatibility?
Scala compiles into Java bytecode. Classes and traits are compiled into class files which can reference other class files. When publishing a library, all its class files are packaged into a single archive, but referenced class files from other libraries are not included. These references are only resolved at runtime by the class loader. Binary compatibility between two versions of a library guarantees that references to its classes remain valid.
Binary compatibility is crucial when dealing with transitive dependencies. When a project depends on two libraries, which depend on a shared third library, that dependency is only included once. If the two libraries depend on different versions of the shared dependency, the Scala build tool will chose the more recent version and evict any other version (unless overridden). If the two versions are not binary compatible, a number of linkage errors may occur at runtime, e.g. ClassNotFoundException, MethodNotFoundException or AbstractMethodError.
Binary Compatibility and Your Application
Let’s look at a minimal example application called App1, which has only two dependencies, server v0.1.0 to implement a web server and database v0.1.0 to access a database.
It turns out, both libraries depend on a common logging library, but with a minor version difference. Server v0.1.0 depends on logging v0.1.0, while database v0.1.0 depends on logging v0.2.0. As a result, app1 has two transient dependencies to different versions of the logging library. This situation is very common and frequently leads to a difficult to resolve dependency hell. It can be impossible to align the various transitive dependencies without recompiling and republishing direct dependencies. Let’s see if our build tool’s eviction policies can help us out: sbt will evict logging v0.1.0 and use logging v0.2.0. The dependency graph now only contains a single version of the logging library.
When updating our application, sbt will display a warning that the eviction may lead to binary incompatibilities.
Running sbt evicted provides additional information on which version was evicted, and where the conflicting dependency originates. In our example, it identifies that database v0.1.0 depends on the older, evicted logging version.
So far so good. Besides the warnings, everything seems fine. The tricky bit with binary incompatibilities is, that they only surface at runtime. The code for the example application will compile without any errors.
Only during the runtime, when the application exercises a code path that utilizes the database library with a now invalid invocation of the logging library, will linkage errors caused by binary incompatibilities surface.
How to Maintain Binary Compatibility
Unfortunately, introducing binary incompatible changes easy, but that doesn’t mean it has to be hard to avoid shipping incompatibilities. There are various code changes that break things in obvious ways, renaming classes, changing package structures, renaming or removing methods or other non-private members of a class, etc. But there are also more subtle and Scala specific ways to break downstream dependencies, including using default values for arguments or inferred return types.
Lightbend’s Migration Manager, or MiMa, is a sbt plugin that will check for binary incompatible changes in a library. In order to use it, we add the sbt-mima-plugin to our list of plugins.
The next step, is to define the list of previous versions of the library, to which the current version should remain binary compatible. The corresponding sbt setting is called mimaPreviousArtifacts.
Running sbt mimaReportBinaryIssues will then identify any existing problems, including the removed value in the Logging class that broke our example application.
Enforcing Good Practices
Given the availability of MiMa’s binary compatibility analysis, it is straightforward to build a process around it. The first step is to automatically run the check during continuous integration. At Zendesk we use Travis CI and cross-publish our libraries to different Scala versions, we therefore run the MiMa checks against all versions on each integration run.
The second step is to automatically include the last published version by default in the list of previous artifacts. We manage our release with the excellent sbt-release plugin and added a custom release step to write the current version into a file called compatible_versions.sbt. This ensures that the master branch will maintain compatibility.
Sometimes, it is necessary to break binary compatibility, in which case it is possible to manually reset the compatible versions to an empty set. The benefit is that the change can be reviewed as part of our regular code review. Instead of silently and unintentionally introducing binary incompatibilities, we can now make informed decisions when to apply and when to avoid these changes.
The last step is to use semantic versioning for all our libraries. When we do make breaking changes, we force a major version bump. While this does not completely eliminate the chance of linkage errors at runtime, it sends a strong signal to downstream consumers that things might potentially break.