Migrating to Bazel from Maven or Gradle? Part 5 — How to avoid Jar dependency hell with Bazel
Dependency management is a difficult task for any build tool. Those of us old enough can remember dll hell in 16bit windows. Since the dawn of the JVM and the creation of the classpath, the term transformed to JAR hell, with multiple contributing factors, such as implicit missing jars, version conflicts, fat jar shadowing and others.
Explicit verbose build configuration is one of Bazel’s pillars. It is also present in the way 3rd party dependencies are configured. No more implicit “magic” that will cause issues only to be manifested at runtime in production.
Bazel Repository Rules
All External dependencies are defined in repository rules that enable non-hermetic download operation at the loading phase of Bazel, even before any build target gets built.
The dependencies’ jars are downloaded and written to the source repo’s external workspaces path. Each jar will get its own external workspace — with WORKSPACE file and BUILD.bazel file with build targets:
jvm_maven_import_external target for commons-lang3One version policy
Unlike Maven or Gradle, where the external dependencies are defined for modules or sub-projects. In bazel, the external dependencies are defined for the entire repository (aka workspace). This as part of the basic design that allows for building from source an entire repo’s codebase at once.
This means that basically, one version of a dependency will be used for all libraries and applications inside the repo.
It will require to change code that does not compile with that single version’s API. Even more troubling is unexpected runtime behaviour. For instance if you use spring DI.
On the other hand having old versions of 3rd party libraries means that you don’t get potential bug fixes and performance improvements.
Multiple versions
One obvious solution for the one version limitation is to split the code into multiple repositories, See my previous post: Can Bazel work with Manyrepo?
If you still want to keep your code in one repo, there is also the concept of nested workspaces, where you can define a separate internal workspace with its own external dependencies definitions.
But theses workspaces will be considered external to the main workspace, just as the ones that are in separate repos, and you will still lose the ability to build the entire repository’s code at once.
You can of course encode the version in the external repo target’s name. E.g. com_google_guava_guava_25 but this incurs the danger of having both versions in the classpath in non-deterministic order, as is the case in the following example:
Version Policy at Wix
Wix tries to have all of its jvm code rely on the same 3rd party versions. There is a global list of “managed dependencies” that is updated in regular intervals with new dependencies and new versions.
On those updates, some code and tests break and it’s up to the code owner to fix it.
This is done in order to prevent potential bugs due to older library versions, and to allow for easier sharing of code and general interoperability.
Closure Resolution
All the transitive jars’ closure of the direct external dependencies your build targets need, have to be explicitly declared for bazel to fetch. Otherwise you can expect compile-time errors or worse — runtime failures.
E.g. if your library depends on guava, you have to declare a repository rule target for it and also for error_prone_annotations as well, because it is a transitive dependency of guava.
There are several available tools/rules to choose from that automatically resolve the external jars transitive closure and serializes it into bazel style configurations:
bazel-deps
Currently (April 2019) the de-facto standard way to import external jar dependencies into a bazel repository. This cli tool expects as input a dependencies.yaml file that should include the direct dependencies:
First, it resolves the dependency graph using coursier, which is a performant artifact fetching library written in Scala.
Next, it serializes the dependency graph into build targets in a special folder in the repo called 3rdparty:
Example of a build target in the dependency graph created by bazel-deps:
Next, by invoking jar_artifact repository rule it will download the jars and write a java_import BUILD rule for each jar in the external workspaces path (See Bazel Repository Rules section above). This jar is referenced in the exports argument of the build target, as you can see in the example above.
Bazel-deps has cool features like:
- support for exclusions
- different modes for version conflict resolutions (take highest, specify default, fail)
- The ability to declare replacements to transitive dependencies with local definitions.
One disadvantage is that on every change in the direct dependencies you have to call the external cli tool and recalculate the entire graph.
Another disadvantage is that information on the 3rd party deps is split into two locations (version information in the .yaml/.bzl file and the transitive dependency information in the 3rdparty folder.)
rules_jvm_external
A new player in this space which was developed by the bazel team and is part of the official bazel github organization. It deprecated the generate_workspace cli tool.
The direct dependencies are defined inside a maven_install repo target:
This repository rule will resolve the transitive closure and auto-generate the following targets in the external workspaces path:
The advantage of having built-in bazel rules instead of external cli is that for any change you don’t need to run any special command, just the regular
bazel build //…Like Bazel-deps it also supports maven semantics such as version pinning and exclusions.
The ability to define multiple maven_install repo targets with different versions looks promising, but I’m not sure if this will allow for true coexistence of multiple versions of 3rd party dependencies in a single repo. Time will tell.
It still has bazel-deps’ disadvantage of recalculation of the entire graph on each change.
And also the transitive dependency graph is only encoded in the external workspaces path, which means it is not source controlled.
What we do at Wix
As mentioned above, wix has a global “managed” dependencies list.
One of our repositories has the transitive dependency closure of this list which all the rest of the repositories load in their WORKSPACE file.
In order to allow specific version overrides locally, each repo also defines its own local dependencies before the managed ones are loaded:
The dependencies are encoded using jvm_maven_import_external repository rule:
The distributed nature of this rule, where every direct dependency is in its own target, means it is possible to override it easily and also means that any change requires incremental fetching.
We also have a cli tool that can update a repository’s local dependency overrides (including version and transitive closure) using the Maven Artifact Resolver library.
To summarize, each dependency closure resolution technique has its advantages and disadvantages, but having one of them is essential for your bazel repository to successfully bring external jars dependencies in a coherent, maintainable way.
The previous post in the series was: Can Bazel work with Manyrepo?
This post concludes the series. I hope you enjoyed it!
Migration to Bazel is not a cakewalk, but for large complex jvm code bases it is definitely worth it.
You can find a link to all of the series posts in the introduction
Thank you for reading!
Please also share on Facebook and Twitter. If you’d like to get updates, follow me on Twitter and Medium.
You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.
If anything is unclear or you want to point out something, please comment down below.
