Java Class Shadowing and Shading

Photo by Justin Leibow on Unsplash

Hello, reader! We’re going to go on a journey through the treacherous world of Java Class shadowing, discuss circumventing using shading, and finish with a walk through using Gradle. If you’re already familiar with the perils of Java Class shadowing feel free to skip ahead to the section on shading.

Java class shadowing

Class loaders and the classpath

A Java application runs by loading Java classes on to the JVM. Classes are loaded by class loaders. In most cases¹ your application code will be loaded by a single class loader: the System class loader. The System class loader looks up classes on the user-specified classpath². The classpath includes some combination of directories, jars, and zip files that contain class files and resources. Classes are looked up by their fully qualified name (FQN), for example org.example.foolibrary.FooClass. This is important! Since they are looked up by the FQN, if the classpath happens to have multiple classes with the same FQN, the class loader will return a single one based on the order of directories, jars, and zip files on the classpath. A common example of this is having two different versions of a library jar on the classpath; they often have classes with the same FQNs. If the classes are identical this isn’t a problem, but if they differ then any code using those classes will likely have unexpected behavior when the class loader loads up the one they weren’t expecting. If the version provided is (binary) incompatible with the one they were expecting, this can even result in a runtime failure. Behavior may even be different in different environments, based on how the classpath is constructed.

This quirk is known as class “shadowing” in the Java ecosystem — one version of the class is ignored in favor of the other one. Which specific one gets ignored depends on ordering on the classpath. One way of working around this is for libraries to be loaded into different, custom class loaders. There are some ways to do this (e.g via OSGi) but in practice they can be difficult to set up, have their own sets of issues, and aren’t widely used. Due to the complexity of managing custom class loaders we will ignore solutions that use multiple class loaders in this post.

Dependency resolution

It’s rare for a project to have no dependencies; a project will often pull in multiple well-supported and battle-hardened³ libraries to avoid reinventing the wheel and to integrate various components. The libraries often have their own dependencies which, in turn, have their own, leading to a complex tree of dependencies. Build systems such as Maven and Gradle will pull down all required dependencies (including all transitive dependencies) and ensure they are on the compile and runtime classpaths as needed. Dependencies are specified using “GroupId ArtifactId Version” coordinates, where the GroupId is a namespace corresponding to an organization or project (e.g. org.apache.curator), the ArtifactId identifies the what the project is generally known as (e.g. curator-recipes) and the Version is the specific version of that jar (e.g. 4.3.0).

Complex dependency tree will commonly leave you with dependencies on two different versions of a library. For example, let’s say that my-app depends on some-library and another-library. some-library depends on org.example:core-library:1.0 while another-library depends on org.example:core-library:2.0. We can visualize this like so:

Two versions of org.example:core-library in the dependency tree

The two versions of core-library likely have classes with the same name, and we’ve already established that having classes shadow each other can cause odd behavior. Before pulling down and supplying those dependencies to your application, build tools run a step called dependency resolution, where they identify dependencies with the same GroupId and ArtifactId but different Versions and resolve them to a single Version. This ensures that a single version of that library ends up on the classpath and the app owner does not run into any unexpected surprises at runtime.

The problem is that this resolution strategy doesn’t really guarantee anything apart from one jar (per GroupId/ArtifactId pair) being on your classpath. The single jar you end up with may not be compatible with all of the jars that depend on it, leading to runtime failures. In practice, library authors are generally good about maintaining binary compatibility when releasing new versions of their jars, and so using a jar that is newer than the requested version doesn’t break applications . However, library owners sometimes make inadvertent incompatible changes, and other times they will intentionally release breaking changes while revving the major version of their library. Build tools will happily resolve the dependency (unless you configure them to fail fast on conflicts), resulting in you having to deal with debugging and working around runtime failures in your application.

Resolving conflicts and shadowing

The ideal way to sidestep this problem would be to put the onus on library owners to never make backwards-incompatible changes to their library. Unfortunately this is unrealistic since APIs evolve over time, and so when library owners need to update their API, they should do two things:

1. Bump their ArtifactId, for example going from org.example:core-library1 to org.example:core-library2
2. Update the package for all their classes, for example from org.example.corelibrary1to org.example.corelibrary2

This way both jars and both sets of classes can coexist on the classpath and are used by the libraries that depend on them. some-library depends on the classes in the package org.example.corelibrary1 and another-library depends on the classes in the package org.example.corelibrary2. Bumping the ArtifactId is necessary to allow dependency resolvers to pull down both jars and add them to the classpath, and package-renaming is necessary in order to have the classes not shadow each other between the two jars. With the ArtifactId and package changes the two versions are effectively different libraries with code that happens to be similar.

While there are a few shining examples of Java libraries that follow this⁴, the vast majority of libraries do not, causing immense amounts of pain to Java application developers around the world⁵. In the winter of despair there is a shining beacon of hope that provides respite to Java application developers — a technique called shading.

Shading in Java

In Java, to “shade” a dependency is to include all its classes and the classes from its transitive dependencies in your project, often renaming the packages and rewriting all affected bytecode. In essence, you copy all the classes from a dependency’s jar (and from the transitive dependencies coming from it) into your project, relocate them (by renaming the package to something specific to your project) and update all internal references to use the relocated classes. Your project’s code then uses the relocated packages and you no longer have a dependency on the original jar.

Shading in libraries

Let’s revisit the example from earlier:

Two versions of org.example:core-library in the dependency tree

The preferred solution here is to ask the maintainers of some-library and another-library to shade and relocate core-library instead of making it a dependency. If the maintainers agree, we end up with the following dependency tree:

No more dependency conflicts

Internally, some-library and another-library still use their respective versions of core-library, but they include the classes from core-library directly and relocate them to prevent class shadowing. Let’s say that some-library relocates using the prefix somelibrary.shading and another-library uses the prefix anotherlibrary.shading. This means that usage of core-library classes in some-library is rewritten like so:

some-library rewrites core-library’s packages

Since another-library does its own shading, the usage of core-library classes in another-library is rewritten like so:

another-library rewrites core-library’s packages

It is important for the maintainers of the libraries to pick different prefixes when relocating the classes in core-library, otherwise its classes would have the same names and would shadow each other in our application. Note also that we don’t actually need both maintainers to shade their usage of core-library; for our purposes only one of them needs to shade since that fixes the transitive dependency version conflict as well as class shadowing.

Sometimes, however, library maintainers decide they do not want to shade their dependencies, leaving the application owners no choice but to shade the libraries themselves.

Shading in applications

Instead of my-app depending on some-library and another-library, we as application owners can shade some-library and another-library wholesale. For this example we shade them into the internal modules com.myapp:some-library-shaded and com.myapp:another-library-shaded. We then make my-app depend on the shaded versions instead:

Under the hood, some-library-shaded still uses core-library:1.0.0, but instead of core-library being a transitive dependency, the classes are included in some-library-shaded directly. In addition, we rewrite all packages in some-library-shaded and core-library and prepend our chosen prefix, myapp.shading.somelibrary, so for example:

some-library and its transitive dependency core-library are relocated

There’s a similar story where it comes to another-library-shaded — it brings in the classes from core-library:2.0.0, and we choose a different prefix myapp.shading.anotherlibrary:

another-library and its transitive dependency core-library are relocated

Similar to the scenario where some-library and another-library did their own shading, we now have two different versions of the classes from core-library on our classpath, but they are included in different jars (in the some-library-shaded and another-library-shaded jars) so we don’t run into dependency resolution issues. We also have no issues with class shadowing, since the different versions of the core-library classes have different packages, making them effectively different classes. some-library’s bytecode was rewritten to use the myapp.shading.somelibrary.org.example.corelibrary classes, while another-library's bytecode was rewritten to use the myapp.shading.anotherlibrary.org.example.corelibrary classes.

The difference between the library owners doing the shading and us doing the shading is that we have now copied and relocated all classes in some-library and another-library and their dependencies, even ones that we probably didn’t need to shade. If the library owners did the shading themselves, they could choose to only shade specific problematic dependencies. In addition, every app owner that runs into this problem will have to shade on their own.

Doing shading by hand is a little involved, but fortunately there are plugins for build tools like Maven and Gradle that make it as easy as adding a few lines of configuration. Later on in this post we’ll go over a real-world example with Gradle that uses the excellent com.github.johnrengelman.shadow plugin, but before that let’s talk about some shading best practices and drawbacks.

General guidelines for shading

• Set up a separate module whose purpose is to shade one jar and any transitive dependencies and do nothing else. You can choose for this to be a submodule in the project that uses it or a stand-alone module. This clear separation makes it easier to reason about what should and should not be on your compile or runtime classpaths.
• Pick a rewritten package name (a prefix, usually) that is specific to your project and the library you are shading. Making it project-specific ensures that there can be no conflicts with other jars on your classpath, and including the name of the library you’re shading allows you to shade in another library that may have some of the same transitive dependencies. For example, shade some-library (and its transitive dependencies) with the prefix myapp.shading.somelibrary, so that a transitive dependency’s class org.example.corelibrary.CoreClass becomes myapp.shading.somelibrary.org.example.corelibrary.CoreClass, allowing you to also have myapp.shading.anotherlibrary.org.example.corelibrary.CoreClass on your classpath.
• Give some thought to which transitive dependencies you shade and which ones you decide not to shade. You can certainly choose to shade all transitive dependencies, but this bloats your classpath and you may end up with multiple identical copies of classes. The more transitive dependencies you shade, the larger and more wasteful your final package will be. For instance, you may choose to exclude a transitive dependency org.apache.commons:commons-lang3 from shading since that jar is pretty stable and may be shared transitively by other libraries you pull in.
• Ensure you don’t have any non-package-renamed classes included in your shaded jar, otherwise you’re forcing class shadowing to occur if you have those classes coming onto your classpath via some other jar.
• If you’re building a library, do not expose shaded classes on your “compile” classpath — you do not want anyone consuming your jar to have access to shaded jars. If the shaded classes are visible, others may end up using them, which means you can never update the version of your shaded dependencies.
• Pin to a specific version of your shaded dependency. The set of transitive dependencies in newer versions of your dependency may be different, and so the set of classes you choose to shade may be different. If you ever decide to upgrade the version you use, you should go through the entire process again to decide which transitive dependencies should be shaded and how package relocation should work.

Shading isn’t a silver bullet; there are a few drawbacks:

• Every dependency you shade adds to your final package size, and every package you relocate can add to the number of classes on your classpath. Apart from the raw footprint of your package, it can also be confusing for developers working on it trying to figure out which version of a class to import.
• Debugging libraries with shaded classes can be fairly painful — IDEs get confused about where to download sources from and usually present you with decompiled class files. It isn’t particularly pleasant to debug without source code and associated documentation.
• Java allows you to load a class via reflection using its name. References to package-renamed classes in bytecode are updated by shading plugins, but package renaming breaks breaks dynamic class loading using reflection.

To complete our arduous journey we’re going to walk through an example shading org.apache.curator:curator-recipes:2.12.0 using Gradle and com.github.johnrengelman.shadow. For a complete version of the example build.gradle, see https://github.com/akhaku/gradle-shading-demo/blob/master/curator2-recipes-shaded/build.gradle

Step-by-step shading with Gradle

1. Add the plugin

First we need to add the plugin we will be using. In this example, we will be using the Gradle plugin com.github.johnrengelman.shadow. Add the following to your build.gradle:

buildscript {
  repositories {
    jcenter()
  }
  dependencies {
    classpath 'com.github.jengelman.gradle.plugins:shadow:5.2.0'
  }
}

This tells Gradle to use the shadow plugin when building. The shadow plugin adds the shadowJar task and shadow configuration to your module.

2. Configure the top-level dependency we want to shade

Add a dependency on the jar you wish to shade. For this example, we will be shading org.apache.curator:curator-recipes:2.12.0.

dependencies {
  implementation 'org.apache.curator:curator-recipes:2.12.0'
}

3. Configure any transitive dependencies we don’t want to shade

Rather than shading every single transitive dependency, we’re going to opt to not shade netty and slf4j-api since they are widely-used libraries with stable APIs. Our shaded jar will instead depend on those libraries directly. This allows sharing those jars with any other library pulled in by the application that needs them.

In the dependencies block in your build.gradle, add the following:

shadow ‘io.netty:netty:3.7.0.Final’ // from ZooKeeper
shadow ‘org.slf4j:slf4j-api:1.7.6’ // from curator-client

Separately, add this to your build.gradle:

shadowJar {
  dependencies {
    exclude(dependency(‘org.slf4j:slf4j-api’))
    exclude(dependency(‘io.netty:netty’))
  }
}

Note that these exclusions only exclude those specific jars and not any transitive dependencies they bring in. netty has no required dependencies and slf4j-api has no dependencies at all, so we’re good here.

4. Set up the packages we want to relocate

We now need to set up package relocation to prevent accidental shadowing. The shadow plugin supports this, and rewrites byte code to update imports in all of the shaded classes to reflect the relocated packages. We want to relocate all packages, so add the following to your build.gradle:

import com.github.jengelman.gradle.plugins.shadow.tasks.ConfigureShadowRelocation
task relocateShadowJar(type: ConfigureShadowRelocation) {
  target = tasks.shadowJar
  prefix = ‘mylibrary.shading.curator2’
}
tasks.shadowJar.dependsOn tasks.relocateShadowJar

5. Publish our shaded jar

The final step is to tweak our module descriptors (usually a Maven pom) to publish our shaded jar with dependencies from the shadow configuration. Add the following to your build.gradle:

publishing {
  publications {
    mavenPublish(MavenPublication) { publication ->
      project.shadow.component(publication)
    }
  }
}

That’s it! When you publish your project your jar should contain all your shaded dependencies and should only depend on the transitive dependencies you explicitly excluded and added to the shadow configuration. You can also depend on your shaded jar in a sibling module — just make sure you specify the shadow configuration.

Final thoughts

Java class shadowing is a common problem in applications that have a complex dependency tree. Shading is one of the most popular ways of tackling that problem. It provides library owners with tools to preempt shadowing in applications that use their libraries, and also provides application owners with a way out when they run into class shadowing caused by conflicting dependencies. While shading has certain disadvantages, it is a powerful tool when used correctly and is often the least brittle way to solve these sorts of problems.

Many, many thanks to Martin Chalupa, Andrés Moreno, and Yuri Zats for reviewing various drafts of this post and providing valuable feedback.

Footnotes
[1] There are other built-in class loaders, but they are generally used for core Java code and certain extensions. To learn more about class loaders see the Oracle docs and this article.
[2] Also some JRE-level locations
[3] A programmer can dream
[4] Thank you org.apache.commons:commons-lang3!
[5] At least we don’t have to worry about left-pad