EXPEDIA GROUP TECHNOLOGY — SOFTWARE

Zen and the Art of Maven Dependency Resolution

A practical guide to untangling Maven dependency issues

Photo by “Michelle Still Creates” from Pexels

“Oh what a tangled web (of dependencies) we weave”

Users of Maven will know that one of its most used and powerful features is its Dependency Mechanism. For the most part, the resolution of the various dependencies in a project “just works” without too many surprises and it’s easy enough to use Maven itself or the tooling provided by various IDEs to figure out version conflicts. However, things can go wrong when there is an interdependent relationship on different versions of the same dependency, especially if newer versions of the artifact are not backwards compatible with previous versions. Sadly not everyone uses semantic versioning properly (I’m looking at you Avro) so relying on version numbers to spot incompatibility issues often isn’t enough. Instead one may need to inspect the contents of various artifacts and how they are used to determine the cause of problems.

Issues caused by these types of dependency problems usually don’t show up at compile time in the code you are writing but instead manifest themselves via exceptions and errors when actually running the code, such as:

• A failing unit test — this is usually the best state one can hope for as it’s quick and easy to try out solutions to see if they work.
• A failing integration test — this often happens if integration tests are started as a separate forked process, possibly with a different classpath to the JVM running the tests.
• When run remotely — this often uses a different classpath which contains additional artifacts that might be scoped as “provided” in the project you are working on. These errors often occur unexpectedly and are only triggered on certain code paths and are generally the hardest to solve due to the time spent to get from build to runtime.

Read on to find out more about the types of exceptions that indicate dependency resolution problems, how to find the root causes, how to solve them, and the application of all this to a real world example.

Typical errors and exceptions

“Okay, Houston, we’ve had a problem here”

Common exceptions that indicate there is a dependency resolution problem include (but are not limited to) the following:

• java.lang.NoSuchMethodError — if an expected method cannot be found on the classpath at runtime. This is similar but subtly different to a NoSuchMethodException as the method may have been present at compile time but was in a version of a dependency scoped as “provided” that is no longer present at runtime. The method might actually exist but the types and/or order of arguments passed to it have changed between versions, look at the detailed error message for more information.
• java.lang.NoSuchMethodException — if an expected method can’t be found dynamically at runtime, often via reflection in some code you have no control over. As above, the method might actually exist but the types and/or order of arguments passed to it have changed between versions, look at the detailed error message for more information.
• java.lang.NoSuchFieldError — if an expected field isn’t present in the version of the dependency that has been resolved.
• java.lang.ClassNotFoundException — if an expected class can’t be found dynamically at runtime when using something like Class.forName(). The offending call often occurs “under the hood” in some other artifact that you probably have no control over and may refer to a class that is present in a different version of the dependency.
• java.lang.NoClassDefFoundError — if an expected class definition cannot be found on the classpath at runtime. This is similar but subtly different to the ClassNotFoundException as the class may have been present at compile time but was in a version of a dependency scoped as “provided” that is not present at runtime.
• Various other child classes of java.lang.LinkageError — there are many weird and wonderful ways in which classes can change in a non-backward compatible way, this parent class captures most of them.

Finding the root cause

“For the love of different versions is the root of all evil.”

There are a number of things you can do to try and figure out where the actual problem is coming from. Ultimately you are looking for a class that is being used at runtime which isn’t what the calling code expects. Below are some options to help you determine whether you have a version clash, where the problematic class is being loaded from, and the version of the dependency that contains it.

Maven Dependency Plugin

The first port of call if you like using the CLI is the Maven Dependency Plugin. This provides a number of useful goals but how to use them can be a bit overwhelming at first. It’s definitely worth reading the documentation to understand what they all do. I find the most useful goals for these kinds of problems are:

• dependency:tree — to get an overview of all the dependencies and how they are related
• dependency:copy-dependencies — to dump all the artifacts to a folder for further inspection

IDE tooling

Many IDEs have built-in functionality or plugins which provide a UI of some form over the Maven Dependency Plugin described above. This section covers Eclipse but many other IDEs offer something similar. Eclipse is pretty good at providing a visual and textual way to explore the dependencies in a Maven project, including transitive dependencies. Generally, if you know the name of the dependency that the problematic class is in you can do the following:

1. Click the “dependency hierarchy” tab.
2. Type the name of the dependency in the search “filter” in the top right.
3. The hierarchy view on the left will now show All The Things which depend on this artifact and you can usually spot problems quickly if vastly different versions of the dependency are used by either your project or upstream dependencies (or both).
4. The “effective POM” view can also be useful to force Maven to show you how it merges all the various parent POMs and dependencies into a single POM which you can then scan to see the final resolved versions of all dependencies.

The screenshot below shows an example of a project with multiple dependencies on different versions of Guava:

Here comes trouble — multiple major versions of Guava

No less than four different major versions of Guava — that’s going to be a lot of fun to sort out.

Code Source Location

If you are able to modify the application’s code before the problem occurs you can ask Java’s class loader to tell you the full location of where it is loading the problematic class from and then log this. For example, if we were trying to figure out which version of the CoreMatchers class was being used and where it was loaded from we could do this:

System.out.println(CoreMatchers.class.
                   getProtectionDomain().
                   getCodeSource().
                   getLocation());

which in turn would print something out like:

file:/some/path/to/1.9.0/mockito-all-1.9.0.jar

This may already provide you a clue as to what is going on if the output shows an unexpected version in the jar file name or perhaps you see the class being loaded from an uber jar which you didn’t realise contained a duplicate version of the class in question.

Verbose Class Loader

If you don’t have the ability to change the code as described above, but you are able to pass flags to the javacommand that is running the application, then you can run Java’s class loader in verbose mode by passing it the argument -verbose:class. This will then log the loading and unloading of every class at runtime. The downside of this approach is that you’ll have a lot of log output to trawl through to find your issue but some judicious use of grep could help.

So, for example, if you run the java command like so:

java -verbose:class com.your.package.TestApplication

You will then see lots of output along the lines of:

[Loaded java.security.UnresolvedPermission from /usr/lib/jvm/jre/lib/rt.jar]
[Loaded java.security.BasicPermissionCollection from /usr/lib/jvm/jre/lib/rt.jar]
[Loaded com.expediagroup.dataplatform.TestApplication from file:/path/to/workspace/test-project/target/test-project-0.0.1-SNAPSHOT.jar]

You can then search this for the class which you suspect is causing the problem and see exactly where it is being loaded from.

Solutions — include, exclude, relocate or hack

“Do not blame Maven. Find a solution.”

Once you have confirmed that the root cause of the problem is a mismatch of different versions of a dependency there are a number of possible solutions. In choosing your approach I recommend you aim to:

• Find the magic combination of dependencies and versions that align and don’t conflict. Unfortunately, it can sometimes happen that making a change fixes the initial problem but introduces a new one so a fair amount of trial and error might be required.
• Be explicit about the dependencies you want rather than relying on implicit dependency resolution of transitive dependencies. Being explicit may make your POM file more verbose but that’s the price you pay.
• Choose the latest version of the dependency and make whatever changes necessary to get everything else using this. Using the latest and greatest is good for a number of reasons — security, features, bug fixes etc. If you’re going to spend time making changes to get things working you might as well have the end result be the newest and shiniest so you get those additional benefits. Of course, this isn’t always possible and you may have to compromise on an earlier version, especially if you have other dependencies out of your control which in turn require older versions of the problematic dependency.
• Reduce the cycle time from making a change to testing whether it fixes the problem. For cases where the issue can only be reproduced when deployed remotely, this might mean writing a few simple shell scripts to do a mvn package and then scp the artifacts elsewhere. Don’t go overboard but the time spent here can save you a lot of time wasted doing manual repeated steps.
• Make one change at a time and test the situation that reproduces it. If this fixes the issue then run as many other tests as possible to check you haven’t inadvertently broken something else.

Now that you know what’s broken it’s time to move on to the options you have at your disposal in order to convince the Universe to align the dependencies in a way that solves the problem. These are covered below in an order going from simplest to the most complex, I hope for your sake you don’t have to try all of them.

Add an explicit dependency

If your POM doesn’t already have a direct dependency on the problematic dependency you should do this and explicitly set the version number rather than have Maven’s dependency mechanism choose this based on various transitive version clashes. This will force a specific version to be used and will override any other transitive versions. If everything else plays nicely with this version then you’re good to go. Often it’s not that simple and you may need to explicitly change the versions of various other dependencies and try to get them to all line up in a way that they work with the version you have chosen. Note that this solution probably won’t help you if you have a runtime classpath which contains versions of dependencies that you have no control over. In this case, you might need to use shading and relocating (discussed later) instead.

For example, to force a particular version of Guava that you might have been inheriting as a transitive dependency before, add a direct dependency to a specific version of it in your POM:

<dependency>
  <groupId>com.google.guava</groupId>
  <artifactId>guava</artifactId>
  <version>30.1.1</version>
<dependency>

Exclude the problematic dependencies

If you depend on an artifact that has a dependency on a problematic version of another artifact you may be able to exclude this dependency and remove it from the picture. Remember that you can use “*” wildcards to remove multiple artifacts that have the same Maven groupId which can make things less verbose.

For example, to exclude a problematic version of Guava that the hive-metastore (which this project requires) depends on, do something like this:

<dependency>
  <groupId>org.apache.hive</groupId>
  <artifactId>hive-metastore</artifactId>
  <version>${hive.version}</version>
  <scope>provided</scope>
  <exclusions>
    <exclusion>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
    </exclusion>
  </exclusions>
</dependency>

Modify the versions in upstream projects

If you have control over upstream projects that you depend on and these are causing problems you may be able to make changes in them to use the versions that will work downstream. Often this happens when upstream projects aren’t kept as up to date with newer versions as your own project. So now you get to be a good girl guide or boy scout and drag them into the future by updating them to use the newer versions that will work for you. Of course, this might introduce all kinds of collateral damage in the upstream projects that you now need to fix and your problem has now grown. I’m sorry for you if this is the case but console yourself that you are doing The Right Thing.

Shading and relocating problematic classes

If an upstream project depends on a version of an artifact that your project also depends on, and you are unable to make changes to the upstream project, then you should consider using the Maven Shade Plugin. This is used to shade (i.e. include) and relocate the classes of the dependency directly into your project’s artifact. This is done by renaming the upstream classes (and updating all imports etc.) so that the class names won’t conflict with the original names. This will allow you to use whatever version of the dependency you want without worrying about other versions of it that may be present, this is especially useful if you have no control over the runtime classpath. Note that if reflection is used to load classes from the problematic dependency or class names are passed in via configuration then this might cause issues - relocation changes the class names so these mechanisms will potentially incorrectly load the classes from the non-relocated version of the dependency. Good luck with that.

As an example, to use a different version of Guava than what is available on the runtime classpath you can shade and relocate the Guava packages into your own artifact’s jar file with a new base package name by using a prefix such as com.eg.shaded.com.google.* instead of com.google.*. This can be achieved like so:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>3.1.0</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>        
        <artifactSet>
          <includes>
            <include>com.google.guava:guava</include>
          </includes>
        </artifactSet>
        <relocations>
          <relocation>
            <pattern>com.google</pattern>
            <shadedPattern>com.eg.shaded.com.google</shadedPattern>
          </relocation>
        </relocations>
      </configuration>
    </execution>
  </executions>
</plugin>

If the framework builders of various services that allow you to deploy your own code within them understood how to do proper classpath isolation you wouldn’t have to resort to such trickery. Sadly the developers of Hadoop, Hive, Spark etc. didn’t learn the lessons from how J2EE and Java Servlet containers did a much better job of this, years before them.

“Those who cannot learn from history are doomed to repeat it.”

Building your own shaded and relocated version of upstream jars

If you find yourself in a situation where you have multiple upstream dependencies (let’s call them project-a and project-b) which in turn require different, clashing versions of the same dependency (library-d) and you’re unable to modify project-a or project-b then you’re in trouble. If you use project-a’s version of library-d then project-b throws an exception and vice versa. I’m sorry that it’s come to this but there is a possible way out. You need to pick which version of library-d you’d prefer to keep as a dependency (possibly the newest version) and remove the other version from the picture. In this case, let’s say you decide that you want to keep project-b’s version. What you can do is create a Maven module/project which repackages project-a (let’s call this project-a-relocated) and also shades and relocates the version of library-d. This will produce a new artifact that you can depend on instead of project-a.

For example, let’s say we have project-a which depends on version 11.0.1 of Guava:

<dependency>
  <groupId>com.google.guava</groupId>
  <artifactId>guava</artifactId>
  <version>11.0.1</version>
</dependency>

and project-b which depends on version 16.0.1:

<dependency>
  <groupId>com.google.guava</groupId>
   <artifactId>guava</artifactId>
   <version>16.0.1</version>
</dependency>

and your project-c which depends on both of them:

<dependency>
  <groupId>com.expediagroup.dataplatform</groupId>
  <artifactId>project-a</artifactId>
  <version>1.0.0</version>
</dependency>

<dependency>
  <groupId>com.expediagroup.dataplatform</groupId>
  <artifactId>project-b</artifactId>
  <version>1.0.0</version>
</dependency>

Let’s say that you now use the advice from earlier and add exclusions or explicit dependencies to force the choice of Guava 11.0.1 but the code from project-b then fails as it can’t use this earlier version of Guava. If instead, you force the choice of Guava 16.0.1 then code from project-a fails as it can’t use this later version of Guava. So what you can do now is create a new Maven project called project-a-relocated to keep all of project-a’s classes “as is” but to relocate its usage of Guava like so:

<artifactId>project-a-relocated</artifactId>
<version>1.0.0</version>
<dependencies>
  <dependency>
    <groupId>com.expediagroup.dataplatform</groupId>
    <artifactId>project-a</artifactId>
    <version>1.0.0</version>
  </dependency>
</dependencies>
<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <version>3.1.0</version>
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>shade</goal>
          </goals>
          <configuration>
           <artifactSet>
              <includes>
                <include>com.google.guava:guava</include>
                <include>com.eg.dataplatform:*</include>
              </includes>
            </artifactSet>
            <relocations>
              <relocation>
                <!-- only relocate guava classes -->
                <pattern>com.google</pattern>
                <shadedPattern>
                com.eg.dataplatform.shaded.com.google
                </shadedPattern>
              </relocation>
            </relocations>
          </configuration>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

And now in your project instead of depending on project-a, you depend on the output of the above, i.e. project-a-relocated:

<dependency>
  <groupId>com.expediagroup.dataplatform</groupId>
  <artifactId>project-a-relocated</artifactId>
  <version>1.0.0</version>
</dependency>

<dependency>
  <groupId>com.expediagroup.dataplatform</groupId>
  <artifactId>project-b</artifactId>
  <version>1.0.0</version>
</dependency>

Now you will have Guava 16.0.1 used by project-b while project-a will instead be using the shaded and relocated version of Guava 11.0.1 which has effectively been removed from the equation. I never said it would be succinct or beautiful but I have seen this work.

Note that you don’t have to use separate projects, you can do something very similar using modules in a multi-module Maven project if you’d prefer to keep the lengths you’ve had to go to in order to get this all working a bit more self-contained.

Change the classpath ordering

If you aren’t able to make any code or packaging changes there is a workaround that might solve the problem. This is a bit of a hack and may only kick the can down the road long enough for you to escape and let someone else face the issue later. If the dependency issue can be solved by favouring just one version of the artifact over another and you are able to modify the classpath of your application then you may be able to make some changes to this to get that version loaded before (or instead of) any others. I would generally recommend fixing the problem earlier in your build and deployment pipeline but sometimes extreme measures like this are necessary.

The horrors of a real world example

“On the rocks of reality, dreams get smashed.”

We’ll end this off with a real world example that used most of the above advice with an extra surprise due to the horror show of an upstream project shading some of its dependencies but then not relocating them. When this happens you may see exceptions caused by certain classes but you will have no idea that they are even present as the Maven dependency mechanism isn’t able to see or report back to you on them. Instead, you will have to unpack and inspect the contents of the dependencies’ actual jar files. Join me now for this final tale of woe.

For this example, we have an open-source project called Beekeeper which has a module that is deployed as a Spring Boot application. On startup, it threw up about 200 lines of nested stack traces in the log file but the final three lines pointed to the problem:

Caused by: java.lang.NoSuchMethodError: 'java.lang.String javax.servlet.ServletContext.getVirtualServerName()'
    at org.apache.catalina.authenticator.AuthenticatorBase.startInternal(AuthenticatorBase.java:1178) ~[tomcat-embed-core-9.0.16.jar:9.0.16]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.16.jar:9.0.16]

So it looks like Tomcat’s AuthenticatorBase is not happy with the version of the Servlet API’s ServletContext class., specifically it appears to be missing the getVirtualServerName() method. So we might suspect that we have different versions of the servlet-api dependency on the classpath via various transitive dependencies. The first thing we look for is multiple servlet-api dependencies with different versions so that we can use the advice from earlier and add some exclusions or explicit dependencies on a certain version to get this working. So we use the Maven dependency plugin to take look and this is what happens:

mvn dependency:tree | grep servlet-api
...
[INFO] |  |  +- javax.servlet:servlet-api:jar:2.5:compile

What?!?! There is only one version of the servlet-api dependency on the classpath so how can there be a version clash? We could try to explicitly depend on a different version of the serlvet-api but there are so many, which one to choose? That’s a lot of trial and error.

The battle hardened among us now wonder whether there could be another version of the ServletContext class sneakily hidden away inside some other dependency in order to totally ruin our day. Let’s take a look by dumping all the dependencies’ jar files to a folder:

mvn dependency:copy-dependencies -DoutputDirectory=target/output

Now let’s scan all those jar files for classes named ServletContext. There are various visual tools you could use to do this or you can use some bash magic along the lines of the following:

for i in $(find . -name "*.jar"); do jar tvf $i | grep "javax/servlet/ServletContext.class" && echo $i ; done

4974 Mon Feb 04 16:30:46 GMT 2019 javax/servlet/ServletContext.class
./tomcat-embed-core-9.0.16.jar
1429 Wed May 10 14:20:30 BST 2006 javax/servlet/ServletContext.class
./servlet-api-2.5.jar

Hang on there, this shows that the ServletContext class is actually present in two of the jar files —“servlet-api-2.5.jar” (which is to be expected) and “tomcat-embed-core-9.0.16.jar” (which at first glance is a surprise). What is happening here is that Tomcat has decided it will only work with a certain version of the servlet-api dependency. It’s not immediately obvious which version as they haven’t included any metadata in their jar file (tsk, tsk) but some internet searches reveal it to be version 4.0. To ensure that this version is used they have decided to include the servlet-api classes inside their jar file so if you have any other version of the servlet-api on your classpath you run the risk of a clash. This is what happened here as the 2.5 version is obviously older than 4.0 and is not forwards or backwards compatible. The runtime classloader has also decided to be cruel and use its “break all the things” class ordering mode to put the older servlet-api jar file higher up on the classpath so that it gets loaded before tomcat-embed-core which breaks everything. So, what can we do? If we run this again

mvn dependency:tree

we can look at the hierarchy to see where the transitive dependency on the 2.5 version of servlet-api is coming from (output trimmed to highlight the key information):

org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.1:compile
 +- org.apache.hadoop:hadoop-yarn-common:jar:2.8.1:compile
    +- javax.servlet:servlet-api:jar:2.5:compile

So the embedded Tomcat doesn’t work with the older 2.5 version of servlet-api that hadoop-mapreduce-client-core transitively depends on. We are lucky in this case in that how we are using hadoop-mapreduce-client-core doesn’t require the servlet-api classes so we can simply go with an exclusion:

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-mapreduce-client-core</artifactId>
  <version>${hadoop.version}</version>
  <exclusions>
    <exclusion>
      <groupId>javax.servlet</groupId>
      <artifactId>servlet-api</artifactId>
    </exclusion>
  </exclusions>
</dependency>

and we’re done. The version numbers now align, a rainbow comes out from behind the clouds and the application starts working again.

In closing, I hope that all your dependency issues will be simple to solve and if not, then hopefully the guidance above will prove useful and save you some pain and misery.

I’ll leave you with a modified version of a quote from Robert M. Pirsig whose “Zen and the Art of Motorcycle Maintenance” inspired the title of this article:

“The place to improve the world’s dependencies is first in one’s own POM, and then work outward from there.”

and then an unmodified quote which may be of use to help you persevere in the face of an especially complicated dependency resolution problem:

″We just have to keep going until we find out what’s wrong or find out why we don’t know what’s wrong.”

https://lifeatexpediagroup.com/