Speed up your Multi Module Maven Builds with turbo-maven-plugin
Fast feedback makes us happy. So if you are only looking for how to speed up your multi module build as fast as possible, go straight to turbo-maven-plugin.
If you want to know more about why, and also how turbo-maven-plugin works, please keep on reading.
The reason we are happy when we get fast feedback, is that it triggers the production of dopamine in our bodies. Dopamine is a happiness drug we can give ourselves for free. This is a smart thing to do as often as possible — it makes us happy.
In addition to making us happy, fast feedback makes us deliver value faster. This not only feels good, it is also good for our team and the place we work.
When working with Kotlin and Java, feedback from building our apps, is something we need often. Where I work, we code and run our tests in IntelliJ. As soon as we want to move the change to production, we normally build the app locally with Maven before pushing the code, to see that the tests run as they should, and that everything is working fine.
A Multi Module Maven repository with 250 applications
We have our 250+ apps in a monorepo. The monorepo is one multi module Maven repository, where everything is on head all the time. When we build our app, we build both the app itself, and all the modules that it depends on.
This is why it is important for us to build smart.
A typical app builds and runs all tests for itself and its dependencies in 2–3 minutes. This is a long time to wait, so we started looking for a way to get faster feedback.
We quite often have a code change in a module the app depends on.
It is possible to ask Maven to build only the modules that we want, using the — projects <list of projects to build> argument, and building from the root pom. This is faster than building all required dependencies with — also-make.
Using a maven command with — projects requires both mental capacity and finger acrobatics on the command line, so we seldom do this. We rather build with variants of cd in and out of modules and mvn clean install, hoping that we have built everything that needs to be built. Or we build everything to be sure.
Building only what needs to be built
We only want to build what needs to be built, without having to hand code a special Maven command for every change we do. Both Bazel and Gradle knows how to do this.
There are several strategies here, and all we have seen, is based on analysing what files have changed, then making scripts or programs calculating what modules the changes reside in, and then create a Maven command that builds these modules.
We have created a Maven plugin helping us with just that. It is called turbo-maven-plugin.
How does turbo-maven-plugin work?
turbo-maven-plugin is based on the same strategy, that is analysing what modules have changes in their source code, and then build only these modules, and the modules depending on them.
For every module, the plugin looks for a file containing one row per source code file in the module. A row contains the name of the source code file and a checksum of the contents of the file.
If it doesn’t find a file for a module, it creates it, and puts it in the the module’s directory in the local m2 repo. It does this for all modules that is required for building the app, and also the app itself.
If we build again, without changing anything, nothing will be built, since all the checksums are the same.
If we do a change, the plugin will first get the complete list of modules that needs to be built from the Maven Reactor. This is the the app itself, and all its dependencies. For each module, it compares the checksum for each source code file in the module, with the checksum in the file in the m2 repo. If the checksums are the same, the plugin removes the module from the list.
For the modules that have changes, we make sure we also add the modules that are dependant on them. In pseudo code, it looks like this:
//Find changed modules:
modulesToBuild = modulesFromMaven.filter(isModuleChanged())//Find the modules dependent on the changed modules:
modulesToBuild.forEach(module -> downStreamProjects.add(module.getDownstreamProjects()))//Return the distinct set of modules to build:
return modulesToBuild.addAll(downStreamProjects).removeDuplicates()
How do we use the turbo-maven-plugin?
The plugin is defined in our root pom, and is disabled by default, so that Maven behaves normally for everyone when using regular Maven commands:
<plugin>
<groupId>no.sparebank1</groupId>
<artifactId>turbo-maven-plugin</artifactId>
<version>${turbo-maven-plugin.version}</version>
<extensions>true</extensions>
<configuration>
<enabled>false</enabled>
<ignoreChangesInFiles>swagger.json</ignoreChangesInFiles>
<alwaysBuildModules>distribution</alwaysBuildModules>
</configuration>
</plugin>We have a tool, that really is just a structured collection of scripts, called bob. When we want to build an app, we run bob mvn build from the app root. This command actually does this:
mvn -T4 -f <path-to-the-root-pom> --projects <path-to-the-app-pom> --also-make -Dturbo.enabled=true clean installBut that is something our developers don’t have to think about.
With this, we have cut the average app build time in half, from 2–3 minutes to 1–2 minutes. We have also reduced the cognitive load of our developers. They don’t have to think about what modules need to be built anymore. They just run bob mvn build, and Maven and maven-turbo-plugin take care of the rest.
If you want to try the plugin, it is on Maven Central, and you find both source code and pom configuration on the turbo-maven-plugin’s home page.
