How to 10x your Maven build
Some of how Intuit reduced a Maven build’s time by a factor of ten.
One of the biggest projects at Intuit is a monolith Java application that is a key part of QuickBooks Online. It has nearly 400 Maven modules, more than 27k files and over 5 million lines of code. With between 200 and 400 engineers working on this code on any one day, 30 seconds of build time has a real impact on the teams' ability to deliver.
At one point in time it took 40m to build QuickBooks Online on your local machine, today it can be as fast as 4m.
This post will tell you some of how that was done, so you could apply them yourself.
Profiling Your Build
Maven allows you to load extensions that spy on the build and gather data. Originally this was done using the Takari Maven Profiler, but as we had a large multi-module build, we needed something that would aggregate total times spent by different plugins on different modules.
We needed something like the Gradle Build Scanner. This gives you charts, summarises plugins, and keeps historical data so you can see the impact of your changes.
So we built the Maven Build Scanner to do exactly that.
When you run a build it produces charts that help you understand what is happening:
Why Was Our Build Slow?
Any plugin that runs once per build will add up, even if individual times are small. The totals chart shows that some plugins (such as the source plugin or check-style) were taking longer to run than compiling the code.
A multi-threaded build (i.e. using Maven’s -T1C option) on a project with hundreds of modules well become effectively a single-threaded build if you have a critical path.
Finally, the tests took a long time to run on CI as the Maven test plugin (aka Surefire) was forking the JVM every test.
Disabling Plugins
The analysis identified the following slow tasks that added small times to each module, but a lot of time overall:
- Running the check-style plugin — why not just run this on the changed code?
- Running the Maven enforcer plugin could be run on CI.
- Setting up Jacoco instrumentation on every build — even when tests were not running.
- Installing source jars — for code developers already had on their desktop,
- Downloading source jars — these are larger and take longer than their binary partners, more than 2x download time
These were not just skipped, but actually removed from the build totally as just initializing plug-ins takes significant time.
As this point, build time was dominated by three tasks: compiling code, creating jars, and installing them locally.
Breaking Dependencies to Remove Critical Paths
When you built the project you could hear something in the noise of your laptop fan — full throttle for the first half of the build, idling for the second half. Even though the build was parallelized, it wasn’t actually using all CPU codes. This was because there was a critical path in the build.
In Maven, inter-module dependencies are specified in the same way as your compiler dependencies — by the <dependencies/> section of your POM.
- Remove unused dependencies (also speeds up compilation by reducing disk and net I/O).
- Breaks dependencies between modules so they could be built in parallel.
Maven makes it easy to add dependencies but doesn’t reveal the performance cost of doing so.
Optimizing Test Execution
Unit tests were running using the Maven Surefire Plugin and it was creating a JVM fork for every test. Forking a JVM is very expensive and consumed 95% of the total time test execution time — tests themselves just 5%.
Appropriately configuring the test set-up reduced the test run time from 44m (on a 40 core machine) to 10m (on a 16 core machine) — a 10x improvement.
Optimizing JVM Start-up Time
So how can you speed up JVM start-up time? Quick improvements to the Surefire Plugin set-up:
- “-Xverify:none” disables byte code verification.
- “-XX:TieredStopAtLevel=1” disables expensive profiling of JVMs which will be short lived.
- “-XX:-TieredCompilation” disables tied compilation — especially speeding up JMockit based tests.
These can be set in your Surefire configuration.
Configuring Test Threading
Running Maven tests you have the following options (from fastest to slowest).
If you have a multithreaded build:
- Run tests single-threaded.
- Run tests in a new fork per test.
If you have a single-threaded build:
- Run tests multithreaded (e.g. one thread per two cores).
- Run tests in multiple-forks (e.g. one fork per two cores).
- Run each test in a new forked JVM (e.g. one per two cores).
By refactoring tests to avoid forking, it was possible to run them single-threaded.
