Speeding up a Maven build on CircleCI

How I reduced my Maven+NPM+Bower+GAE build time down to 8 minutes flat


My company’s codebase consists of several different repositories. We develop two native apps, with a cross-platform library supporting both, powered by a Google App Engine Java backend, which also hosts our website and a rich Single-Page App for our users to create content. We’ve gone back and forth regarding repository size and structure, alternately consolidating these repositories and then breaking them back up again. We’ve tried git subtree, git submodule, and everything in between — at the moment we’ve settled on git submodules, which we use sparingly. We make it work by keeping repositories large enough to prevent some of the hassles that come with managing submodules.

Unfortunately, as our repository has grown in size and complexity, our builds have gotten longer and longer.

Maven as master delegator

We use Maven as our master build system, mainly due to its Google App Engine Java support, but we’ve crammed a lot of other stuff in there along the way. At this point, our Maven build does the following:

  1. Compile & test our Java back end
  2. Build our Grunt-based SPA (our company homepage)
  3. Build our other Gulp-based SPA (our lightshow editing tool)
  4. Test our hybrid app’s HTML/JS code
  5. Package the whole thing up into a WAR file
  6. Deploy to Google App Engine
  7. (stable branch only) Switch production server to new build

We use CircleCI for all our continuous integration needs, and we love it due to its simple GitHub integration, configurability, iOS/Android support, and amazing customer support team.

Through the process described below, we reduced our build time from 14 minutes to 8 minutes.

1. Compiling & testing our Java back end

Getting Maven dependencies into the Circle cache

One of the first things I noticed when investigating our long build times, was the huge amount of data downloaded during the build. I thought Circle had a caching feature that would prevent this, but I learned that it’s a phase / timing issue. As you can see below, Circle saves its cache before Maven actually downloads anything.

I’m not sure what mvn dependency:resolve does, but it definitely doesn’t download jars.

Luckily, Maven has support for “going offline,” presumably before heading to the beach or getting on a long flight.

dependencies:
override:
- mvn --fail-never dependency:go-offline || true

This seems to help, though I had to add those command-line arguments to keep it from breaking the build due to some weird App Engine missing files. And you can tell from the output that it’s missing a lot of jars. This part is a work in progress.

Speeding up Java test execution

I do believe we have some easily parallelizable tests involving audio processing (yes we do audio processing on App Engine :). I may also try using CircleCI’s built-in test parallelization support, which seems possible with some manual intervention.

2. Building our Grunt-based single-page app

There are a few options for calling Grunt from within a Maven build, but we made the mistake of picking grunt-maven-plugin. I don’t know why, but this plugin makes a full copy of your entire single-page app, with dependencies, on every build. This process only takes 10 seconds or so on my MacBook Pro with SSD, but I have an engineer with a magnetic hard drive and this phase literally takes minutes. For every incremental build.

I plan to switch from grunt-maven-plugin to frontend-maven-plugin, but for now we can’t do that.

One benefit of grunt-maven-plugin (and frontend-maven-plugin) is that it automatically installs a specific version of Node and NPM into a temp folder. This ensures a hermetic build regardless of what is installed on the system.

The downside is that Circle doesn’t cache the Node and Bower dependencies — many megabytes of data — which means they need to be downloaded from the Internet on every build. This is time-consuming and flaky. Why won’t Circle cache this stuff?

  1. Circle only caches specific folders, and the grunt-maven-plugin temp folder is not on that list.
  2. Circle saves the cache after the dependencies phase, but before the test phase, which means grunt-maven-plugin doesn’t even run until after the cache is saved.
Luckily, it’s not too hard to get Circle to cache these NPM dependencies — we simply need to modify circle.yml.

We just tell Circle to run npm install and bower install manually— which is all grunt-maven-plugin does anyway — and we do this before Circle saves its cache. (I can see a potential issue if the system-installed NPM/Node is radically different from the one downloaded by Maven, but for now this seems to work.)

dependencies:
pre:
- cd src/editor && npm install && bower install && cd ../..
cache_directories:
- src/editor/bower_components
- src/editor/node_modules
- src/editor/node

I actually have no way of knowing whether this worked — Circle doesn’t provide any way to browse the cache — but it did speed up the build.

3. Building our Gulp-based single-page app

The process here was very similar to the Grunt section. We wised up and used frontend-maven-plugin for this part of our site, which doesn’t copy tens of thousands of files on an incremental build. However, this plugin uses a similar system as grunt-maven-plugin for installing Node and NPM into a temp folder, and installing NPM and Bower dependencies from there.

The solution here was similar to the Grunt solution above: manually run NPM and Bower during Circle’s dependencies phase.
dependencies:
pre:
- cd src/editor && npm install && bower install && cd ../..
- cd src/main/webapp/homepage && npm install && bower install && cd ../../../..
cache_directories:
- src/editor/bower_components
- src/editor/node_modules
- src/editor/node
- src/main/webapp/homepage/bower_components
- src/main/webapp/homepage/node_modules
- src/main/webapp/homepage/node

Again, I have no way of verifying whether this worked, but the builds got faster and the cache phase started taking a few seconds longer ☺

4. Testing our hybrid app’s JavaScript code

(I did not do anything here, as this part is pretty quick.)

5. Packaging the app into a WAR file

I believe there’s a way to speed this up on developers’ machines, by asking App Engine to deploy its local server from disk, instead of from a WAR package. But I don’t think there’s any way to speed this up on the CI server.

Note to self: Does the App Engine plugin actually use the generated WAR file? Is there a way to skip the packaging phase?

6. Deploying to Google App Engine

Our Circle build automatically deploys each commit to a staging server named after the branch. For example, if we had a branch called newfeature, our Circle build would automatically push a given build to, for example, http://auto-newfeature-2015–04–06.whamcitylights.appspot.com. This is great for our internal development process for a few reasons:

  • Easy to test new changes using production data
  • Easy to ask for internal feedback without requiring entire team to check out the code and start the App Engine dev server
  • Easy to deploy new production builds, especially emergency fixes
Our Circle build automatically deploys each commit to a staging server named after the branch.

Removing the redundant build & test step

For deployment, we use Google’s official appengine-maven-plugin, which allows you to upload and configure the various instances of your production app. It’s really convenient to have this built into our core build process, but there is one downside. The deployment command, mvn appengine:update, does a full build, and even runs your tests.

That wouldn’t be an issue, except that Circle is set up to separate the test and deploy phases — which means our code gets built and tested twice.

Luckily, the solution was pretty easy: just skip Circle’s test phase, and run the tests as part of mvn appengine:update in Circle’s deploy phase.
test:
override:
- /bin/true
- echo '{"credentials":{"ubuntu":{"access_token":"XXX","expiration_time_millis":1418827756487,"refresh_token":"XXX"}}}' >> ~/.appcfg_oauth2_tokens_java
deployment:
newfeature:
branch:
newfeature
commands:
- mvn appengine:update:
timeout:
600

Kudos to Google for allowing passwordless App Engine deployment!

7. Switch production server over to the new build

Google’s appengine-maven-plugin allows easily promoting a given build (aka version (aka staging server (aka branch))) to be the new live production build. All you have to do is run mvn appengine:set_default_version, and your current branch will be deployed to all your users.

We have Circle deploy our stable branch automatically, so what our users see is always the latest build of that branch.

It’s a little risky, but our process ensures that no development happens on the stable branch, and changes are not merged into stable until having been tested on their branch of origin.

The problem here? Presumably due to Maven architectural limitations, set_default_version requires a partial build — which means all that Grunt and Gulp stuff runs, and the app is packaged up as a WAR file — even though it’s totally not necessary to invoke the set_default_version command.

Luckily, Maven gives us all the tools we need to bypass Maven itself — so we just call the Google App Engine command-line tools instead.
deployment:
stable:
branch:
stable
commands:
- ./deploy-stable.bash

And here’s deploy-stable.bash:

#!/bin/bash

# Not sure why this isn't executable in the first place
chmod +x ~/.m2/repository/com/google/appengine/appengine-java-sdk/*/appengine-java-sdk/appengine-java-sdk-*/bin/*.sh

# Run appcfg
~/.m2/repository/com/google/appengine/appengine-java-sdk/*/appengine-java-sdk/appengine-java-sdk-*/bin/appcfg.sh \
--oauth2 set_default_version target/server-1.0-SNAPSHOT/

Voilà! Now the set_default_version command takes all of 4 seconds ☺


Conclusion

I’m proud to have nearly cut our build time in half. Here’s a summary of what I learned:

  • Google App Engine’s Maven integration is great, but it is very conservative. If you try to shoehorn this into Circle’s “phase” system, your build ends up doing a lot of redundant work.
  • CircleCI’s caching mechanism seems like a powerful way to speed up a build, but it’s useless for a Maven build unless you intervene. You need to alter circle.yml to ensure dependencies are being downloaded and cached at the right time.
  • If you use NPM and/or Bower, and you don’t store package.json and bower.json in your repository root, you must manually run NPM and Bower during Circle’s dependencies phase.

Next steps: the march to a 4-minute build

8 minutes is fine, but it’s not enough! Here are a few of my ideas for how to cut the build time even further:

  • Switch from grunt-maven-plugin to frontend-maven-plugin to prevent all that unnecessary file copying.
  • Run the entire build in Circle’s dependencies phase, so all the Maven and NPM and Bower files are stored in the Circle cache.
  • Parallelize our Java unit tests.

Stay tuned!

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.