In modern Java web development, the thought of packaging and running applications in anything other than a fat JAR is almost becoming heretical. However, there can be distinct disadvantages to building and deploying these large files. The HubSpot engineering team has created a fantastic blog post that explains some of the challenges they were having with their inner development loop and the use of network bandwidth when deploying large fat JARs continuously to the AWS cloud.
The blog explains how the team initially used the Maven Shade plugin to build and package applications, but this was turning an application with 70 class files — which totaled 210 KB in the original JAR, containing no dependencies — into a 150+ MB-sized fat JAR. Using Shade to combine 100,000+ files into a single archive was also a slow process, and then as the build server copied and deployed the resulting JAR to and from the AWS S3 storage service, this consumed both time and network resources. The situation was magnified by the fact that the HubSpot team has 100 engineers that were constantly committing and triggering 1,000–2,000 builds per day; they were generating 50–100 GB of build artifacts per day!
The HubSpot team ultimately created a new Maven plugin: SlimFast. This plugin differs from the Shade plugin, in that it separates the application code from the associated dependencies and accordingly builds and uploads two separate artifacts. It may sound inefficient to build and upload the application dependencies separately, but this step occurs only if the dependencies have changed. Because the dependencies change infrequently, the HubSpot team states that this step is often a no-op; the package dependencies’ JAR file is uploaded to S3 only once.
The HubSpot blog post and corresponding GitHub repository provide comprehensive details, but in essence, the SlimFast plugin uses the Maven JAR plugin to add a Class-Path manifest entry to the Skinny JAR that points to the dependencies’ JAR file, and generates a JSON file with information about all the dependency artifacts in S3 so that these can be downloaded later. At deploy time, the HubSpot team downloads all of the application’s dependencies, but then caches these artifacts on each of the application servers, so this step is usually a no-op as well. The net result is that at build time only the application’s skinny JAR is uploaded, which is only a few hundred kilobytes. At deploy time, only this same thin JAR needs to be downloaded, which takes a fraction of a second.
The SlimFast plugin is currently tied to AWS S3 for the storage of artifacts, but the code is available on GitHub, and the principles can be adapted for any type of external storage.