TL;DR We were able to reduce build times to 6 minutes (from 17) and test times to 18 seconds (from 1–2 minutes).
At Vendasta, we have many Angular repositories, some large and some small. Over time, our CI/CI pipeline in one of our larger pipelines has slowed down. We were reaching build times upwards of twenty minutes, which was the hard limit of time we added as a safety net to our CI system — so they could have been longer!
20 minutes might seem short for some companies’ build times, but at Vendasta, instead of large releases on a multi-day/week/month cadence, we are shipping small amounts of code to production hundreds of times per day. Even while working on your branch, each commit you make would run a build that would take just as long.
We needed to speed up build times to enable developers to move faster and allow developers to be confident in the build process. It also wouldn’t hurt to save money in Cloud Build charges by reducing build times.
Our current CI/CD Pipeline
We use Google Cloud Build for our CI/CD pipeline, which works great for both our frontend and backend builds since you can specify any configuration. The only problem for our frontend builds is Cloud Build is inherently stateless, meaning from build to build nothing is carried over. That means any of the caching of the Angular build process or Jest test caching is lost between builds.
When building locally, I noticed the build times after running the first build was dramatically faster. I searched around the web to find out what is happening and found almost no results. The documentation for ng build also does not talk about the caching strategy, so I started my investigation.
Searching for some cache 💸
I noticed when you run the
ng build command, after everything is finished running a new directory is created in your
node_modules folder called
.cache with a few subfolders.
I tried removing that
.cache folder then rebuilding the application and, voilà, slow build times are back. Now we know where the cache exists but how do we get that working in Cloud Build since a new `node_modules` is created for every build.
It’s dangerous to go alone! Take node_modules
In our current Cloud Build yaml, we run
npm ci every time we build our projects. If you do not know what
npm ci does, this is a snippet from the docs:
This command is similar to
npm-install, except it’s meant to be used in automated environments such as test platforms, continuous integration, and deployment – or any situation where you want to make sure you’re doing a clean install of your dependencies.
This command is exactly what we want to use in CI but this command also completely removes the previous
node_modules folder which would defeat the purpose of the caching anything within the
node_modules . This is what our current yaml looks like (some steps have been removed):
We decided to cache the entire
node_modules folder. When we move to Angular 9 this would also allow us to cache the
ngcc compilations. (for more information on that https://angular.io/guide/ivy#ivy-and-libraries). Caching the
node_modules folder is usually a bad idea as people running different node versions or different environments can cause problems, but since this is in our CI/CD where we can control each step and version we are using, we were fine with this risk.
Before we can cache
node_modules, we need a way to identify when we should be using the cache or doing a
npm ci . The way we created that check was hashing the
package-lock.json file since that should only be updated when there are new dependencies. This means that if a new dependency is added or the
package-lock is updated in any way, it will not use the cached
node_modules which we found is fine.
As you can see from the above code, we check with the cache first and if we fail to find a cached node_modules we
npm ci and that is it! Simple as that.
Put all the cache in the bag!
After building the application, we need a way to store the new, updated cache. Storing the cache is a simple process using Google Cloud Storage:
As you can see from the above, we only store the cache if we didn’t find a previous cache. This is perfect for non-master builds but on master you most likely want to update the cache every time. So for master remove the line
test ! -f node_modules.tar.gz and it will overwrite the cache.
A little bonus time gain 🎉
If you use Jest or any other test runner for your frontend, you could potentially benefit from also saving the test cache. We use Jest for majority of our frontends, so I am going to only explain how to cache with Jest.
If you add
“cacheDirectory”: “node_modules/.cache/jest” to your Jest config, it will store any test cache in the same place Angular stores its cache. With Jest cached, we have seen some tests go from 60 seconds (!!!) down to 3 seconds.
Make sure your GCS bucket does not have a retention policy, as that will not let you overwrite your cache. You can set the bucket to automatically delete old cache versions after X amount of time to keep costs down.