Nx Affected + CircleCI Parallelism = Faster CI/CD Pipelines
In this post, I’m going to tell our story of how we made our CI/CD pipelines faster for our Nx monorepo by using Nx affected with CircleCI parallelism (and how you can, too).
Note: This post assumes that Nx Cloud is not being used.
Background
In the Cisco CPX developer experience team, we are always looking to maximize the productivity of our many globally distributed development teams.
When we first transitioned to an Nx monorepo and CircleCI, we knew that we didn’t want all developers to have to build, lint, and test all projects for every commit in CI/CD. Hence, we leveraged Nx affected so that each CI/CD pipeline only runs tasks against projects that are affected by the developer’s individual changes.
This was a step in the right direction but was still prone to inefficiency because Nx affected would perform tasks against the projects serially in one executor. This means that if any affected project is slow to complete its task, it will hold up the entire pipeline until the task is completed.
These conditions are not ideal, especially when working with projects that contain “opportunistic” (legacy) code that do take their (sweet) time to complete a task. However, while observing this undesirable CI/CD performance, CircleCI happened to present us with a sign…
This made us think, “… what if we split up affected projects across multiple executors? Is this even possible?” As it turns out, with a couple of CircleCI orbs, UNIX commands, Nx commands, and some fist shaking at the screen (well, we did this so that you don’t have to), it is indeed possible and can result in faster CI/CD pipelines.
CircleCI Parallelism — Out of the Box
To understand what CircleCI parallelism is and how it works, see their documentation.
TL;DR:
- Designed to “speed up the testing portion of your CI/CD pipeline”¹
- Tests can be split by name, size, or using timing data
- Configured by:
1. “Specifying a job’s parallelism level”¹
2. “Using the CircleCI CLI to split tests”¹
Right away, we had two questions:
- Does this really only work for tests?
- Are those really the only three ways to split tests?
After many trials and cartwheels, we’re pleased to announce that the answer to both of those questions is no. With that said, let’s first look at step (2) (order does not matter here in terms of execution), “[u]sing the CircleCI CLI to split tests”¹.
A Fourth Way to Split Tests: By Nx Affected Project
Let’s consider a simple monorepo with the following projects and dependencies:
Furthermore, see the following table describing which tasks are configured for each project (note that not all projects are configured to build, lint, and/or test):
As a quick refresher on how Nx affected works… Let’s say that I create a pull request that makes a change to my-lib-a. Based on the dependency graph, my affected projects will be my-lib-a, all of the three apps, and all of the three e2e apps (seven projects total). Based on the tasks configured for each project, we expect build to run for three projects, lint to run for all seven projects, and test to run for four projects.
So, how can we split these affected projects to run in parallel in CI/CD?
After perusing many docs, blog posts, and forum questions online, we found that A guide to test splitting had our answer with the following example:
This example taught us that you can pipe circleci tests split to a list of projects and assign it to a variable. You can then use that variable with your test command and the variable will resolve to each line in the provided list, set up to run in parallel. Furthermore, that test command can be any command that you desire, including build and lint!
Thankfully, we also have more than just nx affected to work with. To get our list of projects to pipe with circleci tests split, we can use nx print-affected.
Note: whenever we use an Nx affected command in CI/CD, we’ll be using the Nx orb to ensure that the commands run properly.
After a few adjustments, we ended up with the following for our monorepo for lint (the same can be done for build and test, just replace the value for —-target):
Bonus: Affected Projects Distributed Across Parallel Executors
You might be wondering, “why use nx run-many?”
This setup also allows for the ability to distribute projects across parallel executors. Let’s say, for example, we have to run lint for our seven affected projects, but we only want three parallel executors. To achieve this, we can pipe circleci tests split to a list like the following:
my-app-a,my-app-b,my-app-c
my-app-a-e2e,my-app-b-e2e,my-app-c-e2e
my-lib-aNote that the list being three lines long tells us that we want three parallel executors.
This list will result in the following CI/CD execution (note that projects specified in the nx run-many command will run serially):
There is a lot of flexibility with affected project distribution across parallel executors. All you have to do is modify the list of projects that will get piped with circleci tests split, and this can be done with additional UNIX command magic and/or a helper script in conjunction with nx print-affected.
However, this being a bonus configuration, the rest of this post will continue to use nx runy-many but assume each affected project will get its own parallel executor.
Wait… What About Step (1)?
As described earlier, CircleCI parallelism is configured by:
1. “Specifying a job’s parallelism level”¹
2. “Using the CircleCI CLI to split tests”¹
We completed step (2), so how about step (1)?
According to the parallelism docs, “[t]o run a job’s steps in parallel, set the parallelism key to a value greater than 1.”¹.
Unfortunately, it’s not that simple for us…
After a lot of testing, we realized that with our setup, the parallelism key has to be set to the exact number of affected projects, and we are unable to calculate and also set that number in our CircleCI configuration. We tried playing around with the --total flag that is documented to see if we can override anything, but we did not get the results that we desired.
So we researched: is it possible to have a dynamic CircleCI configuration?As it turns out, it is possible. We learned that we can “[u]tilize Setup Workflows and the Continuation orb to easily construct dynamic and multi-config pipelines.”². So, let’s see what we can do with this Continuation orb.
Continuation Orb to the Rescue
In short, the Continuation orb allows you to create a setup workflow to do whatever setup you desire and then “continue” into your main workflow. More importantly, this setup workflow can also pass parameters into your main workflow, which could be our ticket to calculating and setting that parallelism value. So, on a high level, we came up with the following:
With the Continuation orb, we have two configuration files. The first configuration file, config.yml, will do our setup tasks and then “continue” into our second configuration file, continuation.yml, which will run our CI/CD tasks. Setup will retrieve the list of affected projects per task (because not all projects build, lint, and/or test) and also count how many affected projects there are per task so that we can use that value for the parallelism key. Setup can then write the list and number of affected projects to a JSON object and pass that as parameters via Continuation orb so that continuation.yml has those values available.
The (simplified) YML will look like the following for lint (see comments for more details):
config.yml
continuation.yml
The Final Recipe
So, we completed (1) and (2) (in reverse order 🤸♂️) of configuring CircleCI parallelism. Now, let’s tidy things up, get this all together with a pretty red bow on top, and have ourselves a complete, clean, and fast CI/CD pipeline running.
First and foremost, when trying to use the Continuation orb for the first time, you might see the following error in CircleCI:
To fix this, enable the dynamic config option in your project settings:
Next, we take our aforementioned, simplified examples and extend them into full configurations… Which we already did for you in this demo project 😉. In this demo project, we also configured the CI/CD tasks to check if there are any affected projects to begin with; if there are none, no need to run anything! The key files to peek at are .circleci/config.yml and .circleci/continuation.yml.
Finally, with all of this in place, we see the following in CircleCI (in this case, when making a change in my-lib-a):
And when clicking into workflow “build-lint-test”, and then into the lint job, expecting seven projects to be affected…
…we have seven parallel executors, one for each affected project (we see that the first executor ran lint against my-app-a).
Conclusion
In this post, I walked you through the journey we underwent to make our CI/CD pipelines faster for an Nx monorepo by using Nx affected with CircleCI parallelism.
The most important takeaway from this setup is that the time to complete a job in CI/CD will ultimately be the longest time it takes to complete that task for an affected project. In the above screenshot showcasing the lint job with seven parallel executors, the total time to complete the job was 25s, primarily because executor #4 took 24s to complete.
Furthermore, this setup is flexible in the sense that you can distribute your affected projects across parallel executors however you’d like!
While a simple monorepo, like the example in this post, might not benefit much from this complex CI/CD setup, for a monorepo that has hundreds of projects, maybe some containing “opportunistic” code, the overall time to complete CI/CD pipelines can be reduced significantly. As evidence, when we first introduced this setup in Cisco CPX, here’s how much faster our pipelines got:
Overall, an up to 75% reduction in time to complete a job in CI/CD! Although we have more work to do to further improve these times, we’ll take 11 minutes to complete a job over 47 minutes any day.
Once again, a working example of the Nx monorepo and CircleCI setup mentioned in this post can be found in this demo project (last updated with Nx 14.7.6).
Thank you to everyone in Cisco CPX and CircleCI that helped make this all happen, and thank you for reading this post! 🎉
References
[1]: CircleCI. Testing splitting and parallelism https://circleci.com/docs/parallelism-faster-jobs
[2]: CircleCI. CircleCI Developer Hub — circleci/continuation https://circleci.com/developer/orbs/orb/circleci/continuation
