Monorepo with CircleCI Conditional Workflows
There is a new experimental script that allows conditioning workflows on specific path changes using advanced pathspec. See v2 for more details.
Monorepos bring simplicity to the development process, but raise the complexity of automated builds and deploy.
When we first started migration to monorepo in a large project that had over 12 repositories, we faced the challenge of configuring CircleCI so that it still provides a fast feedback. Each time a pipeline was triggered, the CI started to build all services taking a lot of time.
There is already a good article by Reuven Harrison on how you can trigger CircleCI jobs only for services that were changed. That works well for simple services, where you can describe all the steps in a single job. However, if you need a more sophisticated workflow for each service in your monorepo, you need a way to trigger entire workflows.
As of June 2019 CircleCI team released (preview version) their API v2 together with pipeline parameters and conditional workflows. And that’s exactly what we need for monorepos.
Before you continue reading
At the moment of writing of this article, CircleCI API v2 was in Preview release:
The CircleCI v2 API is currently in Preview release. You may use it at will, but this evaluation product is not yet fully supported or considered generally available.
You can check the status of CircleCI API v2 here.
How it works?
CircleCI doesn’t yet support multiple configuration files (which would be nice for monorepos), so you start by creating a
.circleci/config.yml where you’ll define all the jobs and workflows for the entire monorepo.
Firstly, define a workflow that will be triggered on every change and will be responsible for triggering other workflows in case of code changes.
- image: circleci/node
name: Trigger workflows
command: chmod +x ./circle_trigger.sh && ./circle_trigger.shworkflows:
when: << pipeline.parameters.trigger >>
Let’s now explain how the above configuration works.
parameters clause defines custom pipeline parameters that can be provided when making API calls to CircleCI. The
trigger parameter is needed to instruct CircleCI to always start (
default: true) the workflow that checks for changes, but to be able to skip it when triggering pipeline for other workflows. To start the workflow depending on a pipeline parameter, we use the
when clause in the workflow definition.
circle_trigger.sh bash script performs the following steps:
- Using the recent builds API, gets the most recent commit SHA for which there was a CircleCI pipeline completed in current branch . In case there were no builds (which is usually the case with feature branches), it attempts to identify the nearest parent branch and then get its most recent build commit SHA.
- It scans for directories in
./packagesto get the list of services in which to search for changes.
- Using the
git logcommand, it then filters out all the directories in which there are changes since the commit identified in the first step.
- Finally, it makes a single CircleCI API call to trigger the pipeline. For each service that has changed, it adds a corresponding pipeline parameter with the service’s name (same as the directory name in
./packages). Pipeline parameters are then used by CircleCI to validate which workflows have to be started.
Once you have all above configured you’re ready to start configuring workflows for your custom services.
Configuring the pipeline
Suppose you want to add a new service to your monorepo, that should have a separate CI workflow, with its own build, test and deploy jobs.
You first start by creating a root directory for you service implementation. If your service is called
auth you add the directory at
./packages/auth and put there all the code needed to implement the
Next let’s switch to the
.circleci/config.yml configuration file to setup the CI pipeline for
auth service. Add a new pipeline parameter with the same name as the directory:
Now you can use the parameter to add a custom workflow that will be triggered only when there are changes in
when: << pipeline.parameters.auth >>
A CircleCI job can be used multiple times. You can define job parameters and then reuse jobs across multiple workflows:
description: name of the service docker:
- image: circleci/node # Set to the root location of the service
working_directory: ~/project/packages/<< parameters.package_name >> steps:
- api-build auth:
That’s it! Now you can add as many services as you want to your monorepo and CircleCI will build only those that have changed since last build.