Monorepo with CircleCI Conditional Workflows

UPDATE:

There is a new experimental script that allows conditioning workflows on specific path changes using advanced pathspec. See v2 for more details.

Monorepos bring simplicity to the development process, but raise the complexity of automated builds and deploy.

When we first started migration to monorepo in a large project that had over 12 repositories, we faced the challenge of configuring CircleCI so that it still provides a fast feedback. Each time a pipeline was triggered, the CI started to build all services taking a lot of time.

There is already a good article by Reuven Harrison on how you can trigger CircleCI jobs only for services that were changed. That works well for simple services, where you can describe all the steps in a single job. However, if you need a more sophisticated workflow for each service in your monorepo, you need a way to trigger entire workflows.

As of June 2019 CircleCI team released (preview version) their API v2 together with pipeline parameters and conditional workflows. And that’s exactly what we need for monorepos.

Before you continue reading

At the moment of writing of this article, CircleCI API v2 was in Preview release:

The CircleCI v2 API is currently in Preview release. You may use it at will, but this evaluation product is not yet fully supported or considered generally available.

You can check the status of CircleCI API v2 here.

How it works?

CircleCI doesn’t yet support multiple configuration files (which would be nice for monorepos), so you start by creating a .circleci/config.yml where you’ll define all the jobs and workflows for the entire monorepo.

Firstly, define a workflow that will be triggered on every change and will be responsible for triggering other workflows in case of code changes.

version: 2.1
parameters:
  trigger:
    type: boolean
    default: true
jobs:
  trigger-workflows:
    docker:      
      - image: circleci/node
    steps:
      - checkout
      - run:
          name: Trigger workflows
          command: chmod +x ./circle_trigger.sh && ./circle_trigger.sh
workflows:
  version: 2
  ci:
    when: << pipeline.parameters.trigger >>
    jobs:
      - trigger-workflows

Let’s now explain how the above configuration works.

The parameters clause defines custom pipeline parameters that can be provided when making API calls to CircleCI. The trigger parameter is needed to instruct CircleCI to always start (default: true) the workflow that checks for changes, but to be able to skip it when triggering pipeline for other workflows. To start the workflow depending on a pipeline parameter, we use the when clause in the workflow definition.

The circle_trigger.sh bash script performs the following steps:

1. Using the recent builds API, gets the most recent commit SHA for which there was a CircleCI pipeline completed in current branch . In case there were no builds (which is usually the case with feature branches), it attempts to identify the nearest parent branch and then get its most recent build commit SHA.
2. It scans for directories in ./packages to get the list of services in which to search for changes.
3. Using the git log command, it then filters out all the directories in which there are changes since the commit identified in the first step.
4. Finally, it makes a single CircleCI API call to trigger the pipeline. For each service that has changed, it adds a corresponding pipeline parameter with the service’s name (same as the directory name in ./packages). Pipeline parameters are then used by CircleCI to validate which workflows have to be started.

Once you have all above configured you’re ready to start configuring workflows for your custom services.

Configuring the pipeline

Suppose you want to add a new service to your monorepo, that should have a separate CI workflow, with its own build, test and deploy jobs.

You first start by creating a root directory for you service implementation. If your service is called auth you add the directory at ./packages/auth and put there all the code needed to implement the auth service.

Next let’s switch to the .circleci/config.yml configuration file to setup the CI pipeline for auth service. Add a new pipeline parameter with the same name as the directory:

parameters:
  ...
  auth:
    type: boolean
    default: false

Now you can use the parameter to add a custom workflow that will be triggered only when there are changes in ./packages/auth directory.

workflows:
  version: 2
  ...
  auth:
    when: << pipeline.parameters.auth >>
    jobs:
      - build
      - test
      - deploy
          requires:
            - build
            - test

A CircleCI job can be used multiple times. You can define job parameters and then reuse jobs across multiple workflows:

  jobs:
    build:
      parameters:
        package_name:
          type: string
          description: name of the service
      docker:      
        - image: circleci/node
        # Set to the root location of the service
        working_directory: ~/project/packages/<< parameters.package_name >>
      steps:
        - checkout:
            path: ~/project
        ...
workflows:
  version: 2
  ...
  api:
    when: <<pipeline.parameters.api>>
    jobs:
      - build:
          name: api-build
          package_name: api
      - deploy:
          name: api-deploy
          package_name: api
          requires:
            - api-build
  auth:
    when: <<pipeline.parameters.auth>>
    jobs:
      - build:
          name: auth-build
          package_name: auth
      - deploy:
          name: auth-deploy
          package_name: auth
          requires:
            - auth-build

That’s it! Now you can add as many services as you want to your monorepo and CircleCI will build only those that have changed since last build.

A complete working example is published here:
https://github.com/labs42io/circleci-monorepo