Monorepo support with CircleCI

While CircleCI has no direct/inbuilt support for monorepos, setting up support using the various inbuilt commands and the API is possible.

However, the setup needed to achieve that requires a lot of boilerplate code, and for someone to maintain this code. This is where the Swissknife command “trigger workflows” can make working with monorepos a lot easier.

Let us first break down what we mean by supporting a monorepo and what we expect to happen. The following are key features needed when we say monorepo support

• The ability to run only what is modified: If a repo has client and server code when client code is modified only client tests should run.
• One module should not affect another: If we set up a CD pipeline for the client when the server tests break, the client flow should be uninterrupted.

While a monorepo could mean a lot more things, the above two requirements are the basics and this is what Swissknife makes easier.

The way the workflow is structured is as follows:

• Determine what has changed in a given branch compared to the base branch.
• Run tests for modules that have changed.
• If on the base branch (usually master), optionally choose between running last commit, or everything in the repo.

In CircleCI terminology this works by looking at all the modified files for each branch and triggering workflows for each of the modules modified. You make workflows one for each module in the repo and each of them is set to not run by default

client-tests:
  when: << pipeline.parameters.run_client_workflow >>
  jobs:
    - run:
        name: Run tests
        command: ./run_tests.sh

An example of a workflow for client tests as described above. This workflow runs only if the pipeline parameter run_client_workflow is set to true (defaults to false)

Once you have workflows for each part of your monorepo, Swissknife has a command that does the rest. The command takes in an input mapping between directories and workflows and then runs the workflows according to files modified. An example of the input map is seen below

[
  {
    "regex": "^(clients|shared).*",
    "param_name": "run_client_workflow"
  },
  {
    "regex": "^(server|shared).*",
    "param_name": "run_server_workflow"
  },
  {
    "regex": "^(operations).*",
    "param_name": "run_operations_workflow"
  }
]

For each module, the regex defines the directories/files which if modified means the module needs to be tested. The pipeline parameter when sets to true runs the specific workflow.

The Swissknife command trigger-workflows-for-all-modified does the job of triggering the right workflows.

Triggering the right workflows in a monorepo

An example monorepo set up with CircleCI can be seen here.

The swissknife command has built-in support for a variety of edge cases as documented in the orb, however, there are some caveats which are limitations that cannot be fixed without additional support from CircleCI

• All triggered workflows don’t show the commit message in the UI since CircleCI, doesn't support triggering workflows on a branch at a specific commit.
• All triggered workflows are run by the user whose Circle token is passed to Swissknife, rather than the person who authored the commit.

Despite these limitations, this approach works pretty well and makes working with large repositories easy with CircleCI.

Keep watching this publication or follow us on Twitter at @swissknifedev, and learn more about how you can optimize your build pipelines.