Mastering MonoRepos with CircleCI’s new Dynamic Config

Benjamin Possolo
May 3 · 7 min read

Have you ever wished you could consolidate your microservices, libraries and applications into a single monorepo, but avoided doing so because managing builds and deploys would be too daunting? If so, fret no more…

In this article, I’m going to walk you through setting up CircleCI to analyze file changes within a monorepo, and then trigger a CircleCI pipeline that builds and deploys that specific sub-section of the monorepo. When applied to your own projects, this can dramatically streamline your CI/CD pipelines and reduce costs by selectively executing workflows on the minimum set of modules within your repo.

Keep reading to achieve monorepo nirvana…

Overwatch’s Zenyatta achieving “Transcendence” by Claudio Amoroso

How does it work?

This is now possible thanks to a powerful feature that CircleCI just announced called Dynamic Config.

Before deep-diving into this feature, let’s refresh what we know about monorepos and CircleCI piplines.

MonoRepos

It’s safe to say that monorepos can be very polarizing amongst developers. Some people prefer that an application, library or microservice exist within its own dedicated repository. Then there are those who prefer a super monolith repository.

I like monorepos because of benefits like simplified code reuse, atomic commits across modules/apps, easier dependency management and comprehensive search.

One of the major downsides (up until now) has been difficulty in scaling the build and deployment process as a monorepo grows. CircleCI’s Dynamic Config completely changes that…

Conventional CircleCI Pipelines

In CircleCI, workflows are usually triggered when changes are pushed to a repository branch. A project that employs Continuous Deployment may have a development branch which triggers a build/test workflow and a master branch which triggers a build/test/docker-image/deploy workflow.

Out of the box, CircleCI workflows can’t easily be altered based on files/modules that have changed as a result of the incoming changeset. This means you have to “build the world” in order to ensure that modules which may be affected by the changeset are rebuilt to incorporate it.

In a large monorepo with many modules/apps/libraries, this doesn’t scale. Large companies like Google, Facebook, Uber and Twitter (all of which employ super monorepos) have the infrastructure to build elaborate tooling to deal with this issue. Thanks to CircleCI’s Dynamic Config and two powerful orbs, this is now much easier for smaller companies and teams to achieve.

Understanding Dynamic Config

In order to setup selective builds within a monorepo, we first need to understand how Dynamic Config works.

In a standard “static” repository, a .circleci/config.yml file defines the commands/jobs/workflows for the project.

When using Dynamic Config, the process is split into two phases: the setup phase and the continuation phase. A static config file executes during the setup phase and can:

  1. dynamically generate another config file to use in the continuation phase
  2. analyze file changes by comparing the current changeset against a base revision
  3. set build parameters/flags which will be passed to the continuation phase
  4. trigger the continuation phase

Operation 1 is useful for very advanced pipelines but unnecessary to achieve selective builds in a monorepo. Therefore, in this exercise, we’ll only be performing operations 2, 3 and 4.

The two orbs that support this powerful feature are:

  1. circleci/path-filtering
  2. circleci/continuation

The circleci/path-filtering orb allows you to define regular expressions that are applied to a git commit diff, and then set specific build parameters based on the files/folders that have changed.

Under the hood, the circleci/path-filtering orb invokes the circleci/continuation orb to launch the continuation phase config and pass the build parameters to it.

My MonoRepo Structure

Before getting into the nitty gritty of the CircleCI configuration files, I’ll explain my monorepo and the various modules/services that live within it.

High-level view of the major modules within the monorepo

Modules connected via dashed-arrows indicate “weak” dependencies which must be independently buildable/deployable. The upstream contracts they rely on should evolve without breaking the downstream modules.

Solid-arrows represent “strong” compile-time dependencies. When an upstream module changes, the “strongly-connected” downstream modules must be rebuilt.

On the filesystem, the monorepo looks like this:

monorepo
|
|-- service
| |-- site-api
| | \-- pom.xml
| |-- batch
| | \-- pom.xml
| |-- common (service-layer)
| | \--pom.xml
| |-- pom.xml
|
|-- data (common-config)
| \-- pom.xml
|
|-- ui (site-ui)
| |-- build.sbt
| |-- package.json
|
|-- admin-ui
| |-- package.json
|
|-- product-classifier
| |-- api
| | \-- pom.xml
| |-- trainer
| | \-- pom.xml
| |-- common
| | \-- pom.xml
| |-- pom.xml
|
|-- functions (azure serverless functions)
| |-- pom.xml
|
|-- pom.xml

As you can see, there are a variety of build tools employed to build the various subsystems: maven (for Java modules), node.js + vue-cli + webpack + dart-sass (for VueJS modules) and sbt (for the scala-based web frontend).

There’s also a wide-variety of libraries used across the Java applications. For example, the product-classifier uses deeplearning4j and the serverless-functions depend on Azure-specific libraries. In order to avoid cross-contamination between libraries/jar files, these modules are completely independent of each other, have their own pom.xml files and are built in isolated CircleCI docker jobs.

The top-level pom.xml is responsible for defining the submodules, plugins and library versions.

Enable Dynamic Config

In the CircleCI project settings, under the Advanced tab, Dynamic Config is toggled on.

If you forget to do this, you may see an error in your CircleCI dashboard like
Continuation config contains setup stanza whilst not in setup anymore.

CircleCI setup config.yml

The setup phase config is beautifully simple.

When CircleCI executes this workflow, the path-filtering/filter job generates a git diff between the commit that triggered the workflow and the base-revision.

In my case, I’ve defined my base-revision to be the master branch. Any commit pushed to my development branch will inherently yield a diff when compared to my master branch.

Once I’m ready to deploy changes to production, development is merged into master and pushed to GitHub. The orb is smart enough to detect commits pushed to the same branch as the base-revision. In this scenario, the orb compares the new commit against the previous commit on the master branch (aka HEAD~1).

The regular expressions defined in the mapping block are then applied to the git diff file list. The mapping block is comprised of three-tuples delimited by spaces:

<regexp applied to file list> <build property> <property value>

If a regular expressions matches, then its respective build property is set to the defined property value.

In my case, the properties simply act as flags that will inform the workflows defined in the continuation config file specified by config-path.

CircleCI continuation workflows.yml

The continuation file defines the build parameters (line 7), jobs (line 51) and workflows (line 260).

Of note, each build param defaults to false. This ensures that the workflows only execute when the setup phase config determines that respective subsection of the monorepo has changed.

Each workflow is conditionally executed using when blocks (lines 262, 276, 290, 300, 310, 320). Modules that have multiple/transitive sub-dependencies are triggered using or conditions (lines 263, 277).

Finally, deployment jobs are triggered only when commits occur on the master branch.

What you’ll see on the CircleCI dashboard

Here you see the result of making a change to the monorepo/admin-ui module. Note the special setup tag on the setup phase workflow indicating CircleCI recognized it.

And here you see an example when a change is made that affects all modules.

Conclusion

CircleCI Dynamic Config is a really powerful feature for streamlining and manipulating builds at runtime with huge potential for cost savings. This example only begins to scratch the surface of what’s possible… but it clearly shows that monorepos with selective build pipelines based on file changes are finally achievable. Major props to the CircleCI team for launching this amazing feature! Thanks for reading!

Zenyatta at peace by shadowl360

Nerd For Tech

From Confusion to Clarification

Nerd For Tech

NFT is an Educational Media House. Our mission is to bring the invaluable knowledge and experiences of experts from all over the world to the novice. To know more about us, visit https://www.nerdfortech.org/. Don’t forget to check out Ask-NFT, a mentorship ecosystem we’ve started

Benjamin Possolo

Written by

I love ComicCon, fashion, sewing, gaming and coding.📍San Francisco.

Nerd For Tech

NFT is an Educational Media House. Our mission is to bring the invaluable knowledge and experiences of experts from all over the world to the novice. To know more about us, visit https://www.nerdfortech.org/. Don’t forget to check out Ask-NFT, a mentorship ecosystem we’ve started

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store