Using Monorepo? Do not rebuild unchanged components in CI

Reliza Hub Playground Project Built As Monorepo

UPDATE 2021–11–10: Updated version of this article with simplified solution can be found on Work & Life Notes blog: https://worklifenotes.com/2021/11/10/monorepos-how-we-do-it-with-reliza-hub/

While I generally prefer Multirepo approach for most of my projects, Monorepos do have their place. Particularly, in the DevOps world I use Monorepos to store collections of Dockerfiles — would be highly inefficient to have a separate repo for each.

But now new question comes in once we connect a CI system. How do we ensure that only modified components get built?

How To Take A Hash On Component Directory

I spent some time on the subject recently while working on Reliza. The key answer is hash — but the complication is that in Monorepos we’re dealing with hashes on directories and those are not natively supported by OS. Took me half a day to sort this out and for those of you interested in details I documented this process in my blog.

I’ll post the end result here which is:

dir=<mydir>; find "$dir" -type f -exec sha256sum {} \; | sed "s~$dir~~g" | LC_ALL=C sort -d | sha256sum

Now we can compute hash on individual components, but we need some reference point to compare it with. There are 2 main options.

Git Way To Check Changes In Monorepos

First, use Git as a reference point and store hash in a separate file per component on every CI build. And then compare new hash with that one on the new build. While this is workable, it costs an extra commit.

Reliza Hub Way To Check Changes In Monorepos

Second option — leverage Reliza Hub. That is if you want to avoid this extra commit, or you are using Reliza Hub for other use cases already.

Example of the other use case would be full auditability. Since Git on its own is not fully auditable unless you do very frequent full backups (that is because plain Git with all its greatness allows overwriting history which contradicts the principle of auditability).

When using Reliza Hub for this purpose, following needs to be added to your CI.

1. First, you create a project in Reliza Hub for every component that you have, and generate API Key per each project. This is a little tedious, but this is all part of one-time setup.
2. Next, in your CI, per project you add step to compute digest of component folder using the find command I posted above.
3. Then you make a call to Reliza Hub, using Reliza Go Client to check if this hash is already registered per this project in another release as following (using GitHub Actions secrets notation in this case):

docker run --rm relizaio/reliza-go-client checkhash    \          
    -i ${{ secrets.PROJECT_API_ID }}    \
    -k ${{ secrets.PROJECT_API_KEY }}   \
    --hash $hash

4. Then if response to that command equals to empty json, that means that no releases were registered yet, so you need to proceed with the build. Otherwise, you may safely skip the build and use the one from the release that Reliza Hub returned (in case you need it as a dependency).

5. At the end of your build process, you need to register your release on Reliza Hub. This may be done with the following command (again using GitHub Actions as an example):

docker run --rm relizaio/reliza-go-client addrelease \
  -b $GITHUB_REF \
  --commit $GITHUB_SHA \
  -v $version \            
  -i ${{ secrets.PROJECT_API_ID }} \
  -k ${{ secrets.PROJECT_API_KEY }} \
  --artid your-component-name-$version \
  --artbuildid github$GITHUB_ACTION$GITHUB_SHA \
  --artcimeta GitHub Actions \
  --arttype file_system \
  --artdigests $hash

Working sample of the whole GitHub Actions Script is available in the Reliza Playground Helper Project. Note that various client parameters are documented in the Reliza Go Client project.

You can see the whole process on YouTube at 2m:5s of Reliza Hub Playground tutorial. If you want to try that for yourself, I highly recommend starting with Reliza Hub Playground first and trying things there before going to actual Reliza Hub.

Please note that Reliza Hub is currently (as of March 2020) running as public preview offering, our expected pricing and free tier details after the end of preview is available here. If something does not work as expected with Reliza Hub, we have public support running on reddit’s r/Reliza.

Update (2020–03–06): I updated script to compute resulting hash with the reasons discussed in my blog following reddit conversation in r/bash with u/atoponce.

TL&DR

We discussed above:

1. How to take hash on a component directory so that we could compute if component actually changed between git revisions.
2. How to use plain git to store and check hashes between releases.
3. How to leverage Reliza Hub to store and check hashes between releases and thus avoid need of extra commit and also get another benefits, such as auditability.