Sufficient automation coverage is a vital part of any development process. Ensuring problematic code can’t be committed into the main trunk is key to efficient, safe, and trustworthy engineering.

It is not a challenge anymore as many (if not all) source code management cloud solutions (GitHub, GitLab, BitBucket) offer additional continuous integration utilities. Example is “GitHub actions" integrated into code management workflow. Even if not for free, but usually at a very adequate price. It is highly recommended practice to use them as part of the development process.

The only unpleasant side effect of default CI configuration — all required steps in code check usually repeating each time CI is running, even when there is no need for them from their good sense. It is like running unit tests 2 times in the developer machine expecting another result in a deterministic environment.

This problem is more relevant for cloud CI/CD providers as results in the increased cost of these services. But even for on-premise solutions, it results in unnecessary time waste.

Thankfully, the community is aware of such a problem so has solutions for the range of the most common problems, like caching node dependencies using a “package-lock” file as a hash key.

-   name: Cache node modules
    id: cache-npm
    uses: actions/cache@v2
    with:
        path: |
            ~/.npm
            **/node_modules

        key: ${{ runner.os }}-build-${{ hashFiles('package-lock.json') }}
        restore-keys: |
            ${{ runner.os }}-build-

This definitely has a positive impact on CI total execution time as installing modules usually takes minutes.

But “install modules” is only one of the stages in the common CI pipeline. It usually contains also “linting”, “test”, “build”, and “e2e” which are project specific so can’t be covered with generic implementation. At the same time, some steps can be very time and resource-consuming, like the “e2e” and “test” steps.

NX to the rescue

By using NX as the mono-repo framework, I found one good “side-effect” that has a very positive impact on the CI execution time.

If the developers are widely using the NX command to check their works during development (e.g., running tests or linting code before committing it) — all these execution results can be cached. So, if there are no changes after the last run of the “test” command — the cached result is returned.

So, if there is no environment-specific dependency (e.g., build for Linux can be different than for Windows) almost full CI execution sequence of steps can be cached if was executed on the developer’s machine during development.

All you need is to specify operations that are cacheable:

 "tasksRunnerOptions": {
        "default": {
            "options": {
                "cacheableOperations": [
                    "build",
                    "lint",
                    "test",
                    "e2e"
                ]
            }
        }
    }

And define file system location for operation output if needed:

"targets": {
        "test": {
            "executor": "@nrwl/jest:jest",
            "outputs": ["test/coverage"],
            "options": {
                ...
            }
        }
    }

But even with environment-sensitive pipeline the first execution can be cached on remote machine with using machine specific parameters, like node version and then replayed from cache with each next run.

"targetDefaults": {
        "compile": {
            "inputs": [{ "runtime": "node -v" }],
            "outputs": ["{projectRoot}/lib"]
        },
        "generate": {
            "inputs": [{ "runtime": "node -v" }],
            "outputs": ["{workspaceRoot}/output"]
        }
    }

If remote machine expecting to be stateless — cache can be stored in cloud and synced with next build machine during next run.

"tasksRunnerOptions": {
        "default": {
            "runner": "@nrwl/nx-cloud",
            "options": {
                "cacheableOperations": [
                    "build",
                    "compile",
                    "test",
                    "generate",
                    "dockerBuild"
                ],
                "accessToken": "<REGISTER IN CLOUD AND OBTAIN TOKEN>"
            }
        }
    }

Expecting feature development process includes execution tests and build tasks on regular basis and especially before creating merge request, so that almost full pipeline execution results can be cached.

Here is life example of such optimization. Just selecting targets that can preserve cached result instead of re-compute it every time if there are no functional changes in project produces 5x faster CIs.

Overall CI execution time reduced from ~8 minutes to ~1 minute.

Summary

Taking into account that CI pipeline is usually checking every pushed commit, the number of pipeline runs can be dozens per single day. With proper cache optimization we can obtain huge time/money economy for our team and organization.

Thanks for reading. Let me share other posts that have similar background.

Monorepo for micro frontends — from isolation to generalization

Testing complex mono projects

Mutation testing as a more robust check of dependent libraries

Test pyramid as a measurable code metric

What I was doing wrong — dependency management and monorepo