AUTOMATING NPM PACKAGE PRERELEASES
We at the SLIIT FOSS Community are a passionate and committed team dedicated to promoting the use and development of open source software. It has been quite some time since we have shifted focus on building NPM and Dart libraries for use by fellow developers. The subject of this article however will be the former where we’ll be exploring the use of a plethora of tools and concepts to build a fully automated prerelease mechanism for these libraries. Simply, it’s one of the many ways in which we are trying to make open source development easier for everyone.
As of today, we have over a dozen packages which have been released to the NPM registry which you can find over here and till now, we only released everything as either major, minor or patch updates. However as of late, with more complex developments, we had identified the need for prereleases which would primarily help us with a smoother and safer way of getting things done. There were instances that a library was completed to a certain extent, was ready for internal testing but not just yet ready for a full release.
At the time, we had already automated the existing release process with the use of a GitHub Actions Workflow that would be run on a push to the main branch of the repository. This workflow took care preparing the project, executing the required scripts for versioning, releasing the packages to the NPM registry and finally syncing the newly published package versions back into the repository by executing a commit within the workflow itself.
All of our packages are housed within this single repository and it is bootstrapped with Turborepo which was already making things quite easy for us. For example, we could run a single command to execute a script across all packages within the repository and in our case, we had a release script defined within all of our packages and a simple process of running pnpm release would take care of the rest. The release script was organized as follows:-
> Root package.json
{
"scripts": {
…
"release": "turbo run release - concurrency=1",
…
}
}> Workspace package.json
{
"scripts": {
…
"build": "node ../../scripts/esbuild.config.js",
"bump-version": "pnpm build && npx automatic-versioning - name=@sliit-foss/<package-name> - no-commit - recursive",
"release": "bash ../../scripts/release.sh",
…
}
}> release.sh
pnpm run bump-version && pnpm publish - access=public - tag=latest || trueOur versioning logic was quite simple, we decided which type of release it was based on the prefix of the commit which triggered the workflow run, for example:-
- `Fix: <commit message>` would trigger a patch release
- `Feat: <commit message>` would trigger a minor release
- `Feat!: <commit message>` would trigger a major release
This process was quite easy since we anyways follow conventional commits while checking in changes. We have Husky setup with Commitlint which ensures that this safe-zone is never crossed. The version bumping itself is handled by a library of our own making called @sliit-foss/automatic-versioning which evaluates the commit history and
determines the next version based on the type of release.
While this process was working fine, it did have these following problems:-
1. What if there was a prerelease that needed to be done? This would mean that we would have to manually bump the version of the package, publish it to the NPM registry and then sync the version back into the repository. This was quite a tedious process and we wanted to automate this as well. Further to this, all releases were done from the main branch itself which meant that we had to be extra careful when merging pull requests to the main branch since we didn’t want to accidentally release a package which wasn’t ready for release.
2. There was no development branch or anything which matched it. All changes had to be held within the feature branches itself to avoid polluting the main branch. This was quite a hassle since we had to keep track of which feature branches were ready for release and which weren’t.
3. The release process was quite slow. As you might have already seen, the release script at the root level was limited to a concurrency of 1 and the scripts have sequential dependencies on previous scripts. For example, bump-version calls pnpm build within itself. This was primarily to ensure that the scripts were run in the order of build — -> bump-version — -> release. This was unnecessary since Turborepo already has more efficient ways to manage this.
Here is a diagram of the release process as it was before and after this development cycle and a few screenshots of the final pipeline executions.
Fun fact!
The prerelease tag blizzard was actually named after the Witcher potion Blizzard which is a potion that slows down time and increases perception. We thought it was quite fitting since prereleases are meant to be a slow down of the release process and a way to increase perception of the changes that are being made, a concept which is quite similar to Canary Releases.
The Solution
First off, we had two approaches to solve this:-
Quick way
Add Prerelease commit prefix evaluation to automatic-versioning and increment the version accordingly. This would mean that we would be able to merge in prerelease changes to the main branch with the associated commit prefix and let the existing release workflow take care of the rest. But this obviously will be polluting the main branch and doesn’t solve the problem of every release being published with the latest tag which can be quite dangerous.
Complete way
Extract the existing release workflow into a custom Composite GitHub Action. This would make the exact same release steps reusable for both types of releases. The variables which in this case consist of only the release tag were provided as inputs into this action along with the needed repository secrets.
Modify the existing release workflow to consume this new action and add in a second prerelease workflow which gets triggered on pushes to the development branch. The final workflow files and the action looks like this:-
> .github/actions/release/action.yml
name: release
description: Base package release action
inputs:
npm_token:
description: "Token to authenticate with the npm registry"
required: true
runs:
using: composite
steps:
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: '16.x'
registry-url: 'https://registry.npmjs.org'
- name: Configure git
run: |
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
shell: bash
- run: git fetch --prune --unshallow
shell: bash
- name: Install dependencies
run: npm install -g pnpm@8 && pnpm install --production --ignore-scripts
shell: bash
- name: Create .npmrc
run: echo "//registry.npmjs.org/:_authToken=${{ inputs.npm_token }}" > .npmrc
shell: bash
- run: echo "git-checks=false" >> .npmrc
shell: bash
- name: Sync submodules
run: pnpm sync-submodules
shell: bash
- name: Populate prerequisities
run: |
echo "{\"release_tag\":\"$TAG\"}" > cache-control.json
for dir in packages plugins; do
cd "$dir" && for p in */; do
cp ../{.npmignore,LICENSE,cache-control.json} "$p"
done && cd ..
done
shell: bash
- name: Publish packages on NPM
run: |
pnpm --filter @sliit-foss/automatic-versioning build && npm i -g ./packages/automatic-versioning
pnpm release
shell: bash
- name: Cleanup
run: rm -rf cache-control.json && rm -rf p*/**/cache-control.json
shell: bash
- name: Update release info
run: |
git config pull.ff true
git add . && git commit -m "CI: @sliit-foss/automatic-versioning - sync release" || true
git pull --rebase && git push origin
shell: bash> .github/workflows/release.yml
name: CI Release
on:
push:
branches:
- main
workflow_dispatch:
jobs:
release:
runs-on: ubuntu-latest
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
TAG: latest
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/release
with:
npm_token: ${{ secrets.NPM_TOKEN }}> .github/workflows/prerelease.yml
name: CI Prerelease
on:
push:
branches:
- development
workflow_dispatch:
jobs:
release:
runs-on: ubuntu-latest
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
TAG: blizzard
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/release
with:
npm_token: ${{ secrets.NPM_TOKEN }}This on its own was not enough, we had to add in a couple of new features to our library `automatic-versioning` to be able to support this process. The newly added features are as follows:-
- The ability to specify a prerelease tag as a command line argument when invoking the script.
- The ability to specify a prerelease branch as a command line argument when invoking the script which will essentially change the commit prefix evaluation as follows while in that branch:-
- Feat! → Premajor
- Feat → Preminor
- Fix → Prepatch - The ability to designate a list of prefixes as ignored prefixes which will not be considered when evaluating the commit prefix. This was useful as we needed to ignore the prerelease version sync commit from the development branch which is prefixed with `CI:` from being considered when evaluating the commit prefix and to avoid it being considered as an invalid value for version incrementing.
- The ability to recognize the commit history for a specific workspace instead of the whole repository as a whole
- Support for commit scopes. For example we now can add in commits with prefixes as follows -> Feat(automatic-versioning): commit scope support
Further in the midst of this release, we added in the following bugfix to automatic-versioning
- Invalid versioning if the commit contains a URL in the commit message such as a merge commit with a source branch URL
Problem two got automatically solved with the above. The availablity of a development branch with its own release cycle meant that we could now merge pull requests into the development branch and keep them there as long as we wanted to until they were ready to be merged into the main branch.
Finally to address our third and final problem, we removed the concurrency limit from the turbo script and structured the pipeline in our turbo.json as follows:-
{
"pipeline": {
"build": {
"dependsOn": [],
"outputs": ["dist/**"]
},
"bump-version": {
"dependsOn": ["build"],
"outputs": ["package.json"]
},
"release": {
"dependsOn": ["bump-version"],
"outputs": []
}
}
}This indefinitely sped up the release process since the scripts were now run in parallel and the dependencies were handled by Turborepo itself. Further to this, we integrated Vercel Remote Caching in our CI pipeline which meant that all of the steps were now cached and the subsequent runs were much faster. This process turned out be quite simple since we were already using Vercel for a few of our other projects and the integration was quite seamless. Jared Palmer has done a quite a good job of making this process as simple as possible as it just requires 2 environment variables to be set in the CI environment which we added as repository secrets referenced in our workflows. For more information, refer the following docs.
Overall these changes reduced the release time from 1–2 mins to 30–45 seconds which was a huge improvement.
There however was a catch, since we were using Vercel Remote Caching, we had to ensure that the cache was invalidated whenever a new release was made in the main branch. Say for example I’m adding a patch to a package with a version 1.0.0 and merging it with the development branch. This package will be released as 1.0.1-blizzard-0 during the prerelease workflow run of this branch. Now if I merge the same branch to the main branch, in the eyes of Turborepo, the inputs to the release script have not changed since the time it was run in the development branch which will cause Turbo to replay the cache from it thus skipping the actual release of the package to version 1.0.1. Fortunately, dealing this was quite simple, we just needed turbo to maintain two caches for the 2 release types which we did by adding a simple cache-control.json file to each of the project workspaces at the time of the CI run as in the action steps below where the tag is different for the two workflow runs:-
- name: Populate prerequisities
shell: bash
run: |
echo "{\"release_tag\":\"$TAG\"}" > cache-control.json
for dir in packages plugins; do
cd "$dir" && for p in */; do
cp ../{.npmignore,LICENSE,cache-control.json} "$p"
done && cd ..
done
# Release step
- name: Cleanup
shell: bash
run: rm -rf cache-control.json && rm -rf p*/**/cache-control.jsonAs an added bonus, we also updated our existing testing workflow to incorporate linting along with unit tests. We could’ve just added in a separate workflow for linting or another job itself within the existing test.yaml file but that would have been a massive repetition of code. After all only the `run` command would have been different for both jobs which is why we decided to use the `strategy` feature of Github Actions to run the same job twice with different run commands as follows:-
> .github/workflows/lint-test.yml
name: CI Code Quality + Tests
on:
pull_request:
branches:
- main
- development
jobs:
scripts:
runs-on: ubuntu-latest
strategy:
matrix:
command: ['lint', 'test']
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 16
- name: Install dependencies
run: npm install -g pnpm@8 && pnpm install --ignore-scripts
- name: Run checks
shell: bash
run: |
pnpm --filter @sliit-foss/eslint-config-internal build
pnpm ${{ matrix.command }}
env:
GITHUB_ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN_GITHUB }}
FIREBASE_CONFIG: ${{ secrets.FIREBASE_CONFIG }}
