Adding a unique build number to GitHub Actions
At Attest, we have begun trialling GitHub Actions as it is the new hot CI/CD tool. We currently use Jenkins; it’s a great tool, and there are many reasons still to use it, take a read of one of Attests previous blog posts as to why. One of the reasons we have decided to trial GitHub Actions is to help simplify our tech estate and reduce the overhead of managing our CI/CD infrastructure.
As part of trialling GitHub Actions, we have migrated our most significant frontend application to use it. We have made tremendous improvements to the DX for our frontend engineers and reducing build times significantly. There will be a future blog post exploring our migration piece in full. However, this blog post will specifically talk about one of the most significant limitations of GitHub Actions and how you can overcome it.
With many new tools, there are often limitations. For GitHub Actions, it’s the lack of a unique build number; something that in my opinion, is a needed feature for all CI/CD pipelines. There are many cases where a unique build number is required, such as tagging or providing the identifier to external tools such as Cypress.
For Attest, we need this feature to provide a unique shared identifier to Cypress. Cypress Dashboard API connects parallel jobs into a single logical run using a unique identifier. Currently, Github Actions does not provide such a thing. If using Cypress’ Github Action, it will try to create a unique key available which at this time is the commit SHA plus workflow name. Thus if you attempt to re-run GitHub checks, the Dashboard thinks the run has already ended. To truly rerun parallel jobs in this situation, you need to push a commit with a new SHA. You can achieve this with git commit -v --no-edit --amend & git push --force-with-lease . However, this is not great for developer experience; it may seem trivial to many. However, it’s quite frustrating for fast-paced teams because it requires context switching.
The above is an issue because when you run in parallel; each job will spin up in its own context. There is no shared context between the jobs which means we can not assign a unique identifier for the overall build between the jobs, which means you can’t hit that retry build when one of the tests suites failed. It shouldn’t happen, but when it does, it is a painful experience. It may also be that you want to share a unique identifier for other reasons. For example, each job creating an artefact with the same tag/version number.
Fortunately, the community has realised this is a problem and has communicated it with the Github team. Although no simple solution presents its self, there is a way we can achieve a unique identifier per build. It is no by means an obvious or elegant solution, but it is a solution.
If we were to take our example, our pipeline script might look something like below. Please note that it is oversimplified, your build pipelines are likely to be much more complex; the simplicity is to help make the solution more apparent.
The above workflow will run Cypress twice; it cleverly splits the load to multiple machines. However, to know which runs are related, Cypress needs a unique identifier. However, if we rerun this workflow, it will fail because the unique identifier has already been used 😒
Fortunately, this action allows the implementation of a custom build number, so we can manually create a unique identifier. This is where things get fun and ironically messy.
The first thing needed is to create a unique identifier; we will worry about storing and sharing this identifier in a later step. Let’s create a new job in our pipeline that will generate a unique identifier, for simplicity, this will be the date in seconds. Which in Linux you can create with date +%s , you may feel free to use any other method to generate a unique identifier that works for your needs.
In the above workflow, we have added a new job to create a build number which we can use to share with other jobs. We also specify that the test job relies on build_id to complete. Unfortunately, there is no way to reference previous job information from other jobs… Damn roadblock.
To give access to this build number to other jobs, we need to share it somehow. GitHub has a cache action that allows you to cache directories and restore them in future builds. Here we can let the unicorns free and do some magic.
The above code snippet saves the build number to a file .build-id/id , we will now add the saving of the information.
You may notice that we add the new cache step before saving the build number, this is because GitHub will restore the previous cache. Again we are using this action for an alternative use case than what it was initially designed to do. As a consequence, we want to ensure we save the new build number and prevent it from being reset to the previous cache. It makes the logic backwards, however now we have a unique build number stored which we can restore in a future job 👌
In this final workflow, we are restoring the cache from the previous job; we use the restore key ${{ runner.os }}-build-id-${{ github.head_ref }}-${{ github.sha }} . GitHub will restore based on the closest matching cache key; we have given it enough uniqueness to restore the correct cache containing our build number. Later, we then set the output of a second step to the contents of the file (our build number). Finally, we can now use this unique identifier with Cypress 👏 , which will allow us to rerun the workflow, consequently generating a new build number.
In conclusion; this is a lot of effort and quite a unique method to generate a build number for your workflow run. However; if you are excited to use GitHub Actions and you are restricted by this missing feature, then there is a way to get around it. It may not be the most elegant solution, but it works™️. So you can now rest at night knowing your engineers can request a re-build of a workflow without manually having to alter there git commit history.
Hopefully, GitHub Actions will release the ability to reference an external job, so we can remove the hacky cache step, or better yet give each build a unique identifier.
