CI/CD with github actions: an end-to-end guide on how to automatically generate release notes from commits.
The CI/CD is a process of continuously Integrating and Delivering through the cycle of software development that nowadays can be done through a variety of tools and platforms. The integration is the first step of this approach in which the developers can work in several branches at once in different parts of the code ( or even the same) and after all the work is done they need to gather all the modifications in one place, validate them and build the final version. The main purpose of this final version is to be deployed somewhere, but before that we need to make it available on some repository where the developers responsible for the next stage of the project can have access to the files, documentation and changelog. This part is called Delivery.
Github actions is a great platform for doing all of that. If you are familiar with github, the good news is that you already know the interface and you only need to learn how to organize your step by step solution to create your CI/CD workflow. And if you are not, then this is a great opportunity for you to learn how to set up your repository and to create an end-to-end solution to generate your releases. This is not a complete beginners guide though, but you can certainly learn something from this article.
Let’s get started!
I wanted to organize my repository like this one:
This is an image of the release page in a github repository for a python library called towncrier and I will explain later what it is used for but right now focus on the organization of this page. You can see that arrengement of a lot of famous repositories looks like this and there is a reason why. Like I explained in the introduction, when we have the final version of a code, we wanna make it available for others to use, consult or see what changed since the last version and a great way of doing that is through a github repository, specifically on a release page which you can access clicking on the Releases option on the right side.
We can see that the library name, the version (in this case 23.10.0) and below that we have a summary of all things that are different from the last version, such as improvements, bug fixes, new features, changes in the documentation, deprecations and ticket resolution. Each section has a brief description for them stating to the reader the project timeline and the respective modifications.
There are other important elements on this page such as the tag, which is explicit on the top-left side of the page and the assets that can include a lot of file types for instance executable files, but usually include a compressed version of the code present on that release.
So after reading a lot of documentation, articles and forums on the internet I came up with a solution on how to do that and make each step of it be automatic and that is what I would like to share with you guys.
Creating a tag
The first thing to start using github actions is to create a directory on your project root called .github and inside of it you create another folder called workflows that will be the place where you will be creating your workflow scripts responsible for the workflows on github actions.
After that you will have to create a .yml file which will contain the script that will create your tag. But why do we need that anyway? Well.. to create a release it is mandatory that it is based on a tag. And what is a tag? It is a way of referencing a certain state of your code at a certain point of the project timeline but the difference is that the code the tag refers to is supposed to be immutable and not be merged into any other branch.
In your .yml file you will stablish three things: the workflow name, an action that will trigger your workflow and the jobs that will execute the steps of what you need to do, in this case, create the tags. In my case, it looked like this:
name: Create tag
on:
push:
branches:
- main
env:
TAG_VERSION: v0.0.0
PREVIOUS_TAG: 0.0.0
LATEST_TAG: 0.0.0
TOML_VERSION: 0.0.0
jobs:
create_tag:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Check the latest version
run: |
all_tags=$(git tag | sort -rV)
read -ra all_tags_vector <<< "$all_tags"
latest_tag_version="${all_tags_vector[0]#v}"
toml_version=$(grep -E '^version\s*=\s*"[0-9]+\.[0-9]+\.[0-9]+"' pyproject.toml | \
echo "TOML_VERSION=${toml_version}" >>> $GITHUB_ENV
grep -o -E '[0-9]+\.[0-9]+\.[0-9]+')
function version { echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; }
if [ $(toml_version#v) -ge $(latest_tag_version) ]; then
echo "NEW_TAG_FLAG='1'" >> $GITHUB_ENV
echo "::set-output name=tag-created::1"
fi
- uses: butlerlogic/action-autotag@1.1.2
if: ${{ env.NEW_TAG_FLAG != '0' }}
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
with:
strategy: regex
root: pyproject.toml
regex_pattern: $TOML_VERSION
tag_prefix: vTo break this code a little, everytime we push new code to main branch github actions will trigger this workflow described by all the steps under the jobs section. On the env part I define and initiliaze the script internal variables.
On the jobs part I first specify the machine and the Operational System in which this script will run (ubuntu-latest). Then I use the action checkout@v3 that will be checking out to our project repository root using the github token provided by the secrets action environment variable. Then I use a git command git tag to list all tags available on my repo and sort them in descending order, take the most recent one and compare it with the version present on a toml file containing the version matching the specified regex pattern. The tag will be generated by an action called action-autotag@1.1.2 whenever the tag detected from the pyproject.toml file is greater than the latest tag present on github. For instance, if on my toml file I have updated the version number to be 1.2.3 and push the code to main branch, a tag will be generated and it will take the form v1.2.3.
Generating the release notes automatically using commit messages
Like I mentioned previously each release contains an associated tag (we already took care of it), the release notes and the assets. In this part we will be using a python library named towncrier (I told you we would go back to this one!) that summarizes documentation files for each release into a single file and can also create changelogs that store all changes made in all releases of your project.
To start using this library you have to standardize some things on your project, for example the way you document your code. Towncrier assumes that for each feature, bug fix, documentation change or removal (I am leaving one default type out here) your will write a txt file explaining it. Your files will have the format below, first a unique identifier, then one of the default type names and the format. By the way you can create this file by hand or using towncrier CLI. (see the official documentation at the end of this article for that)
1234.feature.txtSo when you have all your changes made and documented on that format, it is time to use the command below to build those files all in one place in a organized way. The version is going to be the same version you will used for your tag.
towncrier build --version 1.0.0 Now we know towncrier basics let’s jump to how to use our commit messages to build those files. There is another standard we will have to follow to get where we want: our commit messages. If we want to use them to document our code, we need to make them as clear as possible for others to understand what changes were done to the code and we also have to use prefixes to identify the change types. For example, suppose you implemented a new feature to the project, you might wanna write a commit message like the one below:
git commit -m "[FEATURE] In order to do X we implemented a feature called Y that is responsable for Z"Notice I wrote a prefix “[FEATURE]” that identifies our commit with one of the default types of towncrier. So in my case I created this pattern and I intend to follow it in my project, but you might wanna use a different one, just remember to link it somehow with default types.
Now we can make use of this to automatically generate the documentation files for each default type by using the bash command below:
git log $TAG_VERSION..HEAD --pretty=format:"%s" | \
grep '^\[FEATURE\]' | sed 's/^\[FEATURE\] //' | \
xargs -I {} bash -c 'echo "{}" > newsfragments/"$(date +%s%N).feature.txt"'Let’s break down this bash script. First we use a git log to list all the commits from one specific version until now (the variable $TAG_VERSION in this case). Then we use the grep to filter only the ones with the “[FEATURE]” prefix and the last line is used to store each commit in a file whose name consists of a date and time in nanoseconds, the default towncrier type and the file format. Before saving the commit message on it, the sed command will be responsable for removing the prefixes and writing just the relevant commit content.
I repeated this part three times on my code, one for each default type I decided to use in my project (feature, bug fix and documentation change). I am sure there are more simplified ways of doing the same thing but I think that writing each case separately helps us to visualize better the job executions and check which steps succeeded or not. After that we can finally build the files and towncrier will create a file called NEWS.rst containing the content organized in sections.
Writing the release workflow
The final step is to finally write the release part. Choose a name for this job and specify that you want it to run after the tag creation job completed.
publish_release:
needs: create_tag
runs-on: ubuntu-latestOn this job part, define the machine and os in which the script will be executed. Check out to the repository root and use an action called “setup-python@v2” to install the specified python version on it.
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up python
uses: actions/setup-python@v2
with:
python-version: 3.11On this part I use a requirements.txt file and one specific action to install poetry library and some other dependencies needed for my project.
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r .github/workflows/requirements.txt
- name: Install poetry
uses: snok/install-poetry@v1
- name: Install project dependencies
run: |
poetry install --no-rootTo catch the latest tag I the same code snippet I used on the first part, the only difference is that I save the latest tag on the $LATEST_TAG environment variable, remove the “v” prefix and save it on another variable called $TAG_VERSION.
- name: Get the tag name
id: get_tag
run: |
all_tags=$(echo $(git tag | sort -rV))
read -ra all_tags_vector <<< "$all_tags"
latest_tag_version="${all_tags_vector[0]}"
previous_tag_version="${all_tags_vector[1]}"
echo "TAG_VERSION=${latest_tag_version}" >> $GITHUB_ENV
echo "LATEST_TAG=${latest_tag_version#v}" >> $GITHUB_ENV
echo "PREVIOUS_TAG=${previous_tag_version}" >> $GITHUB_ENVLike explained before in this part I generate the NEWS.rst file containing the release notes and then build my project using poetry command.
- name: Generate new features commits file
run: |
git log $TAG_VERSION..HEAD --pretty=format:"%s" | \
grep '^\[FEATURE\]' | sed 's/^\[FEATURE\] //' | \
xargs -I {} bash -c 'echo "{}" > newsfragments/"$(date +%s%N).feature.txt"'
- name: Generate bug fix commits file
run: |
git log $TAG_VERSION..HEAD --pretty=format:"%s" | \
grep '^\[BUG FIX\]' | sed 's/^\[BUG FIX\] //' | \
xargs -I {} bash -c 'echo "{}" > newsfragments/"$(date +%s%N).bugfix.txt"'
- name: Generate documentation changes files
run: |
git log $TAG_VERSION..HEAD --pretty=format:"%s" | \
grep '^\[DOC\]' | sed 's/^\[DOC\] //' | \
xargs -I {} bash -c 'echo "{}" > newsfragments/"$(date +%s%N).doc.txt"'
- name: Generate Release Notes
run: towncrier build --version $LATEST_TAG --yes
- name: Execute poetry build
run: |
poetry buildThen we actually create the release by using the create-release@v1 action passing as parameters the tag version, the release name and the release notes generated in the previous steps.
- name: Create github Release
id: create_release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ env.TAG_VERSION }}
release_name: Release ${{ env.TAG_VERSION }}
body_path: |
NEWS.rst The final step will be to upload the assets. In my case when I ran poetry build it created two executable files on a directory called dist on my project root and I upload them by passing their path.
- name: Upload wheel and source distribution files
id: upload_assets
uses: actions/upload-release-asset@v1
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: |
dist/luana_test-${{ (env.LATEST_TAG) }}-py3-none-any.whl
asset_name: |
cli-${{ env.LATEST_TAG }}
asset_content_type: application/octet-stream
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}After all the script is written you just need to perform the action that will trigger the whole workflow, in our case it was a push or merge into main branch. If you do that and go to the actions tab you see the status of the workflow running and clicking on it the the dags representing your deploy steps. It is interesting to observe whilst it runs at least for the first runs, so that you can see warnings and possible errors that might occur during the process.
I hope this (kinda) tutorial helps you in some way to understand github actions a little better and how to successfully use your commit messages as release notes, optimizing your development time and helping your document your code while doing so. If you have any corrections or suggestions about this article feel free to contact me.
References
[1] https://medium.com/reprogramabr/git-e-github-por-onde-come%C3%A7ar-ca88a783c223
[3] https://docs.github.com/pt/actions
[4] https://circleci.com/blog/git-tags-vs-branches/
[5] https://towncrier.readthedocs.io/en/latest/tutorial.html
[6] https://github.com/marketplace/actions/auto-tag
