How did I build automated release notes with Git+GoCD+Jira+Confluence

Published in

IOTICS

7 min readJan 28, 2022

TL;DR

As our product got bigger, the number of engineers went up and the number of our GitHub repositories crossed 100, we have reached the point where it was more and more difficult to keep track of all the changes made across all the GitHub repositories and all the Jira projects.

How could we present all the relevant changes to our customers and internally in a simple way? With release notes, of course, which someone could produce manually.

But how could we do this in an automated way? We could inspect Git commit messages. But adding rich text into commit messages isn’t ideal. Could we instead make use of rich text capabilities in Jira issues and keep Git commit messages simple?

This article explains how I’ve built automated process as a microservice which generates rich text release notes from Jira issues, using information from commit messages across multiple Git repositories using GoCD, Jira and Confluence.

A picture worth 1000 words:

The Basics:

Every work is tracked in Jira
Release Notes are defined as a custom field in Jira
Jira issues can have associated Release Notes
Every Git commit on the main branch contains the Jira issue key
CI/CD pipelines are defined in GoCD; GoCD aggregates commits that make up a build, potentially from multiple repositories. This is built-in into GoCD already.

Release Notes Automation Process:

The automation runs as a microservice
It uses GoCD API to get the list of all commits that make up a specific build
Extracts the list of all related Jira issues and finds the Release Notes in them
Uses Jira Cloud REST API to get the content of the related Jira issues
Aggregates the Release Notes from the Jira issues under the same headings.
E.g. all Bug Fixes across the Jira issues will Bug Fixes heading in the final release notes.
Publishes the final Release Notes as a Blog Post in Confluence using Confluence Cloud REST API.

Read more below for a more detailed overview…

What?

The Release Notes automation leverages information stored in multiple separate systems:

Jira — issue tracking system; knows about the WHAT, WHO, WHEN, WHY
Git — source code repository; knows about HOW, WHO, WHEN
GoCD — CI/CD pipeline; knows about WHO, WHEN
Confluence — documentation; knows about WHAT, HOW, WHY

GitHub can be easily linked to Jira/Confluence and vice versa.

This automation at least allows us to remove one, previously manual, often overlooked, tedious process of generating Release Notes.

Engineers can collaborate on the Release Notes as part of elaboration, even before they change any code. Breaking Changes can be described up in Jira with rich content which wouldn’t be easily possible to do in Git commit messages.

Project Managers can subscribe and get notified when new Release Notes are published to Confluence and refine them if needed.

Why?

These are some of the use cases/stories identified for creating this automation:

As an Iotics customer
I want to have access to timely and accurate release notes
So that I can make the best judgement about upgrading Iotics dependencies
As a Product Owner
I want to be able to easily publish release notes generated from all the changes in Jira
So that I don’t waste time on collating them manually
As an Iotics engineer
I want to quickly see what changes were released by other teams
So that I can be better informed about what the other teams are working on

GitHub integration for Jira wasn’t good enough

Git branches in GitHub, Pull Requests and commits can appear in the Jira issue under the Development panel, when a Jira issue key is included when performing a branch, Pull Request or commit in GitHub.

See atlassian/github-for-jira

Jira integration for GoCD isn’t an integration

There is no GoCD plugin for Jira/Confluence.

There is no Jira/Confluence plugin for GoCD either.

It’s not possible to see or manage GoCD builds and deployments from Jira.

The only available “integration” of Jira for GoCD isn’t really an integration. It can only convert Jira-like keys in commit messages to hyperlinks to the related Jira issues. See Integrating GoCD With Other Tools | GoCD User Documentation

The YAML configuration for this for a sample pipeline looks like this:

pipelines
  sample-pipeline:
    group: your-project
    label_template: v0.0.${COUNT}
    tracking_tool:
      link: https://your-domain.atlassian.net/browse/${ID}
      regex: ([A-Z]+-\d+):

This just wasn’t good enough.

We wanted something better. Automatically generated release notes, from the rich content information stored in Jira, not from commit messages.

How?

What Jira provides

Every work should start with a Jira issue. No Jira = No work.

During story refinement and elaboration, the team identifies which stories will benefit from or require Release Notes, and then add a release-notes-required label to the corresponding Jira issues.

This triggers a Jira automation which pre-populates the Release Notes custom field with a template that contains some pre-defined headings. The template makes it easier to collate the Release Notes under the same headings.

Engineers can then update the Release Notes when they work on the Jira issue.

What Git (GitHub) provides

Our teams work on many Jira issues in parallel and commit code to multiple repositories in GitHub in parallel as well.

Every commit contains the Jira issue key related to the work, e.g. JI-123.

Automated services also commit code to GitHub (e.g. dependabot), but these don’t contain the Jira issue keys.

What GoCD provides

GoCD contains pipelines for almost every repository in our GitHub organisation.

A pipeline polls GitHub every minute and if it finds new commits, it runs the CI/CD pipeline for that repository. We use tomzo/gocd-yaml-config-plugin so that we can define our CI/CD pipelines in YAML (using GitOps).

Still no GitOps article in Wikipedia?

Pipelines can form a workflow aka Value Stream Map.

A release pipeline combines releases from multiple repositories/pipelines into a single deployable package (e.g. a helm chart) with a specific version.

What is missing?

GoCD isn’t really integrated with GitHub, Jira or Confluence.

As long as we use GoCD for CI/CD, then GitHub, Jira and Confluence don’t know what changes make up a release in GoCD. What, when or where is a release built and deployed?

The Release Notes Automation

This is a microservice written in Go that provides the Release Notes automation.

It gets typically triggered via curl at the end of a release pipeline in GoCD, with the name of the pipeline, the version and a build counter.

The automation then follows these four steps:

① Calls GoCD API

gets pipeline version and timestamp
gets all commits since the previous build (counter-1)
finds JIRA issue keys from commit messages (ignore dependabot)

# Example code:
# Getting GoCD pipeline changes compared to the previous build# get list of all commit messages across all repositories
# which make up a particular release in GoCDcurl "https://<gocd-server>/go/api/pipelines/release pipeline/compare/100/101" \
 -s -u "<user>:<token>" \
 -H "Accept: application/vnd.go.cd.v1+json" | jq"

Useful Links:

② Calls JIRA API

gets JIRA issue details and finds the Release Notes custom field
groups release notes by their headings

# Example code: Getting a Jira issuecurl -X GET \
  -u "<user>:<token>" \
  "https://<domain>.atlassian.net/rest/agile/latest/issue/<issue-key>"

③ Calls Confluence API

converts the release notes from Jira (including rich text formatting) to Confluence markup/format (aka representation).

# Example code: Create Confluence Blog Post
# this is a sample using rich formatting as it comes from Jira APIcurl -X POST \
  -u "<user>:<token>" \
  -H 'Content-Type: application/json' \
  <domain>.atlassian.net/wiki/rest/api/contentbody/convert/editor2 \
  --data @- << EOF
{
    "value":"{cheese} - {{code}}
    (http://www.google.com)[link]
    *bold** item 1
* item 2
** nested item
    ",
    "representation":"wiki"
}
EOF

④ Calls Confluence API

publishes the release notes as a blog post
the blog post has a label of the pipeline name

# this is a sample using rich formatting, link and bullet listcurl -X POST \
  -u "<user>:<token>" \
  -H "Content-Type: application/json" \
  <domain>.atlassian.net/wiki/rest/api/content/ \
  --data @- << EOF
{
    "type":"blogpost",
    "space":{"key":"RN"},
    "status": "current",
    "id":"20220101001",
    "title":"2022-01-01 Our Project v5.2.0 - Release Notes",
    "body":{
        "storage":{
            "value":"<h1>Changes</h1><code>code</code>    <ac:link><ri:page ri:content-title=\"link\" /></ac:link>(<a href=\"http://www.google.com\" class=\"external-link\" rel=\"nofollow\">http://www.google.com</a><ul><li><strong>hello</strong> 123</li></ul>",
            "representation":"storage"
        }
    }
}
EOF

Useful Links:

The Confluence Cloud REST API

What have I learnt?

1. Keep testing the APIs with Postman and curl

To get quickly started, I’ve used Postman, curl and jq to quickly figure out the necessary external APIs.

This was much faster and more lightweight than writing the final code in Go.

2. Next time start without Jira custom fields

It might be easier, if we didn’t use a separate custom field to define the Release Notes. It would probably work sufficiently well if the Release Notes were part of the default Description field in Jira.

Using custom fields in the code is tricky. When you want to get the content of the custom field, you have to know the Custom Field ID (e.g. customfield_10101). You could get all the custom fields and find which Custom Field ID corresponds to the desired custom field (in our case called “Release Notes”). Anybody else’s Jira instance would most likely have a different custom field ID.

To start simple, I’ve hard-coded the custom field ID. This could be improved, possibly by making the look up or at least making the custom field ID configurable.

3. Keep using direct REST API calls

The microservice is implemented in The Go Programming Language .

Why using REST API’s directly and not some clients in Go? There are clients in Go for GoCD, Jira and Confluence.

Because I haven’t found any client in Go which would look active and maintained.

Also because of additional dependencies on potentially questionable 3rd parties.

ctreminiom/go-atlassian is new, but too new, Confluence API is not implemented.
Specific Jira or Confluence clients in Go are either years old, not maintained, or work with the on-prem installed products and not with the Cloud versions of Jira and Confluence.

Source: https://github.com/Iotic-Labs/gocd-jira-release-notes/

Hope this helps.

This story first appeared on https://www.linkedin.com/pulse/how-did-i-build-automated-release-notes-miroslav-sommer/