Using Concourse CI/CD to publish Helm charts to ChartMuseum and report results to Slack (part #1)

This guide will walk you through the process of creating Concourse CI/CD pipeline to build and publish Helm charts to ChartMuseum. And then extending it further to include things like Slack notifications.

CD pipeline for building and publishing Helm charts

The post assumes you already familiar with Kubernetes, Helm, ChartMuseum, Concourse and want to build some automation on top.

We will show the resulting Concourse CI/CD pipeline that will continuously build and publish Helm chart on changes on git, which:

  • will be fully declarative and concise (under 40 lines of YAML)
  • will integrate with Git, ChartMuseum and Slack
  • will leverage Concourse resource abstractions and external tasks
  • will have no complex bash scripts

Taking Helm Chart for Consul

As an example, let’s try to build a basic CI/CD pipeline which takes an official HashiCorp Helm chart for Consul (https://github.com/hashicorp/consul-helm) and publishes it into a local ChartMuseum repository on every commit.

Why did we pick this use case?

Basically, it’s one of the things we had to do internally since HashiCorp does not host their own chart repository. We needed a packaged Consul chart in ChartMuseum, so it could be installed from a Helm chart repo instead of from a chart source in git.

Building a basic CI/CD Pipeline

As a first step, let’s define Concourse resources for our pipeline:

After that, let’s define a single job publish-upstream-chart-into-chartmuseum, which uses these two resources to get the contents of a git repo with Consul chart, and then build/publish the assembled chart into ChartMuseum.

You can view the full pipeline at https://github.com/Aptomi/concourse-pipelines/blob/master/consul-chart-blog/01_simple.yaml

Running the pipeline

You can use code from our GitHub repository to load this CI/CD pipeline into Concourse:

$ git clone https://github.com/Aptomi/concourse-pipelines.git
$ cd concourse-pipelines/consul-chart-blog
$ fly -t target set-pipeline -c 01_simple.yaml -p publish-consul-chart -l params.yaml
$ fly -t target unpause-pipeline -p publish-consul-chart

When Concourse runs it, the UI will show:

CI/CD pipeline for publishing Consul chart into ChartMuseum

Basically, it just grabbed chart source from git, built the Helm chart for Consul, and published it into ChartMuseum. From now on, the pipeline will ensure this happens on every git commit.

Pretty cool for such a short pipeline.

CI/CD pipeline build log

You can also verify that the chart is now in ChartMuseum:

$ curl https://your-chart-museum.company.com/index.yaml                                      apiVersion: v1
entries:
consul:
- apiVersion: v1
created: "2019-02-18T13:10:17.953198427Z"
description: Install and configure Consul on Kubernetes.
digest: 7751075c5a6d513017c210fd7927fb6a9ca0a93fda47441efea1020823b01044
home: https://www.consul.io
name: consul
sources:
- https://github.com/hashicorp/consul
- https://github.com/hashicorp/consul-helm
- https://github.com/hashicorp/consul-k8s
urls:
- charts/consul-0.6.0.tgz
version: 0.6.0
generated: "2019-02-18T13:10:18Z"
serverInfo: {}

Adding Slack notifications

With Slack webhooks and Concourse Slack Resource (https://github.com/cloudfoundry-community/slack-notification-resource), it’s pretty straightforward to add notifications.

Configure Slack resource with your Slack Webhook URL:

Then use this resource to send basic notifications from the job:

You can also import it from our GitHub repo:

$ fly -t target set-pipeline -c 02_simple_with_slack_notifications.yaml -p publish-consul-chart -l params.yaml

And voila, you can run the pipeline again and see the result in Slack:

So far:

  • The pipeline is just 40 lines, and it already can get data from git, publish charts, and send notifications to Slack
  • It’s fully declarative, relies on Concourse resources and doesn’t contain any custom scripts

Making it generic

After replacing names and URLs with Concourse pipeline parameters, the pipeline can be applied to pretty much any Helm chart — just pass the right ((chart-name)) and ((chart-git-repo)) when you create it.

And don’t forget to store ((slack-webhook)) in Vault, or any other credential manager supported by Concourse. That’s how we run it in production.

The full pipeline can be viewed at:

Final Words

Even though this example is pretty simple, you can see that Concourse gives you very powerful abstractions to build CI/CD pipelines. If you are switching to GitOps and declarative approach for continuous delivery of your apps and infrastructure, you should definitely take a closer look at Concourse.

Also, keep an eye for part #2, where we will take another iteration on the same pipeline and show how to:

  • leverage external tasks in Concourse
  • use task variables and perform local testing of tasks (new feature in Concourse 5.0)

UPDATE: part 2 is available at https://medium.com/aptomi/using-concourse-ci-cd-to-publish-helm-charts-to-chartmuseum-and-report-results-to-slack-part-2-ddadb65673d2

Contact

If you have any questions, please don’t hesitate to comment below or reach out to me directly via roman@aptomi.io.