Alex Raul
Alex Raul
Jun 16 · 5 min read

I recently had to build a GitOps driven CI/CD pipeline to Kubernetes for a client. Based on a few constraints — I went for an open source solution in the Flux library (https://github.com/weaveworks/flux).

By default, Flux does not have the ability to spin up an entire new environment automatically based on a new branch in Git (which was a requirement) but this functionality was trivial to hack in by adding a new step to the CI pipeline.

The end result is the ability to push to a new branch on the application repository and have an entire environment spin up automatically in Kubernetes based on that branch, with DNS automatically pointing to a subdomain named after that branch (“feature-1423.dev.myapp.com”), courtesy of ExternalDNS (https://github.com/kubernetes-incubator/external-dns).

Flux: The Basics

Image Credit: Weave Flux Github Repo

I am going to focus on Flux in the scenario of a single-app GitOps pipeline — but the ideas presented here are extensible to multi-app deploys.

Note: when I say there is only a single GitOps driven deploy, that means that pushing to only one application repository — and therefore one docker registry — will trigger updates in Kubernetes. This is not to say that the deployment itself does not include multiple containers or even multiple Kubernetes deployments (in my case, both were true), just that only one application repository is triggering updates.

In this simple setup, Flux depends on 2 Git repos.

Firstly, the Application Repository, which ideally has some form of CI that pushes a new version to a Docker registry (which in my case was ECR).

Secondly, the Infrastructure Repository — which contains the relevant YAML or helm charts that specify how the application is deployed to the cluster, as well as a Flux Workload definition:

---
# Create a flux workload that will automatically update whenever
# a new container with tag "dev-1" is pushed
apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
name: dev-1
namespace: dev
annotations:
flux.weave.works/tag.chart-image: glob:dev-1
flux.weave.works/automated: "true"
spec:
releaseName: dev-1
chart:
git: ssh://git@github.com/my-git-repo
ref: master
path: helm/charts
values:
image: 000000.dkr.ecr.us-east-1.amazonaws.com/my-repo-name
tag: dev-1

Flux works by virtue of two mechanisms. Firstly, it syncs with your infrastructure repo and checks for Flux Workloads in a folder you specify when you start Flux on your cluster.

Secondly, it polls any Docker Repositories that are specified in the Flux Workload to check for new versions. If a new version is found (and here you can set a filter in your Flux workload to only look for image tags that start with dev-*, for instance), it will update the relevant Flux workload with the new version.

This is already pretty great, but the real magic happens when setting a Flux workload to deploy a Helm Chart — which is natively supported by Flux (https://github.com/weaveworks/flux/blob/master/site/helm-get-started.md and https://github.com/weaveworks/flux/blob/master/site/helm-integration.md).

To start Flux with the helm operator, you use a command of the form:

helm upgrade -i flux \
--set helmOperator.create=true \
--set helmOperator.createCRD=false \
--set git.url=git@github.com:YOURUSER/infrastructure-repo \
--namespace flux \
weaveworks/flux

Now you can harness dynamic Helm templating to name and tag workloads based on container image tags — which is exactly what I needed to spin up entire environments automatically based on a new feature branch.

How does this work? Flux will pass in values to Helm on an update, which you can use in your charts:

# Simplified Deployment Helm Chart YAML# Helm Variables are used to name release from ECR image tagapiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ .Values.tag }}
namespace: dev
labels:
app.kubernetes.io/name: "{{ .Values.tag }}"
helm.sh/chart: {{ include "dev-release-chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: "{{ .Values.tag }}"
app.kubernetes.io/instance: {{ .Release.Name }}
strategy:
rollingUpdate:
maxSurge: 10
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: "{{ .Values.tag }}"
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
containers:
- name: app
image: "{{ .Values.image }}:{{ .Values.tag}}"
imagePullPolicy: Always

This Helm template shows a deployment that would be named based on the exact feature branch tag on ECR, in addition to having the main app container be pulled based on that tag. In the full version, I made use of the Helm variables to automatically name external-dns entries — but you could use them for anything.

The last piece of the puzzle

With all of this in hand, we can automatically update our cluster on a Git (and therefore Container Repository) update to our application repo, AND we can change our Infrastructure Repo and see the new setup deployed in Kubernetes.

However, each Flux Workload in the Infrastructure Repo is only good for updating one environment by default.

For instance, if you set your Flux Workload to look for container tags of the form “dev-*”, and you push a new container tagged dev-2, that image will override the dev-1 currently running.

This isn’t what I wanted — so I added one last piece to the setup: whenever a new branch gets created and pushed in our application repository, the CI system (in this case CircleCI) will make a Git commit and push a new Flux Workload yaml to the infrastructure repository.

In this case, the Flux Workload would not specify a filter like “dev-*” but would actually specify the exact branch, “dev-1”. So new container images with that tag will automatically update the dev-1 deployment, but dev-2 will instead be associated with an entirely new deployment.

This means that you can look at the infrastructure repository and see, at a glance, what workloads and branches are running in-cluster.

How it works in practice

For example, let’s say a developer makes a new branch of the application repo, called “dev-test-1”. They commit some changes and push the new branch.

Now, our CI pipeline will perform it’s usual steps of building and pushing an image (labeled “dev-test-1” to ECR). Then, it will perform the additional step of making a commit to the Infrastructure Repo and pushing it, which will look like this:

---
# Created by CI Pipeline
apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
name: dev-test-1
namespace: dev
annotations:
flux.weave.works/tag.chart-image: glob:dev-test-1
flux.weave.works/automated: "true"
spec:
releaseName: dev-test-1
chart:
git: ssh://git@github.com/my-git-repo
ref: master
path: helm/charts
values:
image: 000000.dkr.ecr.us-east-1.amazonaws.com/my-repo-name
tag: dev-test-1

On the next Flux sync, it will see this new workload and spin up the entire environment for our branch. In addition, whenever the branch is updated (a new image with tag dev-test-1 is pushed to ECR) it will update the workload running on Kubernetes.

Next Steps

If you’ve read this far, you’ve probably noticed a glaring omission: deleting environments.

I haven’t implemented this yet, but it could be accomplished with a Git Webhook which would delete that specific branch from the Infrastructure Repo, which would trigger a deletion in the cluster on the next Flux poll.

Keep an eye out for Part 2, where I’ll show the exact configuration and CI steps I used to get this working.

Rackner | Dev Blog

The Rackner Developer Blog

Alex Raul

Written by

Alex Raul

CEO @racknerco | Cloud Native | Digital Transformation | Mobility | AWS Consulting Partner

Rackner | Dev Blog

The Rackner Developer Blog

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade