Weave Flux — GitOps

Continuous Deployment to Kubernetes with GitOps (Weave Flux)

I am currently working as a freelancer in a project, where we are building a Java Spring Boot microservice system. In my current role, I am responsible for the devops tasks (setting up infrastructure, creating pipelines, etc…).

In the beginning I created a helm chart, that could be triggered with a values.yaml and then deploys our services in a generic way to kubernetes. This meant, each project had it’s own helm values and then we executed the deployment with a chart and this values. The process worked very well for a while, but it had some downsides.

1. Tiller was running in default configuration with admin permissions in the cluster and every developer needed to access it.
2. There was no way of seeing the changes made to the configuration on first look.
3. The container images weren’t updated automatically if a new container was pushed to the repository.
4. The helm charts weren’t easy to write and read.

Introduction to Weave Flux

After a short research I stumbled over a project called Flux, by Weave (See the repo). This project is actually an operator that is running inside your cluster and solves the downsides from above in a simple way.

Weave Flux — Overview

As you can see in the illustration, the weave works operator scans a remote git repository and one or multiple docker registries. If you commit a yaml file to the git repository, the operator automatically deploys what is described in there (The GitOps way). After the second commit, the operator will update the changed resources. Over a CLI tool called fluxctl, the user can setup certain automation tasks like updating the container automatically if a new image is pushed to the container registry.
As you can see, this tool solves all issues mentioned above. First the operator is pulling the configuration and hasn’t to be accessed directly. This is very different to tiller, where each user has to connect with helm to tiller for deploying stuff.
Also the developer has to commit a change to the central repository, what leads to auditable changes, the resources are plain kubernetes yaml files that are easy to read and to adjust. The operator can automatically update the deployment if a new image version is available.

Setup

The flux operator can be installed by helm or template. I used helm for this task, due to the fact that it was already installed, but the template way is also valid, if you don’t want use Helm at all.

1. Create a git repository containing the templates
2. Add the flux chart to your repo

helm repo add weaveworks https://weaveworks.github.io/flux

1. Then create a values.yaml

# Default values for flux.
nameOverride: "flux"

# Weave Cloud service token
token: "" #Can be left empty if you don't own weave cloud license

replicaCount: 1

image:
  repository: quay.io/weaveworks/flux
  tag: 1.4.2
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 3030

helmOperator:
  create: false
  repository: quay.io/weaveworks/helm-operator
  tag: 0.1.1-alpha
  pullPolicy: IfNotPresent
  tillerNamespace: kube-system
  tls:
	secretName: 'helm-client-certs'
	verify: false
	enable: false
	keyFile: 'tls.key'
	certFile: 'tls.crt'
	caContent: ''

rbac:
  # Specifies whether RBAC resources should be created
  create: true

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

resources:
  # If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  limits:
   cpu: 2000m
   memory: 2Gi
  requests:
   cpu: 100m
   memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

git:
  # URL of git repo with Kubernetes manifests; e.g. git.url=ssh://git@github.com/weaveworks/flux-example
  url: "<your-git-repo-address>"
  # Branch of git repo to use for Kubernetes manifests
  branch: "master"
  # Path within git repo to locate Kubernetes manifests (relative path)
  path: ""
  # Username to use as git committer
  user: "<git-user>"
  # Email to use as git committer
  email: "<your-mail>"
  # Path within git repo to locate Helm charts (relative path)
  chartsPath: "charts"
  # Period at which to poll git repo for new commits
  pollInterval: "30s"

ssh:
  # Overrides for git over SSH. If you use your own git server, you
  # will likely need to provide a host key for it in this field.
  known_hosts: "<the-key-of-the-known-host-here>\n"

#for https://github.com/justinbarrick/fluxcloud/
#additionalArgs:
# - --connect=ws://fluxcloud

In my case, I used a private git repository. This required, to setup first the git url and the target branch and then to store the known host key in the field.

The template required me to have a „n“ at the end of the key, because without it a deployment wasn’t possible. At the end the known hosts key value looked like the following:

"vs-ssh.visualstudio.com ssh-rsa AAAAB3...0lfKob4Kw8H\n"

1. After this you can execute the following command, to install weave flux to your cluster:

helm install \
  --name flux \
  --namespace <your-namespace> \
  -f values.yaml \
weaveworks/flux

1. After the operator started, go up into the logs until you find the public ssh key, that weave flux created. This key has to be added to your git server, because the operator uses this way to access your repository. After the key is deployed on your git server, the operator starts to fetch the contents for deployable templates. Also flux examines your cluster and scans all existing deployments and images used.

Usage

The first obvious step is to place some templates in the git repository.

I created the folder structure as following:

- /project-name
  - /backend
    - template1.yaml
    - template2.yaml
  - /frontend
    - ....

Actually the folder structure is completely up to your preferences, because weave scans all folders under the path that has been set in the values above.

After you add the templates, weave starts automatically to deploy your templates to the cluster.

Private Registry

If you use a private docker registry it is important, to provide a registry secret to the according namespace and write this secret as pull secret into the deployment itself.

Weave flux can’t read the secret just from the namespace.

First you need to create the secret for the registry in the namespace (See instructions here). After this, the secret has to be added to the deployment:

---
apiVersion: v1
kind: Service
...
---
apiVersion: apps/v1
kind: Deployment
metadata:
  ...
	spec:
	  containers:
	  - name: containername
		image: "private.registry:test-v100"
	...
	  imagePullSecrets:
	  - name: registry-secret

Automation

Now we have deployed the templates to the cluster, but weave flux doesn’t update the container after a new one has been pushed, whats wrong there?

Actually we need to enable the automation explicitly. This can be done by either the command line tool fluxctl (Download here) or by adding labels to the templates.

1. Fluxctl

The usage of fluxctl is a little bit complicated, because it targets by default the weave cloud address. After some research I found out how to use it with the standalone version:

kubectl port-forward deploy/<deploymentname> 10080:3030
export FLUX_URL="http://localhost:10080/api/flux"
fluxctl list-controllers -a

This will list all deployments in the cluster. The port-forward deploy is just a shortcut to not use the pod name. You can also do this just by forwarding the pod directly.

After this you can enable automation with fluxctl by using the fluxctl automate option or adjust which tags should be updated by fluxctl policy. To see all available commands just type:

fluxctl -h

2. Adding annotations

The fluxctl commands from above, just add annotations to the templates that are stored in git. You can easily set them on your self without using the fluxctl command line tool.

apiVersion: apps/v1
kind: Deployment
metadata:
...
  annotations:
	flux.weave.works/tag.<containername>: glob:develop-v*
	flux.weave.works/automated: 'true'
spec:
  replicas: 1
...
	  containers:
	  - name: <containername>
		image: "test-image:develop-v100"

This two lines will do the following:

Update the container if a new develop-v.. image is pushed and enable automation.

Summary

I am very happy with the weave works operator. The technology is easy to use and there are also some benefits that I haven’t mentioned yet. For example if you deploy to multiple clusters in multiple regions, you can just setup the weave operator in each cluster and it pulls the configs from git and deploys your services fully automatically. I will play around with the operator for a while and battle test it. Also I will still use Helm to deploy a lot of applications, because there are well written templates already there. But for simply deploying a self developed service to kubernetes, I will stick to Flux now. Is there something I overlooked or where I am wrong? Please tell me :)