GitOps for your Kubernetes workloads

Gaurav Agarwal

Follow

May 26, 2020 · 7 min read

Flux CD is a continuous delivery tool that is quickly gaining popularity. Weaveworks initially developed the project, and they open-sourced it to the Cloud Native Computing Foundation.

The reasons for its success are that it is Kubernetes-aware and straightforward to set up. The most promising feature it delivers is that it allows teams to manage their Kubernetes deployments declaratively.

Flux CD synchronises the Kubernetes manifests stored in the source code repository with the Kubernetes cluster through periodically polling the repository, so teams don’t need to worry about running kubectl commands and monitoring the environment to see if they have deployed the right workloads. Instead, Flux CD ensures that the Kubernetes cluster is always in sync with the configuration defined in the source code repository.

It allows teams to achieve GitOps, which has the following principles:

1. Git is the single source of truth.
2. Git is the single place to operate all environments, and all configurations are code.
3. All changes are observable/verifiable.

Why Flux CD?

Traditional CI/CD deployments with Kubernetes follow the below pattern:

1. Developers create code and write a Dockerfile. They also create Kubernetes manifests and Helm charts for the application.
2. They push the code to the source code repository.
3. The source code repository triggers a Jenkins build using a post-commit hook.
4. The Jenkins CI process builds the Docker image and Helm packages and pushes them to the artefact repositories.
5. The Jenkins CD process then deploys the Helm charts on the Kubernetes cluster.

The process sounds reasonable, and it is more or less the standard in the industry. However, there are a few limitations:

• You need to store the Kubernetes credentials in the Jenkins server. As the server is shared, there are chances of a compromise.
• While you can create and change configuration using Jenkins, it is not possible to remove existing resources using it. For example, if you remove a manifest file from your repository, kubectl does not remove it from the server. That is the greatest hindrance for automated GitOps.

How Flux CD Works

Flux CD allows teams to specify all required Kubernetes configurations using YAML manifests declaratively.

• Teams write the Kubernetes manifests and push them to a source code repository.
• The memcached pod stores the current configuration.
• Flux periodically (default five minutes) polls the repository using a Kubernetes operator for changes. The Flux container compares that with the existing configuration in memcached.
• If it detects changes, it syncs the configuration with the cluster by running a series of kubectl apply/delete commands. It then stores the latest metadata on the memcached store again.

Also, Flux CD allows you to poll container registries and update the Kubernetes manifests on your Git repository with the latest images if you want to automate upgrades to your workloads.

As Flux CD runs as a Kubernetes operator, it is simple to set up and great to start.

Let’s look into a hands-on demonstration so that we understand it better.

Prerequisites

Ensure that you have a running Kubernetes cluster and the cluster-admin role to deploy Flux CD.

Install fluxctl

Flux CD provides a fluxctl binary that helps you deploy and manage Flux CD within the Kubernetes cluster. Download the latest release of fluxctl and move it to the /usr/bin directory.

$ wget https://github.com/fluxcd/flux/releases/download/1.19.0/fluxctl_linux_amd64
$ mv fluxctl_linux_amd64 /usr/bin/fluxctl
$ sudo chmod +x /usr/bin/fluxctl

For this illustration, let’s use GitHub as the source code repository. Fork the bharatmicrosystems/nginx-kubernetes repository within your GitHub account.

The repository contains nginx-deployment and nginx-service manifests within the workloads directory and the web namespace definition within the namespaces directory.

├─ namespaces
│  └─ web-ns.yaml
├─ workloads
│  ├─ nginx-deployment.yaml
│  └─ nginx-service.yaml
├─ .gitignore
├─ LICENSE
└─ README.md

Provide the name of your GitHub user within the GHUSER environment variable and the GitHub repo within the GHREPO environment variable, as shown below. Create a new namespace called flux and install the Flux CD operator in the Kubernetes cluster.

The fluxctl install command generates the required Kubernetes manifests according to the following options:

• git-user — The Git user. In this case, the GitHub username
• git-email — The Git user email. In this case, the default GitHub email
• git-url — The URL of the Git repository
• git-path — The directories within the Git repository to sync changes from
• namespace — The namespace to deploy the flux operator

$ export GHUSER="<YOUR_GITHUB_USER>"
$ export GHREPO="<YOUR_GITHUB_REPO>"
$ kubectl create ns flux
namespace/flux created
$ fluxctl install \
--git-user=${GHUSER} \
--git-email=${GHUSER}@users.noreply.github.com \
--git-url=git@github.com:${GHUSER}/${GHREPO} \
--git-path=namespaces,workloads \
--namespace=flux | kubectl apply -f -
service/memcached created
serviceaccount/flux created
clusterrole.rbac.authorization.k8s.io/flux created
clusterrolebinding.rbac.authorization.k8s.io/flux created
deployment.apps/flux created
secret/flux-git-deploy created
deployment.apps/memcached created

Check if Flux deployment is successful.

$ kubectl -n flux rollout status deployment/flux
deployment "flux" successfully rolled out

Let’s get all resources within the flux namespace to see the current state of the objects.

As you see, there is a flux pod and a memcached pod. There is also a memcached service as the flux pod needs to interact with it.

$ kubectl get all -n flux
NAME                             READY   STATUS    RESTARTS   AGE
pod/flux-86d86b868-lndhn         1/1     Running   0          2m
pod/memcached-86869f57fd-qwnts   1/1     Running   0          2mNAME                TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
service/memcached   ClusterIP   10.8.11.199   <none>        11211/TCP   2mNAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/flux        1/1     1            1           2m
deployment.apps/memcached   1/1     1            1           2mNAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/flux-86d86b868       1         1         1       2m
replicaset.apps/memcached-86869f57fd 1         1         1       2m

Authorise Flux CD to Connect to Your Git Repository

We now need to allow the Flux CD operator to interact with the Git repository, and therefore, we need to add its public SSH key to the repo.

Get the public SSH key using fluxctl.

$ fluxctl identity --k8s-fwd-ns flux
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCryxSADyA+GIxtyCwpO3R9EuRcjZCqScKbYO246LZknyeluxKz0SlHYZHrlqxvla+k5GpPqnbImLLhuAD+YLzn0DbI58hUZLsrvxPWKiku--REDACTED--MKoPyEtQ+JiR3ZiADx6Iq8tYRRR+WBs1k5Hc8KNpg+FSRP8I8+CJRkCG4JQacPwK8FESP4qr1dxVv1tE8ZXyb8CdiToKpK7Mkc= root@flux-b9b4cc4f9-p9w88

Add the SSH key to your repository so that Flux CD can access it.

• Go to https://github.com/<YOUR_GITHUB_USER>/nginx-kubernetes/settings/keys
• Add a name to the Key in the Title section.
• Paste the SSH Key in the Key section.
• Check “Allow write access.”

Flux CD synchronises automatically with the configured Git repository every five minutes. However, if you want to synchronise Flux with the Git repo immediately, you can use fluxctl sync, as below.

$ fluxctl sync --k8s-fwd-ns flux
Synchronizing with ssh://git@github.com/bharatmicrosystems/nginx-kubernetes
Revision of master to apply is 8db9163
Waiting for 8db9163 to be applied ...
Done.

Now let’s get the pods to see if we have two replicas of nginx.

$ kubectl get pod -n web
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-7fd6966748-lj8zd   1/1     Running   0          20s
nginx-deployment-7fd6966748-rbxqs   1/1     Running   0          20s

Get the service, and you should see an nginx load balancer service running on port 80. If your Kubernetes cluster can spin load balancers, you should see an external IP appear.

$ kubectl get svc -n web
NAME            TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)        AGE
nginx-service   LoadBalancer   10.8.10.33   35.222.174.212   80:30609/TCP   94s

Test the service using the external IP. If your cluster does not allow you to spin load balancers, you can use the NodeIP:NodePort combination.

$ curl http://35.222.174.212/
This is version 1

Update the image to bharamicrosystems/nginx:v2 on workloads/nginx-deployment.yaml

$ sed -i "s/nginx:v1/nginx:v2/g" workloads/nginx-deployment.yaml
$ git add --all
$ git commit -m 'Updated version to v2'
$ git push origin master

Let’s now wait for five mins for the auto-sync, and meanwhile, watch the pods updating.

$ watch -n 30 'kubectl get pod -n web'NAME                                READY   STATUS        RESTARTS   AGE
nginx-deployment-5db4d6cb84-8lbsk   1/1     Running       0          11s
nginx-deployment-5db4d6cb84-qc6jp   1/1     Running       0          10s
nginx-deployment-6784c95fc7-zqptk   0/1     Terminating   0          6m43s

And as you see, the old pods are terminating, and new ones are spinning up. Check the pod status to ensure all pods are running.

$ kubectl get pod -n web
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-5db4d6cb84-8lbsk   1/1     Running   0          1m
nginx-deployment-5db4d6cb84-qc6jp   1/1     Running   0          1m

Now let’s call the service again.

$ curl http://35.222.174.212/
This is version 2

And as you see, the version is now updated to v2.

Congratulations! You have successfully set up Flux CD on your Kubernetes cluster.

Conclusion

Thanks for reading! I hope you enjoyed the article.

Flux is one of the most lightweight methods for declaratively syncing your Kubernetes configuration within your Git repository with your cluster, especially if you are making a start with GitOps.

In the next article “How to Build Containers in a Kubernetes Cluster With Kaniko” we will discuss another cool Google tool called Kaniko, so see you there!