GitOps with Weave-Flux
I recently read about Weave-Flux, I wanted to see how it works and ended up making a Minikube boilerplate with Traefik handling Kubernetes ingress traffic and path redirection.
I will start by assuming that you know what DevOps, Git, Kubernetes, Reverse Proxy or any other term that is common in the age of cloud computing. If not, you should look them up in your favourite search engine.
My understanding of the GitOps concept is to keep the Ops from DevOps in Git and benefit from Git features. This is nothing new, what dev doesn’t already do this? I think we all use Git for any other purpose than it was designed.
MyApp Scenario
Build a web app composed of multiple scalable services.
- Each service will have its own path.
- MyApp will be deployed on Kubernetes.
- Kubernetes is set-up to only allow access to my own namespace. Access to the other namespaces is forbidden.
- Web requests will be sent to a Kubernetes Traefik deployment that manages the ingress traffic.
- Web requests sent to my namespace and traffic path redirection based on each service will be done by a second traefik deployment that acts as a reverse proxy or api gateway.
- Use flux to maintain my Kubernetes namespace state based on a separate Git branch
Traefik
Most of the examples for a reverse proxy/api gateway on Kubernetes are with nginx. I don’t want to use nginx because it misses important metrics. Therefore I have no idea what my users are experiencing. A very good alternative is traefik.
The Cloud Native Edge Router — A reverse proxy / load balancer that’s easy, dynamic, automatic, fast, full-featured, open source, production proven, provides metrics, and integrates with every major cluster technologies
Works way better than nginx, for example if you need to create, disable or update a new traffic rule, you just have to update the kubernetes ingress definition and traefik will detect and update in real time. It also has a cool dashboard. If you want to know more about traefik check their docs section.
Weave Flux
I find the idea of weave-flux very cool, you keep all your Kubernetes objects in a Git branch from which Kubernetes state will be maintained. If you want to know more about flux check their github readme file.
Flux is a tool that automatically ensures that the state of a cluster matches the config in git. It uses an operator in the cluster to trigger deployments inside Kubernetes, which means you don’t need a separate CD tool. It monitors all relevant image repositories, detects new images, triggers deployments and updates the desired running configuration based on that (and a configurable policy).
Gluing everything
- Install and launch Minikube by following the instructions from the Minikube git repo:
- Use my github minikube boilerplate for what follows. Deploy Kubernetes ingress controller that will manage the traffic sent to Kubernetes.
# deploy kubernetes ingress controller
kubectl apply -f 01-k8s-traefik-ingress- Create the developer namespace
# create developer namespace
kubectl apply -f 02-k8s-namespace- Create a new git repo and optional a new dev branch. Then set-up flux to connect to it by modifying the deployment file from
03-k8s-flux-deploy. Deploy flux and add flux ssh key to git repo.
# edit flux deployment and add your own repo / branch
# 03-k8s-flux-deploy/flux-deployment.yamlkubectl apply -f 03-k8s-flux-deploy# get the key and add it to github -> settings - deploy keys
# with "Allow write access"kubectl logs --namespace=reliability-pp deployment/flux | grep identity.pub | cut -d '"' -f2ssh-rsa <...key...>
- In order to send traffic to minikube, you have to update your workstation host database and route all traffic for the minikube domain to the minikube host. Below example is from my machine with the latest mac operating system.
# update your workstation Host Database and route
# traffic to minikube host instead of querying a DNS.
echo "$(minikube ip) traefik-ui.minikube" | sudo tee -a /etc/hosts
echo "$(minikube ip) reliability-pp.minikube" | sudo tee -a /etc/hosts
echo "$(minikube ip) reliability-pp-proxyapi.minikube" | sudo tee -a /etc/hosts
echo "$(minikube ip) reliability-pp-flux.minikube" | sudo tee -a /etc/hosts- Add your kubernetes objects into the dev branch and look how your namespace gets updated.
You can check the state of the namespace by usingfluxctl
fluxctl list-controllers --url http://reliability-pp-flux.minikube/api/flux --namespace reliability-pp
CONTROLLER CONTAINER IMAGE RELEASE POLICY
reliability-pp:deployment/flux flux quay.io/weaveworks/flux:1.4.2 ready
reliability-pp:deployment/memcached memcached memcached:1.4.25 ready
reliability-pp:deployment/reliability-proxy proxy traefik:v1.6.5-alpine readyFlux Up’s
What made me want to test flux:
- Its pull strategy
- Declaratively describe the entire desired state of your system in Git
- You don’t have to know where your app is deployed
- You don’t have to know Kubernetes api’s and passwords.
- If your cluster fails completely you can recover without interacting with the CI/CD pipeline.
- and everything mentioned in flux git repo
Flux Down’s
If you want a simple config, it will work out-of-the-box (sort of …).
- The documentation is too ambiguous, you have to look through a lot of pages to find what you need. I prefer to have a working example on the front page and after that read the docs for advanced configuration.
- The flux api path used by fluxctl is actually
/api/fluxand not the root path/as mentioned in the docs. I have spent 10 seconds to look for the handlers in the golang code, but for some users this might be to complicated. - I noticed it opens one connection to the docker repository for each deployment in order to detect image changes. If flux was depoloyed to kube-system namespace, without any namespace beying whitelisted, it would have detected hundreds of microservices with hundreds of images.
Join us
If you like what we are doing, test new ideas, develop cool stuff in golang and nodejs, you can join the Metrosystems’s SRE team in Bucharest.
