How to make GitLab Kubernetes GKE deployments in a flexible way including 100% GitOps?
Well, gomplate }} and some ideas are all you need!
Helm and Kustomize are great tools for Kubernetes to create application packages and application deployments. But, there are situations or circumstances where already existing tools cannot help you. Every environment is different and so is every company or enterprise. Therefore default solutions or standard tools do hardly fit the specific needs you might have and the same is true for us. In this story, you will get information and ideas about a flexible and 100% GitOps related way you can adopt for you too.
What you need to achieve this goal is a tool called
gomplate and some small scripting skills. The
gomplate tool was invented by Dave Henderson and it is very useful in many different ways because mostly any tool needs some kind of configuration. These configurations are often static which means that there are no options to dynamically change the final configuration based upon environment variables or data-based information in general. And obviously that is true for Kubernetes yaml-files. I know that there is the
kubectl edit command, but this is not what you want to use because it changes the running configuration without leaving any clue. If you are targeting a GitOps workflow, where every change must be Git committed, this will and should be a no-go!
Therefore we need some small gluing between the GitLab pipeline and the Kubernetes infrastructure. In this case, we are using Google GKE, but this will work with ever Kubernetes infrastructure because at last, we are not using more than
kubectl apply -f commands. But what were the points why we are doing and why we want it this way?
Overengineering is a common problem today. Often a tool-chain is designed to fulfill a workflow. Something that my team and I learned in the last 20 years is, that if someone is talking about a chain of tools or a tool-chain something might be already over-engineered. You should step back and ask yourself if there is a more KISS like approach for solving the situation. That’s what we did. Why? Our first thinking about solving the GitOps driven deployment with Kubernetes was to use Helm. But after some time we figured out that for us Helm won’t be enough and we need Kustomize too. And after some more time, we recognized that we might need additional tools! A tool-chain was born.
But seriously, for what benefit? What all tools do in the end is to create yaml files that are finally applied with
kubectl apply-f! That’s all, no magic here!
Our self-defined goals and guardrails for this approach were the following:
- No over-engineering, avoid too many abstraction layers
- Easy to understand for everyone who is using it
- Easy to maintain centrally
- Flexible enough to fulfill different deployment types now and in the future
- 100% GitOps — no manual side-cheating
- 100% reproducible and replicable
Therefore, we created a workflow where
gomplate, a small Golang powered web-service, and some lines of Bash scripting is enough.
Step 1: The pipeline part
As written above, at the end you need a set of yaml-files which are applied with
kubectl apply -f and therefore the first step is to generate them.
We are using a central template for our GitLab pipelines. This makes it easy to change the functionality within a central place and without changing every depending GitLab project which is using the .gitlab-ci.yml. The following picture shows everything that is needed to run GitLab deploy pipelines against a Kubernetes cluster from a GitLab project.
KUBERNETES_DEPLOY variable inside the GitLab .gitlab-ci.yml controls what stages and jobs are running from the central GitLab CI template. In this case, the following “code” is running:
As you can see, the
KUBERNETES_DEPLOY variable from the .gitlab-ci.yml is reflected in the
rules section. When the
stage: deploy runs, the
kploy repository is pulled, which will to the rest. The
kploy.sh script, which comes from the pulled repository, is started with the build Docker image tag as a parameter and that's it. As you can see, both, the central template repository and the kploy repository are public! There is no sensitive information inside the templates and therefore they can be public and they should be public because it makes the use of them easy. There is no need to authenticate against the GitLab repository and everyone can “see” what is going on there!
In addition, the
kploy.sh script will take information from a folder that is called
kploy-customize which includes some important files and is located inside the deploy repository, as seen in the next screenshot.
Inside this folder, there are two important pieces of information. First, a file called
data.yaml which is mandatory and includes information about the deployment and second, a folder called
data.yaml contains information that is later used to fill the
kploy templates which appropriate values and also defines which template should be used for the deployment.
We’ve defined some mandatory variables. The most important ones here are the
kploy_deploy_template , the
k8s_deployer_endpoint and the
k8s_deployer_apikey. The labels, the namespace, and the other variables are depending on the
kploy_deploy_template that is used for this deployment. In this case, we are using our own ingress controller based on the Kubernetes operator pattern. This system is very flexible because there can be many different deployment types with different settings which can be configured centrally.
yaml folder can contain any yaml-file which should be applied during the
kubectl apply -f run in addition. This allows the user of the system to inject yaml-files which might be needed by the software itself, like Kubernetes
secretes or anything else which is already supported by the Kubernetes API or will be supported in the future. There is no dependency on what Kubernetes version you are using or which additional components are installed in your Kubernetes cluster — maximum flexibility is guaranteed!
Step 2: The kploy repository
kploy repository, which is pulled in the script part of the .gitlab-ci.yml, are the templates located which are managed centrally. The templates are located inside the
kploy/deploy_type folder, the selectable templates are located. As seen in the example above, we have used the
hci-bosnd-3_medium-v1 template. This template contains
environments like test, dev, staging, prod or whatever you like. The
prod.yaml defines the numer of replicas, the image and the resource that are allowed for this type of template. The
components for the
prod environment are containing the Golang template based Kubernetes yaml-files. This files will then be processed by
gomplate with the data given by the
data.yaml and the data that is stored in the environment, in this case in the
As an example for a Kubernetes deployment.yaml we can have a look into the file 40-deployment.yaml:
Ok now, that we have all that we need, we will take a closer look into the
kploy.sh script and the
kploy.sh script picks up the
data.yaml and the
run-tpl.sh and renders out the final
run.sh . This is needed, because the
run.sh is environment specific and therefore the
run.sh needs to know what to data environment is used!
run-tpl.sh will do the rendering of the template files which are in the specified template, in our case inside the
hci-bosnd-3_medium-v1 (see above) folder. At first, it will render out the
environment.yaml and based on this file, everything else is processed by
gomplate , including custom yaml-files if some are existing.
In the end, the result is stored inside a tar-ball which is then delivered to the deployer service of the specific cluster. Everything, the source, and the result will be stored as GitLab-artifact and stored for 100 years. At any time, someone could take the yaml-files which are inside the GitLab-artifact-file and make a deployment with
kubectl-apply -f! It is fully complete, nothing is missing and the GitOps idea is fulfilled!
Step 3: The deployer
The deployer is a simple service, which is running inside the Kubernetes cluster. It retrieves the upload (tar.ball)from the GitLab pipeline run extracts the content, does some security checks, and applies (if dry-run is false) the yaml-files with
kubectl apply -f . We’ve established a system that uses a combination of Kubernetes-namepsaces and API-keys to verify if the deployment is permitted into the given namespace or not. But you can do whatever you like here. Our service is implemented as a simple Golang-based web service but you can do the same with some lines of Python-code or whatever you like too.
The result is given back to the calling pipeline via the http-response and is therefore visible in the output of the GitLab pipeline as you can see in the following screenshot.
The ordering of the yaml-files which are applied with the
kubectl apply -f command is determined by the file-name numbering. Lower numbered files will be applied first — simple and comprehensible.
Bird Picture of the deployment process
In this picture, you see the whole deployment process steps including the information from above. It should give you an idea of how you can do something similar.
With the showed idea, it is possible to do highly flexible deployments with a lot of cool features Multi-Cluster, multi-repository, multi-branch, multi-template, and multi-environment support.
- Full GitOps driven, everything is stored as artifacts
- Just 80 lines of script code
- Highly transparent
- Everything can be used which your Kubernetes cluster supports, no limitation from a
- Only depends on Bash and
- Changeable and expandable at any time
Hopefully, you can benefit from this story and if you like, leave a comment!
Last edited on 22th September 2020