Kubernetes and the Encyclopedia of Install Technologies

6 min readMar 3, 2023

If you have ever worked in the Cloud and/or Kubernetes space, then I am sure you have seen a project that has an installation guide the size of an encyclopedia because the project allows installation through a number of technologies. In my experience as a DevSecOps practitioner, I have discovered this pattern has become all the more common over time. At its root, install flexibility from a project can be a good thing because it is easy to plug and play with your existing pipeline. However, let’s assume you are creating a CI/CD pipeline for your enterprise and are attempting to navigate this wealth of install technologies and don’t know where to start. This article aims to give pros/cons and my personal opinion on a few popular install technologies, so that you can chose the right one for your enterprise.

Overview

I will call out the install technologies this blog will investigate:

Standard kubectl CLI interaction with raw k8s manifests
Kustomize
Helm
Operators and Controllers

It is important to remember that there is no single install technology to rule them all, and each method called out merits its own benefits. However, there are a few key tenants for installation that we can abide by, no matter the install technology used:

Repeatable (the process can be repeated multiple times and result in the same end state)
Auditable (there is a clear audit trail built into the change process)
Securable (there exists a safe way to store and mount secrets)
Maintainable (the more you update a project, the quicker the next update should be)

Kubectl

The kubectl command line tool marks the basis for all k8s interactions, and is a common starting point for installation. The CLI tool allows a user to apply a set of raw k8s manifests directly to the cluster. An example command would look like kubectl apply -f my-application.yaml and this is an extremely powerful tool. You can get up and running in a k8s environment in the matter of seconds with the right manifest. So what’s the catch? The catch is what happens tomorrow. If I need to make a change, then this is simple enough, I just update the manifest file and check it back into git. However, what happens when I need to deploy another application? Most likely 1 of 2 scenarios would occur:

I would copy and paste the existing manifest, save it as my-application-2.yaml and then make small delta changes (like image) there.
I would take a templated base and define a script that deploys my 2 applications based on some configurations I provide.

In scenario (1) you would quickly find that this is not scaleable and over time opt for scenario (2). However, why reinvent the wheel? There are other tools like kustomize and helm that were designed exactly for this use case.

Kustomize

Kustomize is a great stepping stone in the k8s install ecosystem. It allows you to define a base for what an application looks like, and then publish deltas called overlays that define specific application deployments. Kustomize also lends itself greatly to gitops deployment pipelines because the entire cluster state is stored directly in git. I like to think of kustomize as kubectl apply++ because you get every feature base kubectl gives you, plus the ability to manage a well defined set of objects and components to reduce the amount of copy/paste and/or scripting needing. So what’s the catch? Kustomize is a great tool for static manifests that do not change from environment to environment (namespaces, RBAC rules, service accounts, inception manifests), but when you add variability to the resources this leaves something to be desired. In my experience for an enterprise pipeline, you still need to create some type of templating engine that sits on top of kustomize to deploy applications e2e. This templating engine usually looks like a combination of environment variable configurations that are read with sed to replace specific parts of the kustomize files. This detracts from a gitops mentality because the final cluster state now depends on an external configuration factor not held in git. Luckily, we do not have to reinvent the wheel here either, and we can look to helm as a solution.

Helm

Helm is a great tool that takes the best aspects of kustomize and layers a built in templating engine to the manifests. A helm chart allows you to define an application as a self contained atomic unit of installation. You define the k8s manifests once, and template out any information that is overridable by the consumer. Helm charts even extend the component functionality of kustomize and you can define aggregations of charts through a parent/subchart relationship in order to define install dependencies. So what’s the catch? There are 2 catches here: day 2 operations and getting up and running.

Helm is a great tool for initial project inception (day 0) and ensuring an application is installed correctly. However, the DevSecOps team will still need to define a set of day 2 operations (DR, backup/restore, roll-forward, etc…) that are manually triggered through a runbook based on some alerting metric.
There are a lot of components that could go into a helm chart and it is not always a trivial exercise. Sometimes it is much simpler (and fits the use case better) to just define a kube manifest or a kustomize overlay and apply it directly to the cluster.

Note: I recommend using at least Helm version 3 when using the technology. Helm 2 had security concerns in a multi-tenant environment through the use of the Tiller server agent. Helm3 removes Tiller completely and inherits all permissions from the caller’s kube config file.

Operators and Controllers

An Operator is a controller + a custom resource definition. A controller is a piece of software that constantly watches the cluster state for deviations and then remediates the deviation (imagine having a member of the operations team always on watch). A custom resource definition (CRD) is an extension to the k8s API, and defines a new manifest that can be applied to etcd. An operator acts on a CRD to extend the k8s control plane, in order to bolt onto k8s custom cluster state handling. Operators are a shift in paradigm from a helm chart or a k8s manifest because they are not gitops friendly. An operator responds to an instance of a CRD and generates the cluster state from that custom resource (CR). This means that the end cluster state is not known until after an operator performs a reconciliation on the CR against the existing cluster state. Operators also have a steep bar of entry just to install an application on a cluster (you must define an entire API to respond to and write the actual go code that handles the response). So is an operator worth the effort, what’s the benefit? If you are only concerned with day 0 operations, an operator is not worth the effort because you could define a helm chart much simpler and quicker. However, if you are concerned with day 2 operations, then an operator is most definitely worth the effort of development. An operator shines tomorrow, after an application is installed. It watches the cluster state for drift and automatically re-enforces the correct state when any is detected. Advanced operators can also use machine learning side cars to detect problems with the application and apply remediations (think auto-pilot mode). There are even some operators that act as global load balancers in an HA/DR configuration. Operators are extremely powerful tools that allow an operations team to codify their practices in a 24/7 sentinel for the cluster. These operations are also subject to a CI test pipeline and the entire e2e controller functionality can be codified in unit tests.

The Conclusion

We have discussed many install technologies and some pros/cons to each, but the question still remains “Which one should I use?” In my professional opinion we can use the following as a guideline:

If you have a single application to deploy, then a k8s manifest handled through a kubectl apply is probably the best fit.
If you are deploying multiple applications, or an application in different states, then using a kustomize pipeline is probably the best fit.
If you are deploying multiple applications with many configuration knobs, then a helm chart if probably the best fit.
If you are concerned with day 2 operations and streamlining that process, then an operator is probably the best fit.

So in conclusion, is there one install technology to rule them all? Unfortunately, the answer is no, this is a grey area. In my experience, I find a combination of kustomize and operators will solve my use cases. I use kustomize to define initial cluster state (namespaces, RBAC, OLM artifacts, security enforcements, etc…), and then I use an operator to manage the installation and day 2 operations for my applications. This becomes a balancing act of time spent on development against the return you see from this development in production.