Why it is not about “Helm vs. Kustomize”

8 min readMay 29, 2020

During the last months, I gave a lot of workshops and talks around Kustomize — the newcomer in the area of packaging and deployment tools for Kubernetes. The by far most asked question was “Should I use Helm or Kustomize?”. My short answer is: It does not have to be a binary choice! It is about choosing the right tool chain for your use case at hand, so you might end up using both. Here is my recommendation.

TL;DR

Go for…

Helm, if you distribute your deployment manifests externally, i.e. outside of your team or even your organization, and if it contains a high level of complexity that is worth hiding. Helm is your tool of choice for application packaging and configuration.
Kustomize, if the manifests are either mainly used within your own team or are simple enough (in a positive way) to refrain from an abstraction layer which hides a bunch of magic.
Helm and Kustomize, if the Helm chart you are using does not provide all the flexibility you need! Use Helm for application packaging & configuration and Kustomize for patching the manifests with very specific runtime configurations (e.g. sealed secrets or sidecars).
Your own operator, if you want to automate operational aspects of your application. Most likely this will make the deployment of the operator itself very straightforward.

Let’s have a short look at Helm.

Helm is (or has been?) the de-facto standard package manager for Kubernetes. It uses the Go templating system to resolve placeholders in YAML templates to obtain their intended values during the rendering process.

So, if you want to deploy your application in multiple environments, you need multiple variants for your deployment resource. In Helm, this means creating a template file with placeholders:

Helm calls the bundle of all template files a Chart. Besides some meta data, a chart also contains a values.yaml with default values for all placeholders.

After putting it all together, you can type helm install in your shell to install the chart into your cluster. Alternatively, helm template simply renders the result to Stdout.

For sure, Helm has it advantages:

There is a huge amount of community charts available out there which are really worth looking at: https://github.com/helm/charts
Helm offers a ‘plug & play’ experience for end users as everything is packaged as a deliverable chart and is easy to install/update.
The values.yaml file can be seen as the API of the Helm chart. Only the attributes which are represented by placeholders (and therefore are intended to be changed) can be set by the user. This provides full control of what users can do with it. Think about having to maintain a Helm chart which is used by hundreds of customers… From a support perspective, it simply makes things so much easier if you can consider every deployment as a standardized package. But note that this might become a limitation, too, if your users need more flexibility than you have thought of in advance. We will come back to this in a second.

But Helm also comes with several drawbacks:

Template files can be really hard to read due to the sheer amount of placeholders and control structures
IDE support is limited (think about how helpful auto-completion would be), although basic support is provided in most editors
There is a tendency to make every value a placeholder to provide users with the highest possible level of flexibility. This is also facilitated by the archetype / scaffold which is created by the helm init command. At the end, this results in values.yaml files which have basically the same size as your original plain YAML descriptor. Does this sound like a lightweight solution? Not sure.
Helm charts are hard to maintain, especially when they grow and contain a lot of complex configurations. I have seen Helm charts with 3000 LOY (lines of YAML) where debugging sessions can be really time-consuming.
You can argue whether you want to call Helm a descriptive approach or not. For me, it is not descriptive. The charts simply contain too much complexity and control structures to provide an intuitive understanding about what the result of helm install looks like.

What does Kustomize do differently?

I’d like to introduce Kustomize by a collection of quotes each highlighting important characteristics — especially to understand the discussion “Helm vs. Kustomize”.

Kustomize should expose and teach native k8s APIs, rather than hide them. (GitHub)

So it is not meant to build a custom abstraction layer to hide the actual spec from the user (like Helm’s values.yaml which every user needs to ‘learn’ individually based on the project’s readme file).

It’s actually quite the opposite, i.e. it exposes the Kubernetes API resources to the user. Those resources are well-known, easy to understand and have the same structure for all projects.

The tool provides a new, purely declarative approach to configuration customization that adheres to and leverages the familiar and carefully designed Kubernetes API. (kubernetes.io)
Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is. (GitHub)

The first thing to notice is that Kustomize, unlike Helm, does not understand itself as package manager. It is a lightweight tool to customize raw Kubernetes YAML manifests — hence the name. This is an important difference to understand: Bundling resources as a chart and make them distributable as a package is not in scope of Kustomize.

Ok, let’s have a look at how Kustomize handles the rendering process. In contrast to Helm, it does not use any templating mechanisms. Instead, you can define overlays which are applied to your baseline, plain YAML files.

So you start with writing a plain Kubernetes YAML file — without any special syntax changes, feeling happy about full IDE support while doing so 😇 This is your baseline which belongs in a ‘base’ folder. For all variants, you define overlays, i.e. you create another plain YAML file which consists of

The header stating the base resource to which the overlay should be applied and
The new values of the spec which should override or complement the base definition.

In the following example, the overlay changes the replica count from 1 (base) to 3.

Kustomize comes with additional, handy features which I don’t want to dive into right now. Please find an example on GitHub or consider my more detailed Blog post on my employer’s website (German only — sorry).

What I like about Kustomize:

At any time, you write valid Kubernetes YAML manifests and enjoying full IDE support.
The files are easy to read and maintain.
The ConfigMap Generator is a great feature to trigger the rollout procedure of a deployment once an attached ConfigMap has been changed (a hash value of the content is automatically added to the ConfigMap’s name). The same is true for Secrets.
With custom transformers or the JSON6902 merge strategy, you can implement low-level modifications to make the manifests exactly fit to your needs.
Starting with v1.14, the rendering commands of Kustomize are included in the kubectl CLI and can be accessed via kubectl -k ~/someApp/overlays/prod

And the downsides?

Most community projects use Helm (which is not surprising given the fact that Helm has been around for years). Kustomize might be new to your users.
If you use Kustomize, you still need to think about how to distribute your application (might be a tagged GitHub repo) and how to handle rollouts of new versions safely.
As a vendor, you loose control of how customers are tweaking your manifests. You need to shift your mindset from a fully supported plug&play package to a reference base manifest which users adapt to their needs based on a (hopefully) good documentation. Make sure the users are aware of breaking changes in the way your application is configured because now they are in charge of making sure that all ‘kustomizations’ are still working as intended.

What to use when?

Disclaimer: General advice always comes with a high level of simplification that might not fit to everyone’s use case. Feel free to get in touch with me to discuss yours!

If you want to deploy your own application to your own clusters, use Kustomize. If your application comes with a lot of dependencies, you might consider Helm with its dependency feature.
If you want to provide the deployment manifests to external users (e.g. other teams inside your organization or open source projects) and your descriptors have a low level of complexity, give Kustomize a try. But if you need to hide a lot of complexity, go for Helm.
If you are not only interested in deployments but also want to automate operational aspects of your application, write your own operator. Usually, the complexity is then contained inside the operator which in turn makes the operator deployment very easy to handle (i.e. this is a good candidate for Kustomize).

But wait, in the title you said that it is NOT about “Helm vs. Kustomize”?

Right, although the tools partly cover same areas of functionality, they are also complementary.

I’m sure everyone who has used someone else’s Helm charts came to the point where a necessary modification was not possible. Why? Because the author has not thought of the edge-case at hand and, thus, did not provide any placeholder or hook to add your custom behavior. For example, you want to add a specific environment variable but the chart does not provide a mechanism to extend the predefined list. Or you need to add a sidecar container with a proxy required in your environment. And what about adding runtime configurations like sealed secrets?

This is when the combination of both tools begins to shine! Use the provided Helm chart as far as possible and tweak its rendered output with Kustomize.

Helm and Kustomize used together in a combined tool chain

The following snippet renders the nginx- ingress chart locally and uses the result as a baseline for modifications with Kustomize.

$ mkdir ./base  $ helm template --values ./values.yaml stable/nginx-ingress \
> ./base/manifests.yaml$ cat <<EOF > ./base/kustomization.yaml   
resources:   
— manifests.yaml    
EOF  # create your overlays based on the rendered Helm chart
$ kubectl apply -k ./overlays/prod/

Well, you might think whether this looks a bit too hacky or if it’s actually the way to go? In cases you are using someone else’s Helm chart (e.g. one of https://github.com/helm/charts) and you still need to tweak the output, the reality is that you don’t have much of a choice. You could use sed in shell scripts to manipulate the Helm chart’s result but this is way more error-prone. Kustomize is aware of the Kubernetes API definition and is therefore the tool of choice for safely modifying given Kubernetes resources.

From my experience in customer projects, this combination works quite well. But keep in mind that newer versions of the Helm chart at hand might have a different output. So always check your Kustomize overlays when updating the source.

What’s your current state in your Kubernetes journey? Are you just about to start or already consider yourself as an advanced user? I am happy to discuss your thoughts and questions with you. You can find me on Twitter or LinkedIn.

Also, feel free to contact me in case you need support or consulting services offered via my employer (viadee).