Helm vs Kapitan vs Kustomize
TLDR;
- Kapitan (and presumably Ksonnet) is the more flexible and customizable (json and jsonnet)
- Kustomize if the more straightforward, just released so we’ll need a bit more documentation on built-in functions (yaml only)
- Helm combines a package approach and releases management that is powerful, with the caveats of Tiller for the release management part (additional source of truth)
UPDATE: Good article on managing complexity by Matt Farina https://codeengineered.com/blog/2018/helm-kustomize-complexity/
Overview
During the implementation of https://www.weyv.com environments in Kubernetes, we went through various stages. From plain yaml files, to Helm charts releases and finally helm charts but with helm template output. Now with the announcement of Kustomize, I take the opportunity to re-evaluate our choice of tool vs our requirements with 3 contenders: Helm, Kapitan, Kustomize. I left out Ksonnet (https://github.com/ksonnet/ksonnet) it seems very close to Kapitan.
Helm: https://github.com/kubernetes/helm
Kapitan: https://github.com/deepmind/kapitan
Kustomize: https://github.com/kubernetes-sigs/kustomize
Requirements
We need a tool for managing our Kubernetes resources that allows ut to:
- Specify resources
- Reuse and recombine above resources specifications across environments
- Version control the desired state of resources
Helm
Specify resources
- Set of resources: Chart
- Definitions: Template files, Go template syntax
- Variables: default values per chart, values override, values formatting
Pros: yaml templates are close to the yaml output that is sent to Kubernetes
Cons: No template reuse inside a chart , no custom class definitions
Reuse and Recombine
- Chart can reference and use other charts
- Sub charts variables can be overriden
Version control
- The default usage is to do “helm release” and Tiller (server side in Kubernetes) manages the versions deployed to the cluster
- Though the ouptut of a “helm template” can be checked in version control
Pros: Tiller can manage rollback of releases
Cons: Discrepancies between version controlled resources (desired state) and the version Tiller controls and manages. This created issues several times for us when the state of a resource is modified directly without using Helm and pushed us to switch to “helm template”.
Kapitan
Specify resources
- Set of resources: Component
- Definitions: jsonnet templates, json
- Variables: hierarchical database of variables (inventory classes and targets)
Pros: jsonnet allows class, inheritance, hierarchical database of variable with 2 levels: one for reuse and one target specific.
Cons: json does not look like the generated yaml, json is a bit more verbose to type
Reuse and Recombine
- Everything can be reused and recombined
Version control
- The ouptut of a build can be checked in version control
Kustomize
Specify resources
- Set of resources: Base
- Definitions: plain yaml files
- Variables: properties file can be used to generate configMap (configMapGenerator)
Pros: everything is yaml. input and output look like Kubernetes ready yaml
Reuse and Recombine
- Reuse by default, patch resources in overlays folders
- Recombine easily with declaration in kustomization.yaml
Version control
- The ouptut of a build can be checked in version control
Conclusion
For our current situation moving from “helm template” to kustomize may make sense and require limited investment. Staying with helm, using only template functionality is viable but seems overkill since we don’t even import packages from other sources.
UPDATE: we stayed with Helm and use values file templates to generate values files and feed Helm template to generate our manifests.
Changing to Kapitan would give us more power and flexibility but would require an extensive rewrite of our helm templates to jsonnet.
What is your return of experience?
Feedback always appreciated
