Writing yet another Kubernetes templating tool
If you are working with Kubernetes environment then you probably make use of several existing templating tools, some of them being a part of package managers such as Helm or Ksonnet, or just templating languages (Jinja2, Go template etc.). All of them have their own drawbacks as well as advantages and we are going to go through them and write our own tool that will try to combine the best features.
So, why not Helm?
There are a number of articles criticizing Helm (e.g. just one of them: Think twice before using Helm). The main issue with Helm is that it works with string representations and Kubernetes manifests are (json) objects. The hell for a Helm chart developer begins when (s)he needs to calculate indents for a yaml manifest, sometimes it looks like this (it’s a real example from my chart):
spec:
jobTemplate:
spec:
template:
spec:
containers:
- name: my-awesome-container
resources:
{{ toYaml .Values.resources | indent 14 }}
But Helm today is de-facto the standard of Kubernetes applications packaging. The main advantage of Helm is large community and a big number of public repositories with charts. And recently Helm developers have announced a Helm hub. So Helm today is like Docker — it’s not the only one but it has community and support.
There are promising changes coming with Helm 3 release but nobody knows when it could be.
To conclude, Helm advantages:
- Large community and a number of public charts
- (Relatively) human-friendly syntax. At least it’s yaml + go template ;)
Drawbacks:
- Working with strings and not objects
- Limited number of operators and functions you can use
OK, then maybe Ksonnet?
If you are comparing Helm to Ksonnet the latter has a huge advantage, namely it works with objects. Ksonnet is a tool based on JSON templating language Jsonnet. Another cool feature about Ksonnet is that it has Kubernetes-API-compatible Jsonnet libraries that you can import into your template and work with Kubernetes objects like in any OOP language:
local k = import "k.libsonnet";
local deployment = k.apps.v1beta1.deployment;local appDeployment = deployment
.new(
params.name,
params.replicas,
container
.new(params.name, params.image)
.withPorts(containerPort.new(targetPort)),
labels);
Looks impressive, isn’t it?
It is a little less neat when you are working not with API but with just json objects imported from yaml/json file:
{
global: {},
components: {
"deployment-nginx-deployment-dkecx"+: {
spec+: {
replicas: 10,
template+: {
spec+: {
containers+: [
{
name: "nginx",
image: "nginx:latest",
ports: [
{
containerPort: 80,
},
],
},
],
},
},
},
},
},
}But still it is something and it’s better than working with strings in Helm. The disadvantage of Ksonnet is that it has less community and less packages than Helm (though you can import Helm charts into your Ksonnet project, but you will be working with them as json objects, not as jsonnet-library objects). And as a result of a smaller community and contribution there is lack of some features when you trying to write your own chart. One of them I experienced myself: you know that in Helm you can build up a ConfigMap from a directory containing your config file like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: conf
data:
{{- (.Files.Glob "foo/*").AsConfig | nindent 2 }}You can imagine my frustration when I found out that there is no such a feature in Ksonnet. There are workarounds though. But the point is that it’s just example of the situation when you are happily writing your chart and then suddenly a lack of some feature stops you on the halfway.
In total, advantages:
- Working with objects
- Kubernetes-API-compatible Jsonnet libraries
- Helm chart import support
Drawbacks:
- Smaller community and smaller number of Ksonnet-native packages
- Lack of some functionality you can use in Helm
- New syntax => increased learning time => increased bus-factor
- Syntax can sometimes get ugly and less human-readable (especially when making workarounds for lacking features)
Let’s think of an ideal templating tool
Here some criteria for the “ideal” templating tool:
- It should work with objects, not strings
- It should have a possibility to work with Kubernetes-API-compatible objects
- It should have a decent set of a functions for working with strings
- It should work nicely with json and yaml formats
- It should be human-friendly
- It should be simple
- It should have an ability to import existing Helm charts (because this is reality and we want to make use of Helm community)
That’s enough for now. I went through this list in my head and thought to myself: okay, why not try Python? Let’s see if it fits into our criteria:
- Work with objects, not strings. Yes, we can use `dict` and `list` types for that.
- Have a possibility to work with Kubernetes-API-compatible objects. Yes, `from kubernetes import client`
- Have a decent set of a functions for working with strings. Plenty!
- Work nicely with json and yaml formats. Very nicely.
- Human-friendly. No shit.
- Simple. Yes.
- Ability to import existing Helm charts. That, we are going to add ourselves.
Okay, looks promising. I decided to write simple templating tool atop of Official Python client library for kubernetes and now let me show you what came out of it.
Meet Karavel
There nothing so special or complicated about this tool. I just took Kubernetes library (which gave me an ability to work with Kubernetes objects) and wrote some basic functionality for existing Helm charts (so one could fetch them and add into their own chart). So, lets have a tour.
First of all, this tool is accessible at Github repo and there you can find a directory with examples.
Quick start with Docker image
If you want to try it out, the simplest way is to use this docker image:
$ docker run greegorey/karavel -h
usage: karavelcli.py [-h] subcommand ...optional arguments:
-h, --help show this help message and exitlist of subcommands:
subcommand
template genaretes template
ensure ensure helm dependencies
Of course, if you want to template charts you need to mount your chart’s directory:
$ cd example
$ docker run -v $PWD:/chart greegorey/karavel template .So, let’s have a look at chart structure. It is very similar to one of Helm:
$ cd example
$ tree .
.
├── dependencies
├── prod.yaml
├── requirements.yaml
├── templates
│ ├── custom-resource.py
│ ├── deployment.py
│ └── service-helm.py
└── values.yaml2 directories, 6 files
Like Helm, it has requirements.yaml file with the same syntax:
dependencies:
- name: mysql
version: 0.13.1
repository: https://kubernetes-charts.storage.googleapis.com/Here you just list your Helm dependencies you want to import to your chart. The dependencies go to the “dependencies” directory. To fetch or update them use the ensure command:
$ karavel ensure .After that this your dependencies directory will look like this:
$ tree dependencies
dependencies
└── mysql-0.13.1
└── mysql
├── Chart.yaml
├── README.md
├── templates
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ ├── configurationFiles-configmap.yaml
│ ├── deployment.yaml
│ ├── initializationFiles-configmap.yaml
│ ├── pvc.yaml
│ ├── secrets.yaml
│ ├── svc.yaml
│ └── tests
│ ├── test-configmap.yaml
│ └── test.yaml
└── values.yaml4 directories, 13 files
Now that we ensured our dependencies let’s have a look at templates. First, we create a simple nginx deployment:
from kubernetes import clientfrom karavel.helpers import Valuesdef template():
values = Values().values
# Configure Pod template container
container = client.V1Container(
name='nginx',
image='{}:{}'.format(values.nginx.image.repository, values.nginx.image.tag),
ports=[client.V1ContainerPort(container_port=80)])
# Create and configurate a spec section
template = client.V1PodTemplateSpec(
metadata=client.V1ObjectMeta(labels={'app': 'nginx'}),
spec=client.V1PodSpec(containers=[container]))
# Create the specification of deployment
spec = client.ExtensionsV1beta1DeploymentSpec(
replicas=3,
template=template)
# Instantiate the deployment object
deployment = client.ExtensionsV1beta1Deployment(
api_version='extensions/v1beta1',
kind='Deployment',
metadata=client.V1ObjectMeta(name='nginx-deployment'),
spec=spec)return deployment # [deployment], (deployment, deployment) are valid
So, for the template to be valid you need to have `template()` function which returns either single Kubernetes object or a list/tuple of them. The list of API objects for Python client you can find here.
As you can see, the code is clean, simple, readable. You can wonder where from `values.nginx.image.repository` comes? It gets values from the value files you pass when templating the chart, just like in Helm: karavel template -f one.yaml --values two.yaml ./ We will have a look at them later.
Okay, what about Helm charts?
Now, we created own Deployment. But what if we want to import Helm chart or a part of the chart? Let’s take a look at templates/service-helm.py:
from kubernetes import clientfrom karavel.helm import HelmChart
from karavel.helpers import Valuesdef template():
values = Values().values
# Initialize the chart (== helm template --values)
chart = HelmChart(name='mysql', version='0.13.1', values=values.mysql.helm)
# Get the desired object from chart
service = chart.get(name='svc', obj_class=client.V1Service)
# Create custom objects to add
custom_ports = [
client.V1ServicePort(
name='my-custom-port',
protocol=values.mysql.protocol,
port=values.mysql.port,
target_port=39000,
)
]
# Add custom objects to the service
service.spec['ports'] = custom_ports
# Change Helm-generated label
service.metadata['labels']['release'] += '-suffix'
# Delete Helm-generated label `heritage: Tiller`
del service.metadata['labels']['heritage']return service # [service], (service, service) are valid
Simple, huh? Note this line: service = chart.get(name='svc', obj_class=client.V1Service) — we created object of class `V1Service` form Helm yaml file. If don’t want/need to do that — you can always work with just `dict`.
What if I want to create custom resource?
Well, there is a small issue with that. Kubernetes API doesn’t add CRD objects into swagger json definition at /openapi/v2, and Python-client objects are build upon this definition. But you can still easily work with `dict` objects. Like this:
from kubernetes import clientdef template():
resource = {
'apiVersion': 'stable.example.com/v1',
'kind': 'Whale',
'metadata': client.V1ObjectMeta(
name='my-object',
),
'spec': {
'image': 'my-whale-image:0.0.1',
'tail': 1,
'fins': 4,
}
}return resource # [resource], (resource, resource) are valid
Still looks nice, doesn’t it?
Can I have values for different environments, e.g. dev/prod?
Yes, you can!
Let’s look at `values.yaml` first:
nginx:
image:
repository: nginx
tag: 1.15-alpinemysql:
port: 3307
protocol: TCP
helm:
releaseName: my-release
namespace: prod
imageTag: '5.7.14'
service:
type: NodePort
Note the `helm` key inside `mysql` dict: we used it when specifying values for helm chart. Some Helm charts need `releaseName` for application naming and `namespace` for RBAC policies. These two values are passed to Helm as --namespaceand NAME arguments in `helm template`.
Now, you can specify additional file for prod env, and template all our examples:
$ karavel template -f values.yaml -f prod.yaml .
---
# Source: templates/custom-resource.py
apiVersion: stable.example.com/v1
kind: Whale
metadata:
name: my-object
spec:
fins: 4
image: my-whale-image:0.0.1
tail: 1---
# Source: templates/deployment.py
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx:1.14-alpine
name: nginx
ports:
- containerPort: 80---
# Source: templates/service-helm.py
apiVersion: v1
kind: Service
metadata:
annotations: null
labels:
app: prod-release-mysql
chart: mysql-0.13.1
release: prod-release-suffix
name: prod-release-mysql
spec:
ports:
- name: my-custom-port
port: 3308
protocol: TCP
targetPort: 39000
selector:
app: prod-release-mysql
type: NodePort
After that you can do `kubeclt apply` and deploy this objects to the cluster.
Cool! What about encoding and base64?
import base64What about using Vault for secrets?
import hvacFetching urls?
import importlibSecure hash functions?
import CryptoYou got it. With Python you can do a lot of things with your Kubernetes manifests.
Is it NIH syndrome?
No :)
I am h̶a̶p̶p̶i̶l̶y̶ using Helm in my current projects. There are thins that I miss though. I used Ksonnet in some of my projects as well.
I would like to think of this tool as a proof-of-concept that we can have templating tools better than Helm and it’s not very difficult to do it in Python. If there is a community interest/need in such a tool we can together continue to develop it. Or we can wait for Helm 3 release ;)
Conclusion
I have showed you Python-based templating tool for Kubernetes which has Kubernetes-API-compatible objects support and support for importing Helm charts. Any comments and discussion from community are welcome, and again welcome to the repo.
Thank you and have a good day!
