Move Your Certs to Helm
TLS is a pain. Setting up certificates is delicate work, and if you mess something up you need to start from the beginning most of the time. While I strongly encourage encryption and security everywhere possible, sometimes you just need a simple self-signed certificate for a trusted backend service. We had exactly this case recently, here’s how we did it using Helm.
Some Background
We’ve been working on setting up the k8s-prometheus-adapter in our clusters in order to be able to use custom metrics for our horizontal pod autoscalers. The process requires that you’ll have Prometheus for monitoring (which we already have), setting up a custom server and APIService, and other subsequent tasks (this great post by Weave explains it in greater detail, and the example repository can get you up and running in no time). While the guide and setup instructions are pretty simple to follow, once that’s done, we immediately began thinking about packaging for easier distribution and installation on other clusters.
If you’ve been doing any amount of non-trivial application deployments with kubernetes, you are probably using Helm. Helm makes it easy to group multiple kubernetes configuration files to a package (chart in helm jargon) for easy distribution and installation. Using the repository as reference, “chartifing” the configurations wasn’t a big effort, as most of the resources are pretty straight forward.
But there’s a caveat.
Taking a closer look at the example repository, you’ll notice that before installing the custom metrics server, you need to create some self signed certs and place them in a Secret in order to establish trust between the k8s-api-server and the custom-metrics-server . While the example project has a makefile that makes it easy to create those certs, it’s one more thing to remember before installing the chart, and doesn’t really create that self-contained package that we like about Helm so much.
The Spring Crypto Library
Helm has a lot of built-in function that help create flexible and powerful templates. Some of those functions are part of the Go template language itself and some are part of the Sprig template library.
Sprig comes with some handy crypto functions, such as sha256sum, genPrivateKey and even handier in our use case — genSignedCert. Combining the crypto functions with Helm’s built-in templating features will allow us the generate that CA we need as part of the chart itself.
In order to move the certs from the makefile to Helm, we’ll need 2 things:
- A named template (partial) the we’ll use to create the tls cert and key
- A secret that will hold the generated cert and be loaded to the custom metrics deployment
The partial
You’re probably already using Helm’s partial files. Those are the files that begin with underscore (_) and end with .tpl extension. These helpers are used to define named templates (partials) which are embedded in several places inside your Helm templates. One of the most common one is chart.name which is used in labels and resource names. Our custom metrics _helpers.tpl looks like this —
The first two partials (custom-metrics.name and custom-metrics.chart) are pretty standard and you can find similar ones in almost every chart.
The third one (custom-metrics.gen-certs) is where the TLS magic happens.
We begin with creating a list of alternate names for our server which will be used later in the signed certificate (line 19). The names in the list correlate to the Service names that will be used in the custom metrics chart. Using Sprig’s genCA function we create a new self-signed x509 certificate authority, which we name custom-metrics-ca (note that the cert is valid for 365 days).
line 21 actually creates the final certificate itself. Again, using Sprig’s genSignedCert function, we pass the named partial, an empty IP list (the nil part), the alt names we defined earlier, validity and the CA we created in line 20.
The genSignedCert function creates an object with a pair of items in it — the Cert and Key which we base64 encode and use in a yaml like object (we’ll use that in a bit)
The Secret
To hold the signed certificate and key we created above, we can use a kubernetes Secret object. Secrets are great when working with sensitive data (such as passwords, token or certs). Using our cert partial, the Secret will look like -
Most of the fields are pretty straight forward, but there are a couple of things to note here —
We’re using Helm’s hook annotations in order to ensure that the certs will only be generated on chart install. This will prevent overriding the certs anytime we upgrade the chart’s released instance.
Inside the data block, we’re including the partial we created earlier. It’s already structured properly (the last two lines of the partial) and base64 encoded. We just make sure it’s indented properly using indent 2 as yaml is picky about indentation.
That’s that. In order to view the compiled template, we can run helm template -x templates/secret.yaml <chart-folder> from the chart folder’s root, the output should look like this —
# Source: custom-metrics/templates/secret.yaml
apiVersion: v1
kind: Secret
type: kubernetes.io/tls
metadata:
name: custom-metrics-certs
labels:
app: custom-metrics
chart: custom-metrics-0.2.1
heritage: Tiller
release: RELEASE-NAME
annotations:
“helm.sh/hook”: “pre-install”
“helm.sh/hook-delete-policy”: “before-hook-creation”
data:
tls.crt: <REDACTED>
tls.key: <REDACTED>I’ve redacted the base64 parts for brevity, but you’ll see them in the compiled template.
The last piece of the puzzle is consuming the secret itself inside our deployment, which should look like this —
You’ll notice that we’re mounting the certs secret as a volume and than passing the adapter args pointing it the cert and key files, which we’ve prepared in our partial.
The beauty of using the partial to generate our certs is double —
- We get a unique certificate and key for each chart release. No more storing key files in the source control or sending them over emails
- We removed the dependency in the makefile or any additional step for generating those self-signed certs pre-installation
A Word of Caution
While generating certificates this way has many advantages, please remember that those are still self signed certs, meaning that browsers will complain when they load them. The above method might not be adequate for any human facing service. If that’s your case, go ahead and create a formal CA signed certificate.
Conclusion
Helm has many functions and features that make it flexible and powerful. Utilizing Sprig’s additional capabilities allows us to create complex charts for truly self contained application packages.
You can find the complete chart in my GitHub repository, along with the other required resources we haven’t covered here.
