Kubernetes & Vault

Mike Tougeron
Sep 28 · 4 min read
A bank vault door opening
A bank vault door opening

Two of my favorite pieces of technology have a marriage made in heaven with the vault-secrets-operator from @rico_berger. It allows, in a GitOps manner, the syncing of Vault secrets into Kubernetes secrets. While syncing between one secrets store and another sounds like a waste of bits it can make a lot of sense in some situations. Where I work we run multiple Kubernetes clusters across several regions, both on-premise and in AWS. Getting secrets securely to all of the necessary applications & platforms can be difficult without a good toolset. Vault is an incredible tool that provides a simple, yet powerful, secure method of storing and managing secrets. Enabling Kubernetes applications to authenticate to Vault via a ServiceAccount even allows for secure bootstrapping of the vault-secrets-operator.

Let’s take a common scenario where a group of applications need a password to connect to the database. You could write a library that each of the applications use to authenticate to Vault, read the appropriate secret, use it, and be aware of when the secret changes. That assumes that all of the applications use the same programming language and that you can modify the source code. It’s repetitive, time consuming and it’s not declarative. Instead we can use the vault-secrets-operator and consume these secrets in a Kubernetes native method.

I like Helm because of how easy it makes templating Kubernetes yaml resources. While you could have Helm automatically apply charts to the cluster, you could also use the helm template command to have it render the yaml so that it can be saved and used in a GitOps workflow. Using Helm, with a cluster name variable, the VaultSecret can be created that declares which secret should be added to Kubernetes.

apiVersion: ricoberger.de/v1alpha1
kind: VaultSecret
metadata:
name: myapp-db-creds
spec:
path: secrets/myapp/{{ .Values.global.cluster_name }}/db-creds
type: Opaque

After the VaultSecret has been applied to the cluster, a Kubernetes secret will exist with the values from Vault. The vault-secrets-operator keeps the secret in-sync so if it is modified in Vault, the resulting changes are automatically applied in Kubernetes. If you want to be extra precise, in the VaultSecret you could even define which version of the secret you want applied to the cluster/namespace. This is useful for when you want to stage rolling out the secret rotation or if you need to revert back to an older version of a secret.

At this point, the application manifest simply needs to reference the outcome and use the resulting secret in a Kubernetes native fashion. There’s no logic on the application side to test, validate & debug that it’s getting the correct secret. This manifest is the same no matter which cluster or environment you deploy it into.

apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
spec:
containers:
- name: myapp
volumeMounts:
- name: db-creds
mountPath: "/etc/secrets/db-creds"
readOnly: true
volumes:
- name: db-creds
secret:
secretName: myapp-db-creds

Personally, when I rotate a secret, I want the new value used immediately; I don’t want to require a deployment of the application. An app deployment seems like overkill to me for a simple secret rotation. There could be some debate as to whether or not this approach breaks the GitOps philosophy but I don’t believe it does. The benefits outweigh the concerns especially when dealing with large, slow to start applications.

To support the automatic reload of the secret, the application could accept a SIGHUP or provide a private endpoint (/sys/reload) to be called when the secret changes. Instead of writing a complicated process for watching the Kubernetes objects, you can run a sidecar container that watches for filesystem changes and sends a http request when the file(s) change. I frequently use the jimmidyson/configmap-reload container for this.

apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
spec:
containers:
- name: myapp
volumeMounts:
- name: db-creds
mountPath: "/etc/secrets/db-creds"
readOnly: true
- name: config-reload
image: jimmidyson/configmap-reload:v0.4.0
args:
- --volume-dir=/etc/secrets
- --webhook-url=http://127.0.0.1:8080/sys/reload
volumeMounts:
- name: db-creds
mountPath: /etc/secrets
readOnly: true
volumes:
- name: db-creds
secret:
secretName: myapp-db-creds

The setup of the vault-secrets-operator itself is pretty straight forward. The documentation in the project does a good job of explaining it in detail. But there were a couple of lessons I learned that are worth sharing.

If you are using Vault Namespaces, the authentication path wasn’t what I expected. I had to include the namespace in the path instead of using the VAULT_NAMESPACE environment variable.

env:
- name: VAULT_KUBERNETES_PATH
value: "mynamespace/auth/k8s-cluster1"

The other challenge I faced was due to running many Kubernetes clusters. Each cluster has its own apiserver and certificates which means that each cluster needs a different Kubernetes Auth backend in Vault. While that’s not a bad thing, it did lead to more work setting things up than I expected.

GrepMyMind

Mike Tougeron

Written by

Lead SRE @Adobe , #kubernetes fan & gamer (board & video). he/him. Remember, reality is all in your head…

GrepMyMind

Feel free to grep & grok your way through my thoughts on Kubernetes, programming, tech & other random bits of knowledge. My randomness is my own & not those of any company I might be working for. I may be right, I may be wrong, but as Deep Thought said, “42.”

Mike Tougeron

Written by

Lead SRE @Adobe , #kubernetes fan & gamer (board & video). he/him. Remember, reality is all in your head…

GrepMyMind

Feel free to grep & grok your way through my thoughts on Kubernetes, programming, tech & other random bits of knowledge. My randomness is my own & not those of any company I might be working for. I may be right, I may be wrong, but as Deep Thought said, “42.”

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store