The auto rollout of Kubernetes pods upon change of Hashicorp Vault secrets: kopf operator

Kubernetes has become the de facto platform for running containerized applications, and as the complexity of these applications grows, so does the need for more advanced management tools. One such tool is Kubernetes Operators, which automates the deployment and management of complex applications.

One popular implementation of Kubernetes Operators is the “Kopf” operator. In this blog post, we will see how to roll out pods in a Deployment upon change of secrets in Hashicorp Vault.

Many organisations are using Hashicorp Vault to store their secrets and secrets are being referred to by the pods. All is good! But, there was NO native way to restart pods upon change of secrets linked with respective deployment. In this blog, we will set up an Operator that will monitor the secrets in Vault and restart the respective pods and will update the status in the Custom Resource.

What is Kopf operator?

The Kopf operator is a python-based Kubernetes operator framework developed by Zalando. It provides a simple and intuitive way for developers to create custom operators for managing complex applications in a Kubernetes environment.

The Kopf operator uses a declarative approach to manage Kubernetes resources, which means that it defines the desired state of an application and automatically makes the necessary changes to the resources to achieve that state. This approach allows developers to focus on the application logic rather than the underlying infrastructure.

Features of the Kopf Operator

The Kopf operator has several key features that make it a powerful tool for managing complex applications in a Kubernetes environment:

1. Declarative Configuration: The Kopf operator uses a declarative configuration model, which makes it easy to define the desired state of an application and manage Kubernetes resources using simple YAML files.
2. Custom Resource Definition: The Kopf operator allows developers to define custom resources that can be used to manage complex applications in a Kubernetes environment. These custom resources can be used to define application-specific settings, such as database configurations, application logs, and more.
3. Event-Driven Architecture: The Kopf operator uses an event-driven architecture to manage Kubernetes resources. This means that it listens for changes to Kubernetes resources and automatically updates the state of the application to ensure that it remains in the desired state.
4. Reconciliation: The Kopf operator uses a reconciliation loop to manage Kubernetes resources. This loop continuously checks the desired state of the application and makes the necessary changes to Kubernetes resources to achieve that state.

What exactly we will be doing?

We will be setting up kopf based Kubernetes operator running as a Deployment based Pod which will keep a watch on a Custom Resource with Kind as VaultSyncStatus taking spec values for deployName, deployNamespace, secretEngine, secretPath, and timeStamp. The operator will pick these inputs, communicate with Vault to verify if the secret has been updated (Operator will verify the updated_time on the secret path and if updated_time has been changed then Operator will consider this event as secret change), and if there has been a change in the secrets then Operator will restart the pods of respective deployment mentioned in the Custom Resource spec and upon successful restart, Operator will modify the timeStamp field of the Custom Resource with the last updated time from vault secret path (This field is the source of truth for the Operator as the Operator will compare timeStamp with the updated_time metadata of Vault before every action).

Let’s begin!

1. Clone the repo.

$git clone https://github.com/AA047267/vault-kubernetes-operator.git

2. Create Service Account and Cluster Role Binding, this will provide ‘vault-operator-sa’ SA in the default namespace the admin access and the same SA will be embedded with the Operator deployment.

$cd vault-kubernetes-operator
$kubectl create -f service-account.yaml
$kubectl create -f cluster-role-binding.yaml

3. Apply the CustomResourceDefinition file which defines the Name (vaultsyncstatus), Group(stable.example.com), Version(v1), Specs(deployName, deployNamespace, secretEngine, secretPath, and timeStamp), and Printable columns(Deployment_Name, Deployment_Namespace, Secret_Engine, Secret_Path, and Sync_Status) of the Custom Resource.

$kubectl create -f custom-resource-def.yaml

4. Our Operator will be communicating with Vault to get the metadata of the secrets. We need to create a Vault token that would have access to read the metadata of all secrets and the Operator will be using that Token from a mounted Config Map. The operator will read updated_time from the metadata to identify whether the secrets have been changed. Let’s first create a Vault policy (these steps need to be performed against where Vault is running).

$vi vault-metadata-read-policy.hcl
  path "+/metadata/*"  {
    capabilities = ["read", "list"]
  }

$vault policy write read-metadata vault-metadata-read-policy.hcl
$vault token create -policy="read-metadata" 

Make a note of the token that gets generated. We will use it in the next step.

5. Now, we need to create a ConfigMap that would contain the Token generated in the previous step and Operator will use this mounted ConfigMap to read the token.

$vi vault-token-configmap.yaml (Edit the ConfigMap and mention the Token generated above against the VAULT_TOKEN key)
$kubectl create -f vault-token-configmap.yaml

6. It’s time to deploy our Operator. There is an environment variable VAULT_URL in the deployment file that needs to be changed. Mention your VAULT_URL in that place and then we are good to apply the deployment file.

$kubectl create -f operator-deployment.yaml

7. We are very close to completing our setup. One last step remains, creating a Custom Resource object to let the Operator know what Secret Engine and Path to watch for and what pods to restart. In the metadata of the VaultSyncStatus object, mention the name that you want to provide for the object and the namespace where you want to deploy it. Then in the spec section, mention the name and namespace of the deployment that you want to restart with this Custom object and then provide Secret Engine and Secret Path in the Vault that the Operator will watch every 60s for any change. Leave the timeStamp as it is as this is a placeholder and will automatically change whenever the secret changes in the Vault.

$kubectl create -f vault-sync-status-crd-sample.yaml

You may create as many Custom Resource objects targetting different deployments and Vault Secrets. In order to get the status of each sync, run the command to see the status.

$kubectl get vaultsyncstatus -A (SYNC_STATUS will show the sync status)

Note:

1. Have tested the above setup on Vault 1.12.1 and EKS 1.22.
2. Make sure the readiness probe in Deployment has been configured correctly and has Pod Disruption Budget. This is to ensure that at a given point in time, one or more pods are there to serve the traffic while the restart of pods happening.
3. The first deployment of the Custom Resource object will always restart the pods as we are mentioning a dummy timeStamp and timeStamp will be modified for the Custom Resource with the updated_time metadata of secret.
4. The operator code and Dockerfile are available on the GitHub repo. You may tweak/contribute as per your requirements.