Configuration and Secret Management with Consul Template on Kubernetes
--
As the platform team at Trendyol Tech, one of our goals was making application configuration and secret management dynamic, reliable, secure and be abstracted away from applications. We have been using Kubernetes ConfigMaps and Secrets for a long time and they don’t satisfy our requirements completely. For example; when we change a configuration on a ConfigMap, we need to restart our application pods for this change to take effect. Sometimes it takes several minutes for the change to be reflected on all application instances. Another problem is that Kubernetes secrets are not secure enough. We need our secrets to be secure and auditable.
Our main requirements were;
- We should be able to change applications configurations dynamically without restarting application processes. When we change a configuration, it should effect all application instances instantly.
- Our application secrets must be secure and auditable. We also want to provide role based access control to secrets.
- We need to abstract away configuration management from applications. In other words, applications should not know where and how secrets and configurations are kept and provided for them.
- Our configuration and secret management system must be high available and scalable.
What is Consul Template and why we chose it?
After some research, we came up with 2 applications; envconsul and consul-template.
We already knew that Vault is the industry standard tool for storing secrets. It satisfies all of our requirements and has features like dynamic secrets, which might be useful for us in the future. You can also use Consul as the storage backend for Vault.
Consul is a distributed, highly available and scalable tool for service discovery and configuration. In our case, we will only use the key-value store feature of Consul to keep non-secret configurations.
We could use envconsul or consul-template mainly for the 3rd requirement: abstracting away configuration management from applications.
Envconsul provides a way to launch a subprocess with environment variables populated from Consul and Vault.
Consul-template populates values from Consul and Vault into the file system and keep the values sync while working in daemon mode.
A Twelve-Factor App rule is, store configurations in the environment. But the problem with Envconsul (or environment variables) is that you cannot change a process’ environment variables without restarting it. This goes against to our first requirement. We want to be able to change configurations without restarting application processes. Thats why we decided to go with consul-template.
In consul-template, you can use a template file for rendering configuration files. This gives us great flexibility. Here is a sample template file;
server:
port: "{{ key "apps/example/serverPort" }}"
toggles:
fooEnabled: {{ key "apps/example/fooEnabled" }}
database:
username: "{{ key "apps/example/dbUsername" }}"
{{ with secret "secret/apps/example" }}
password: "{{ .Data.data.dbPassword }}"
{{ end }}Consul-template reads this template, replaces placeholders with values from Consul if you use “key” keyword and from Vault if you use “with secret” keyword. You can find out more about templating language here. The rendered output file would look like this;
server:
port: "8080"
toggles:
fooEnabled: true
database:
username: "username"
password: "password"Making it Work on Kubernetes
So the next question was, how to make it work on Kubernetes? We came with two ideas;
- Running consul-template process in our application containers
- Running consul-template in a separate container in our application pods and sharing volumes between containers for accessing the same file.
First idea is not recommended by Docker and it would be hard to manage application and consul-template processes separately if they work in the same container. It would also make us change applications docker images and add consul-template binary to them.
Second idea is an example of sidecar pattern. With sidecar containers, you can separate responsibilities completely. Service mesh solutions like Istio also uses sidecar containers to manage networking between applications. In our case, we want to make applications know only a file in their file system. Application and sidecar containers can share files using Volumes.
Sometimes applications depend on the configuration files to start up. Kubernetes starts containers in a Pod at the same time. So if the application process starts before consul-template renders the configuration file, it would crash and make the pod restart. We had to make sure consul-template renders the file before application container starts and used Init Containers to solve this problem.
Init Containers are specialized containers that run before app containers in a Pod.
Consul-template can run in once mode if you use -once flag. In once mode, it runs, renders the configuration file for one time and stops. We used an init container with once mode and a sidecar container in daemon mode. This way, when a pod starts;
- Consul-template-init-container runs with once mode, renders the configuration file on shared volume and stops.
- Application container and consul-template-sidecar container start at the same time, application can read configuration file immediately and consul-template daemon keeps file content sync with values in Consul and Vault.
Authentication & Authorization with Vault
Another question was how to make authentication and authorization between applications and Vault. Vault has authentication methods for some third party applications and one of them is Kubernetes. With Kubernetes authentication method, you can authenticate to Vault using Kubernetes Service Accounts. This is a great solution for us because creating service accounts for our apps is pretty easy.
We wanted to limit applications access to secrets and make them read only their own secrets for security reasons. Vault has policies for that. You can create a role in Vault to bind a Kubernetes Service Account to a Vault policy. Our security engineers will create and manage policies and roles to control teams’ access to secrets.
It is really simple to enable Kubernetes auth method on Vault, configure it, create policies and roles using Vault cli. Here are some example commands;
# enable the Kubernetes authentication method
vault auth enable --path="my-kube" kubernetes
# configure the Kubernetes authentication method
vault write auth/my-kube/config \
token_reviewer_jwt="<token_reviewer_jwt>" \
kubernetes_host="<kubernetes_host_addr>" \
kubernetes_ca_cert="<kubernetes_ca_cert>"
# create a policy named example-policy that enables the read capability for secret at path secret/apps/examplevault policy write example-policy - <<EOH
path "secret/data/apps/example" {
capabilities = ["read"]
}
EOH
# create a Kubernetes authentication role named example-role to bind service account in K8s to policy in Vaultvault write auth/my-kube/role/example \
bound_service_account_names=example-sa \
bound_service_account_namespaces=test \
policies=example-policy \
ttl=24h
After this configuration, you can login to Vault using k8s service account and get a Vault token with read access to “secret/data/apps/example” path. An example request to Vault API to get a token looks like this;
$ curl \
--request POST \
--data '{"jwt": "<example-sa-jwt>", "role": "example"}' \
http://<vault_address>:8200/v1/auth/my-kube/loginThe response will contain token at auth.client_token path. We provide this token to consul-template when starting it up.
To make this setup easy to implement, we created a Mutating Admission Controller on Kubernetes to inject consul-template containers to our pods using only a few annotations on applications deployment specs. My colleague, Batuhan Apaydın, will write a story about it soon.
Next, I’m going to show you a demo. I will deploy Consul, Vault and a demo application with consul-template sidecar to Minikube and wire them up all together. The application is a simple web api with a single http endpoint which returns configuration values read from a file. You can check the code here.
Demo:
In the demo I will do the following step by step;
- Deploy Consul and Vault to Minikube on my local machine using helm charts
- Write key-values to Consul using Consul cli
- Configure Vault and create a secret using Vault cli
- Deploy demo app with consul-template sidecar
- Http request to demo app and get configurations
- Change a value on Consul and request again to see the value changed instantly!
You can check the commands I use at https://github.com/Trendyol/consul-template-demo/tree/master/scripts
You can watch the demo at https://asciinema.org/a/bjPQqdD0oIKN3bquCu5ia3yQ9
