Mount Environment Variables Safely with Kubernetes Secrets and Helm chart

Prerequisites

This article assumes that you have basic understanding of Kubernetes and Helm charts.

Introduction

Kubernetes has quickly become one of the most popular go to solution for deploying and managing complex docker based micro-service architectures.
With its replication controller managing the desired number of replicas, running and auto scaling capabilities, more and more organisations are switching their architecture into using Kubernetes.
But as the components in the architecture grows, it soon becomes quite clumsy to manage and update so many .yaml files. Furthermore, Kubernetes does not provide any feature to remember historic deployment states.
To mitigate these problems, entered the perfect partner for Kubernetes — Helm Charts

Why Helm charts?

Helm charts are basically a way to organize and maintain Kubernetes related resources. It let’s you “code your configuration”. It has gained popularity due to two main reasons:

1. Code your configurations: Helm lets us use many of the programming language like constructs in our .yaml config files.
Techniques like looping on an array, conditional (if-else) blocks, and reusable methods (template helpers) can be used to add dynamic configurations to our Kubernetes cluster.

2. Maintaining History or Releases: What git does to our code, helm does to our releases. It keeps track of previous versions of our Kubernetes deployments.
We can easily roll back to any earlier release version and come back to the latest version with simple commands.

Mounting Environment Variables in a Kubernetes deployment

Now as we know, Kubernetes uses Secrets to mount sensitive data as environment variables in a Pod.

Kubernetes secret objects let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys.

A typical pod deployment using a secret to mount environment variables may look like this:

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: PORT
        value: 3600
      - name: MAX_THREADS
        value: 5
      - name: DB_USERNAME
        valueFrom:
          secretKeyRef:
            name: app-env-secret
            key: username
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: app-env-secret
            key: password
  restartPolicy: Never

Here we mount two sensitive environment variables DB_USERNAME and DB_PASSWORD to the pod and the values are taken from a secret named app-env-secret .

There are also 2 non sensitive environment variables PORT and MAX_THREADS

There are two main problems with this approach:

1. To mount one env variable,we need 5 lines of YAML content. So if we have 20 sensitive env variables to mount, we would end up having 100 lines of YAML just to mount variables.
2. The configuration is not reusable. If we have another pod which needs same set of variables, then we would again have to write those 100 lines in a new YAML file.

Helm Chart provides a neat way to mitigate this problem. we can write reusable helpers in helper.tpl file and then call the helper like function anywhere.

Writing your custom helper in helm chart

Now before you start writing your reusable helper function there are few things you need to take care of.

1. List out Sensitive and Non Sensitive Environment Variables in values.yaml file: Recognise which are the non sensitive env variables(eg: port number, max_threads, etc). Those can be directly mounted by name/value tags in YAML. The sensitive env variables that hold information that can’t be disclosed will be first stored in a secret and then their value should be fetched from secret and mounted in pod.

Add a new block env in your values.yaml file and add all your environment variables as in the block as below:

secret:
  name: app-env-secret
# Environment variable listing
env:
  # non sensitive variables
  normal:
    AZURE_SERVICE_BUS_DOCUMENT_QUEUE_NAME: "demo"
    PORT: 3600
    MAX_THREADS: 5
  # sensitive variables
  secret:
    DB_USERNAME: "demo"
    DB_PASSWORD: "demo"
    GRAPHQL_ADMIN_SECRET: "demo"
    AZURE_STORAGE_ACCOUNT_NAME: "demo"
    AZURE_ACCOUNT_ACCESS_KEY: "demo"
    AZURE_SERVICE_BUS_CONNECTION_STRING: "demo"
    KEYCLOAK_BASE_URL: "demo"

2. Gitignore your values.yaml file: This is the MOST IMPORTANT step. Once you add your sensitive data in your values.yaml file, you should gitignore the values.yaml and place a sample.values.yaml instead having fake values so that other team members are aware of the structure of the values.yaml file.

Add following line at the bottom of your .gitignore file

charts/values.yaml

Remember, this file is now gitignored and would have to be safely placed in your server while deploying your helm chart using helm upgrade command.

3. Create a secret from env.secret list in values.yaml: create a file templates/app-secrets.yaml and paste following content in it:

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.secret.name }}
type: Opaque
data:
  {{- range $key, $val := .Values.env.secret }}
  {{ $key }}: {{ $val | b64enc }}
  {{- end}}

The above code, iterates over env.secret list in values.yaml and create a secret with the key and values in the list.

Notice how it uses b64enc pipe to base 64 encode the values before adding them in secret. You can find more about base 64 encoding of secrets here.

Using this secret, we will safely mount the sensitive data as env variables

4. Write your helper function: In your templates/helpers.tpl file, add following helper at the bottom

{{- define "helpers.list-env-variables"}}
{{- range $key, $val := .Values.env.secret }}
- name: {{ $key }}
  valueFrom:
    secretKeyRef:
      name: app-env-secret
      key: {{ $key }}
{{- end}}
{{- range $key, $val := .Values.env.normal }}
- name: {{ $key }}
  value: {{ $val | quote }}
{{- end}}
{{- end }}

Now let’s see what’s happening in the code above.

firstly we defined a reusable helper by define keyword

{{- define "helpers.list-env-variables"}}

Then we use range to iterate over env.secret and env.normal list from values.yaml file. For sensitive data, it mounts the value from secret using secretKeyRef keyword.

For non sensitive data, it mounts the env variables simply by name and value keywords.

Usage and working of the helper

Once you’ve written your helper it’s time to test it out in your pods and deployment.

In your templates/pod.yaml call the helper instead of manually mounting the env variables:

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
    {{- include "helpers.list-env-variables" . | indent 6 }}
  restartPolicy: Never

Notice that I have used the indent 6 pipe just to make sure that the output generated by the helper gets indented by 6 characters. Calling the helper with correct indentation also works fine, but has a risk involved of creating wrong indented yaml.

Conclusion

The result would be neater and shorter files for deployment and pods in our helm chart. Only one line added in yaml instead of 100s of clumsy repetitive lines of config.

There are so many other features that can be achieved by using custom helpers. You can find the documentation here.