Using Kubernetes Secrets

Manage, stage and automatically update your application’s production-level environment variables and sensitive files using Kubernetes Secrets

There are many ways to manage your production-grade secrets (sensitive data such as tokens/keys/passwords/etc) in your applications. Regardless of how you or your company prefer to manage them, Kubernetes has it’s own take on managing your secrets for you — and if you’re working with cloud native applications, it’s especially worth taking a look at.

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in an image; putting it in a Secret object allows for more control over how it is used, and reduces the risk of accidental exposure. — Docs

This guide is to get you started with Kubernetes Secrets. We’ll look into storing two types of common production variables using Kubernetes Secrets:

1. Environment Variables
2. Mounting to Files (key files/ssh certs/folders/pictures of your cat/etc)

The Basics

Now there are two ways to create a Secret:

1. Using kubectl
2. Using a .yaml file (Just like another K8s Object)

We’ll go through both methods.

Method 1 : Using kubectl

Probably the easiest and the most hassle free method to generate your secrets. (This will also automatically generate a .yaml file that you can use later if you want).

For demonstration, I’m going to work with a simple application that requires 3 sensitive data pieces:

1. A file that should be present in /root/ of the Container. (Mounting a secret file when the container is created)
2. Two environment variables. (Loading OS environment variables to the Container — similar to using the ENV in your Dockerfile)

Copy over the secret file (Let’s call this key.json) to a machine with access to kubectl that has access to a cluster you can work with.

Now in the terminal let’s run a simple command to generate our secret

$ kubectl create secret generic my-secret \
--from-file=service_account_key=key.json \
--from-literal=webhook_token=sdfdgerww4dhgsf643 \
--from-literal=slack_token=sffrt64t7uk

If you want this secret to be added to a specific namespace or context add the --namespace or use-context arguments to this command. (Otherwise it will be added to the default namespace)

This creates a Kubernetes Secret object named my-secret and the secret will have 3 key-value pairs. (Note the Key-value pairs).

Use --from-literal to create a secret for any environment variable and --from-file to create a secret from any file.

Viewing your secret

Use $ kubectl describe secret my-secret to view a summary of your secret. Using kubectl get secret or kubectl describe secret won’t reveal the information in the secret.

Name:         my-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>
Type:  Opaque
Data
====
service_account_key:  38 bytes
slack_token:          11 bytes
webhook_token:        18 bytes

Now we have our secret! The next step is to mount these secrets into our application by changing a few lines of it’s deployment.yaml.

Method 2 : Using a custom .yaml file

Before we move onto our application’s deployment.yaml. Let’s take a quick detour to checkout how to make a secret using a .yaml file.

— Personally I prefer to create secrets using the kubectl create secret command since it deals with the encoding and makes the process that much faster and less troublesome. You’ll see why in a bit.

Encoding? What encoding?

K8s secrets are all about that Base64 encoding.

Let’s extract the yaml file of our secret we just created through kubectl.

$ kubectl get secret my-secret -o yaml

This gives us the secret my-secret we just created in yaml format:

apiVersion: v1
data:
  service_account_key: eyBoZWxsbzogInNkZmFzZCIsCnBhc3N3b3JkOiAid2hhdCIgfQo=
  slack_token: c2ZmcnQ2NHQ3dWs=
  webhook_token: c2RmZGdlcnd3NGRoZ3NmNjQz
kind: Secret
metadata:
  creationTimestamp: 2018-01-04T07:19:33Z
  name: my-secret
  namespace: default
  resourceVersion: "2453821"
  selfLink: /api/v1/namespaces/default/secrets/my-secret
  uid: 9ce562e7-f11f-11e7-a5c3-42010a9800e4
type: Opaque

Interestingly enough, you might have noticed how all our values are encoded! — Kubernetes does this to avoid accidentally exposing secrets.

Kubernetes encodes all it’s secrets in base64. So if you want to create a secret from a .yaml file, you need to manually encode all the values in base64 — or it won’t accept the values!

Thankfully though, encoding in base64 is pretty easy. Just run the following command with your secret value to encode it:

$ echo -n some_text_to_encode | base64
c29tZV90ZXh0X3RvX2VuY29kZQ==

— Don’t drop the -n flag, you may run into issues if you drop it because k8s secrets don’t play nice with newline characters.

Similarly you can decode a base64 encoded text as well:

$ echo c29tZV90ZXh0X3RvX2VuY29kZQ== | base64 -d
some_text_to_encode

Writing our own Secret.yaml file

Let’s create our own my-secret.yaml file.

Remember, we need to manually encode all the values in base64 before we add them to this file! — This is why I prefer the kubectl method we discussed earlier.

my-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: default
type: Opaque
data:
  service_account_key: eyBoZWxsbzogInNkZmFzZCIsCnBhc3N3b3JkOiAid2hhdCIgfQo=
  slack_token: c2ZmcnQ2NHQ3dWs=
  webhook_token: c2RmZGdlcnd3NGRoZ3NmNjQz

— Add your secrets (key-value) pairs under data:

Warning regarding encoding FILES with base64

You can’t use echo <filename> | base64 to encode a file! You must manually extract all it’s contents and then encode it and this in my experience can be a very troublesome. It’s always better to use kubectl to create your secrets from files.

Now create this Secret by running:

$ kubectl apply -f my-secret.yaml
secret "my-secret" created

Adding our secrets to our app’s Container

Now that we have our secret, let’s make a few modifications to our Container’s deployment.yaml to use the secrets we need.

There are two things to do as discussed earlier:

1. Load (Mount) the file (key.json) to the Containers /root/
2. Add the two token literals to the Container’s OS as environment variables.

Mounting a file from a Kubernetes Secret to the Container

Here’s the deployment.yaml for our sample application. (I’m using a random image from gcr.io, you can use the same image to test this out)

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sample-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
      app: sample-app
    spec
      containers:
      - name: sample-app
        image: gcr.io/google_containers/defaultbackend:1.0
        ports:
        - containerPort: 8080

Let’s edit this file to mount our secret file (key.json) to the /root/key.json of our container when it’s created.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sample-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
      app: sample-app
    spec
      containers:
      - name: sample-app
        image: gcr.io/google_containers/defaultbackend:1.0
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: service-key
          mountPath: /root/key.json
          subPath: key.json
      volumes:
      - name: service-key
        secret:
          secretName: my-secret
          items:
          - key: service-account-key
            path: key.json

Now that’s done. When we run kubectl apply -f deployment.yaml , our container will be automatically mounted with the key.json in it’s /root/.

Adding Environment Variables from the Secret

Adding environment variables from a secret is a pretty straight forward process. This is how our file will be once we add them as well.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sample-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
      app: sample-app
    spec
      containers:
      - name: sample-app
        image: gcr.io/google_containers/defaultbackend:1.0
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: service-key
          mountPath: /root/key.json
          subPath: key.json
        env:
        - name: "AUTH_TOKEN"
          valueFrom:
            secretKeyRef:
              name: my-secret
              key: webhook_token
        - name: "SLACK_TOKEN"
          valueFrom:
            secretKeyRef:
              name: my-secret
              key: slack_token
      volumes:
      - name: service-key
        secret:
          secretName: my-secret
          items:
          - key: service-account-key
            path: key.json

And that’s it! We have now mounted a file and added two environment variables from a Kubernetes Secret!

Our file should be in /root/key.json and we should have the two environment variables AUTH_TOKEN and SLACK_TOKEN in our OS.

Check first — then celebrate!

It goes without saying;
Right after you deploy your application with secrets it might be good idea to either check if the secrets are added correctly through some mechanism in your application code or by manually ssh’ing into the container using kubectl exec -it <pod_name> /bin/sh and taking a quick peek around to see if everything is set up as expected. (just a one-time check)

Updating Secrets

Whenever you update a secret, Kubernetes will automatically update the values across all the resources that use it — this is one of the other great things about using Kubernetes secrets.

• You can use kubectl apply -f updated_secret.yaml like for any other Kubernetes object or
• You can delete the existing secret kubectl delete secret my-secret and create a new one secret with the new values (but the same metadata) — although if you’re production applications are consuming these secrets constantly it’s better to follow the previous method to avoid breaking anything. (Extract the secret yaml as shown in this guide, update it and then apply it through kubectl)

Follow the Platformer Blog for more articles on Kubernetes and Container Orchestration!