How To Use Kubernetes Secrets for Storing Sensitive Config Data

Because your keys and secrets should be just that—secret

Abhishek Gupta
Dec 19, 2019 · 8 min read
Image for post
Image for post
Photo by JOSHUA COLEMAN on Unsplash

Hello and welcome 👋👋 We continue the Kubernetes in a Nutshell journey. In one of the previous blogs, we saw how to configure Kubernetes apps using the ConfigMapobject. In this post, we will explore Kubernetes Secretsand how they can be used to store sensitive configuration data that needs to be handled securely (e.g., database credentials, API keys, etc.).

Image for post
Image for post

As usual, this is going to be example-driven, and you will learn:

  • how to create Secrets (CLI, yaml etc.) and
  • various ways of using them in your apps (env variables, volumes, etc.)

The code (and YAML) is available on GitHub.

This piece has been divided into two logical sections: ways to create Secrets and techniques to use Secrets in your applications.

Prerequisites

To go through the examples in this post, all you need is a Minikube cluster and kubectl CLI tool to access the cluster.

Install Minikube as a single-node Kubernetes cluster in a virtual machine on your computer. On a Mac, you can simply:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 \
&& chmod +x minikube
sudo mv minikube /usr/local/bin

Install kubectl to interact with the Minikube cluster. On a Mac, you can:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectlchmod +x ./kubectlsudo mv ./kubectl /usr/local/bin/kubectl

Creating Secrets

Let’s look at techniques using which you can create a Secret.

Use the data section

It’s possible to create a Secret along with the configuration data stored as key-value pairs in the datasection of the definition.

apiVersion: v1
kind: Secret
metadata:
name: service-apikey
data:
apikey: Zm9vYmFy

The Secret contains key-value data representing the sensitive info, with apikey being the key and value is a base64 encoded string.

To create this Secret in Kubernetes:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-data.yaml

To keep things simple, the YAML file is being referenced directly from the GitHub repo, but you can also download the file to your local machine and use it in the same way.

To confirm that the Secret has been created:

kubectl get secret/service-apikey -o yaml

You will get a YAML response similar to:

apiVersion: v1
data:
apikey: Zm9vYmFy
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"apikey":"Zm9vYmFy"},"kind":"Secret","metadata":{"annotations":{},"name":"service-apikey","namespace":"default"}}
creationTimestamp: "2019-12-17T11:11:27Z"
name: service-apikey
namespace: default
resourceVersion: "113009"
selfLink: /api/v1/namespaces/default/secrets/service-apikey
uid: 671b547c-3316-4916-b6dc-be2b551b974e
type: Opaque

Fetching the Secret details using kubectl get does not disclose its contents.

Notice that apikey: Zm9vYmFy was what we had provided in the YAML manifest. You can check the plain text form by decoding it:

echo 'Zm9vYmFy' | base64 --decode//foobar

Use the stringData section

The data attribute used in the above example is used to save base64 encoded information. If you want to store plain text data securely, you can use the stringDatasection. Here is an example:

apiVersion: v1
kind: Secret
metadata:
name: plaintext-secret
stringData:
foo: bar
mac: cheese

The values for foo and mac are being passed as plain text. Create this Secret and confirm:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-plaintext.yamlkubectl get secret/plaintext-secret -o yaml

Here is the data portion of the YAML response. The actual data is stored in base64 encoded format.

data:
foo: YmFy

If you decode the data, you can confirm that it matches the original plain text input (bar) we had provided

echo 'YmFy' | base64 --decode//bar

Note that thedata section does not accept plain text attribute. Trying to do so will result in an error similar to this: illegal base64 data at input byte 8

File contents

You can provide the contents of an entire file as input to the stringData section as well. Here is what this might look like:

apiVersion: v1
kind: Secret
metadata:
name: secret-in-a-file
stringData:
app-config.yaml: |-
hello: world
john: doe

Create this Secret

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-file.yaml

This resulting Secret will contain a key named app-config.yaml and its contents (value) will be the base64 encoding of the provided data.

As usual, you confirm this in Kubernetes as well as decode the contents.

kubectl get secret/secret-in-a-file -o yaml
echo '<"data" content in yaml response> | base64 --decode

Note: When you use this technique, your application is responsible for parsing out the data that represents the Secret configuration. In this case, it happens to be newline-separated key-value pairs, but it could be anything else.

Using kubectl

You can use the kubectl create secret command to create Secret objects

Using --from-literal

You can use plain text data to create Secret using the CLI (this will be stored in base64 encoded format in Kubernetes):

kubectl create secret generic redis-credentials --from-literal=user=poweruser --from-literal=password='f0ob@r'

Using --from-file

kubectl create secret generic topsecret --from-file=api_keys.txt

This will create a secret (topsecret) with

  • a key with the same name of the file i.e. api_keys.txt in this case
  • and, value as the contents of the file

From files in a directory

You can simply point to a directory and all the files within will be used to create the Secret.

kubectl create secret generic topsecrets --from-file=/home/credentials/

You will end up with

  • multiple keys which will the same as the individual file name
  • the value will be the contents of the respective file

Using Secrets

For Secrets to be useful, we need to ensure that they are available to our applications (i.e., Pods). Let's explore the ways in which we can do this

Environment variables

You can consume the Secret data as environment variables in a Pod (just like ConfigMap). Here is an example:

apiVersion: v1
kind: Pod
metadata:
name: pod1
spec:
containers:
- name: nginx
image: nginx
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: service-apikey
key: apikey

We use the key apikey from Secret service-apikey and make sure its value is available as an environment variable API_KEY inside the Pod.

Create the Pod (assuming you have the Secret created from the example before) and confirm.

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-env.yamlkubectl get pods -w

Wait for the Pod to transition to Running state. Then, confirm that the environment variable has been injected into the Pod.

kubectl exec pod1 -- env | grep API_KEY

You should get this response — API_KEY=foobar.

Instead of referring to individual entries in a Secret, you can use envFrom to conveniently use all entries as environment variables in a Pod. This is how you might use this:

apiVersion: v1
kind: Pod
metadata:
name: pod2
spec:
containers:
- name: nginx
image: nginx
envFrom:
- secretRef:
name: plaintext-secret

We are referring to the plaintext-secret Secretusing envFrom.secretRef. To create this Pod:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-envFrom.yamlkubectl get pods -w

Wait for the Pod to transition to Running state and then confirm the presence of environment variables.

kubectl exec pod2 -- env | grep foo
//foo=bar
kubectl exec pod2 -- env | grep mac
//mac=cheese

This confirms that both foo and mac were added as environment variables into the Pod along with their decoded values i.e. bar and cheese respectively

Volumes

You can mount Secrets as Volume within a Pod. For example:

apiVersion: v1
kind: Pod
metadata:
name: pod3
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- name: apikey-config-volume
mountPath: /secret
readOnly: true
volumes:
- name: apikey-config-volume
secret:
secretName: service-apikey

The apikey-config-volume volume refers to the service-apikey Secret. To create this Pod:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-volume.yamlkubectl get pods -w

Wait for the Pod to transition to Running state. Then execute the following command:

kubectl exec pod3 -- cat /secret/apikey
//foobar

This confirms that the key apikey in secret service-apikey was mounted as a file (with name apikey) in the /secret directory (as specific in Pod). The content of the file is nothing but the secret value i.e. foobar in this case.

Using imagePullSecrets

There is a way to use Secrets such that your application Pod can use it to authenticate and pull Docker images from private Docker registries.

There are actually three types of Secrets:

  • generic - used to store key-value pairs, as we have seen in the examples so far
  • tls - store public.private key pair info as Secrets
  • docker-registry - credentials for authenticating to a Docker registry.

The way this technique is used is very simple:

  • Use docker-registry Secret type to store private Docker registry credentials in Kubernetes.
  • And then, imagePullSecrets (in a Pod) to reference the Secret containing the Docker registry credentials.

An example always helps:

apiVersion: v1
kind: Pod
metadata:
name: pod4
spec:
containers:
- name: privateapp
image: abhirockzz/test-private-repo:latest
command: ["/bin/sh"]
args: ["-c", "while true; do date; sleep 5;done"]
imagePullSecrets:
- name: docker-repo-secret

See how imagePullSecrets.name refers to a Secret called docker-repo-secret. Let's create it.

But before that, please ensure that you have a private Docker registry. I used DockerHub, but you can choose any other.

Start by creating a Secret (with the name docker-repo-secret) which contain your Docker credentials using kubectl create secret docker-registrycommand.

kubectl create secret docker-registry docker-repo-secret --docker-server=DOCKER_REG_SERVER --docker-username=DOCKER_REG_USERNAME --docker-password=DOCKER_REG_PASSWORD --docker-email=DOCKER_REG_EMAIL

For Docker Hub:

kubectl create secret docker-registry docker-repo-secret --docker-server=https://index.docker.io/v1/ --docker-username=foobarbaz --docker-password=t0ps3cr3t --docker-email=foobarbaz@gmail.comkubectl get secret/docker-repo-secret -o yaml

https://index.docker.io/v1/ is the Docker Hub registry server

To test things out, we will use a busybox image and tagit

docker pull busybox
docker tag busybox [DOCKER_REG]/[DOCKER_PRIVATE_REPO]:[IMAGE_TAG]
e.g.
docker tag busybox abhirockzz/test-private-repo:latest

… and push it

docker push [DOCKER_REG]/[DOCKER_PRIVATE_REPO]:[IMAGE_TAG]e.g. 
docker push abhirockzz/test-private-repo:latest

Once the private repo is ready, you can create the Podwhich will pull the image from the private repo using the registry credentials supplied to it via the Secret

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-docker.yamlkubectl get pods -w

Wait for the Pod to move to Running status.

If you see an ErrImagePull error, it indicates that there might be problem authenticating to the Docker registry. To get details, use: kubectl describe pod/pod4

To confirm that the pod is working fine: kubectl logs -f pod4

Since the busybox image does not really do anything by itself, we execute: while true; do date; sleep 5;done (as provided in the Pod spec). As a result, you should see the logs (printed every 5 secs).

Tue Dec 17 14:17:34 UTC 2019
Tue Dec 17 14:17:39 UTC 2019
Tue Dec 17 14:17:44 UTC 2019
Tue Dec 17 14:18:49 UTC 2019

All good! What that means is that the Pod was able to pull down your image from a private Docker repo using the Docker credentials which were injected into the Pod using imagePullSecrets, which itself referenced a Secret

Good to Know

Here is a (non-exhaustive) list of things that you should bear in mind when using Secrets:

  • Secret has to be created before any Pod that wants to use it.
  • Secrets are applicable within a namespace i.e. they can only be used by Pods in the same namespace
  • The Pod will not start if there is a reference to a nonexistent key in a Secret (using secretKeyRef)
  • 1MiB is the size limit for individual Secrets

That’s it for this edition of the Kubernetes in a Nutshell series. Stay tuned for more!

If you are interested in learning Kubernetes and Containers using Azure, simply create a free account and get going. A good starting point is to use the quickstarts, tutorials, and code samples in the documentation to familiarize yourself with the service. I also highly recommend checking out the 50-day Kubernetes Learning Path. Advanced users might want to refer to Kubernetes best practices or watch some of the videos for demos, top features, and technical sessions.

I really hope you enjoyed and learned something from this article. 🙌

Better Programming

Advice for programmers.

Sign up for The Best of Better Programming

By Better Programming

A weekly newsletter sent every Friday with the best articles we published that week. Code tutorials, advice, career opportunities, and more! Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Abhishek Gupta

Written by

Currently working with Kafka, Databases, Azure, Kubernetes and related open source projects | Confluent Community Catalyst (for Kafka)

Better Programming

Advice for programmers.

Abhishek Gupta

Written by

Currently working with Kafka, Databases, Azure, Kubernetes and related open source projects | Confluent Community Catalyst (for Kafka)

Better Programming

Advice for programmers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store