Kubernetes Secrets

How to Consume Environment Variable Secrets within a Pod

Brandi McCall

8 min readJun 5, 2023

Prerequisites:

Basic Kubernetes knowledge
kubectl installed
A running k3d cluster with at least 2 worker nodes

Objectives:

Discuss Kubernetes Secrets
Create literal secrets in source files
Create a Secret object using the kubectl utility
Observe how Kubernetes encodes Secrets
Deploy a PersistentVolume and PersistentVolumeClaim to the cluster
Create a mySQL Deployment to consume the Secrets
Confirm Secrets were consumed by the Pod

Kubernetes Secrets

Kubernetes allows you to incorporate secrets like passwords, tokens, and keys into clusters via a Secret object. It’s important to note that Secrets are not encrypted by default but they are Base64 encoded. To make Secrets more secure, they should be encrypted before being applied to a Kubernetes cluster. There is much more to securing Secrets than what can be discussed here, but the purpose of this tutorial is to show how a Secret can by consumed by a Pod.

Secrets can be created and stored in a manifest separate from the Pods that use them. This allows you to keep your confidential data and your application code separate. Secret objects are declared and consumed similarly to ConfigMaps, but the latter should never be used for confidential data.

Pods can consume Secrets via the following methods:

As a file or files within in a volume mounted to a container
As environment variables within the container
By the kubelet when it pulls images required for the Pod

In this tutorial, we will create a Secret using the kubectl utility, then consume that Secret by a Pod. Confirm you have a k3d Kubernetes cluster with at least 2 worker nodes running, then move on to the next section.

Store Secrets in Files

When using kubectl to manage your Secrets, you can either pass raw data into a command, or store the secrets in files, and pass those files in the command. Placing your Secrets in plain text files is not the most secure format. We will create our Secrets in source files, then pass those file paths in the kubectl command. The Secrets we create will consist of a sample username and password.

We will now create a username Secret and password Secret. The username we want to store is “admin” and the password is “1f2d1e2e67df”. Use the following commands to write the secrets into new txt files:

echo -n 'admin' > ./username.txt
echo -n '1f2d1e2e67df' > ./password.txt

The above commands take our “admin” username and “1f2d1e2e67df” password, and sends the encoded strings to new files within the current directory. You can view the new files with the ls command (disregard my other files).

If we look at the contents of those files, we will see this for the username.txt file:

And this for the password.txt file:

Create the Secrets with kubectl

We can now create our Secret objects using the following command:

kubectl create secret generic db-user \
    --from-file=username=./username.txt \
    --from-file=password=./password.txt

The above command uses kubectl create secret to create a generic Secret with the name “db-user”. It creates the Secret by reading the contents of the files “username.txt” and “password.txt” from the local directory and storing them as key-value pairs within the Secret. The key for the “username.txt” file is “username” and the key for the “password.txt” file is “password”. The value for each key is the contents of the files.

It looks like the Secret was created, but let’s verify that with this command:

kubectl get secrets

Perfect, the Secret was in fact created. Note the “2” under the “Data” column. This 2 denotes that there are 2 secrets (a username and password). Typically when you create a Kubernetes object, you can get some information using the kubectl get all command. Secrets, however, are intentionally not included in this output. To view more details of the Secret, use the following command:

kubectl describe secret db-user

When creating a Secret in Kubernetes, it gets Base64 encoded in the background. To confirm the Secrets have been Base64 encoded, use the following command:

kubectl get secret -o yaml

You can see the password value is MWYyZDFlMmU2N2RmCg==, which is 1f2d1e2e67df (our literal password) Base64 encoded. If you weren’t sure, you could paste the password into a web browser decoder like base64decode.org and click the “decode” button.

Likewise, YWRtaW4K is our literal username of admin Base64 encoded.

Deploy a PersistentVolume and PersistentVolumeClaim to the Cluster

MySQL databases typically use persistent volumes to store data, but that storage must be requested via a persistent volume claim. Persistent volumes and their claims must be created before a workload (in this case mySQL) attempts to consume it. Create a new file called storage.yaml and paste the following code:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

Apply the PersistentVolume and PersistentVolumeClaim to the cluster with the following command:

kubectl apply -f storage.yaml

We can check that these were deployed with kubectl get pv and kubectl get pvc . Note that the persistent volume has a status of “BOUND” meaning the PVC found a volume that met its requirements, and the PV and PVC are now bound. Now that the PV is bound, it is available for use.

Create a mySQL Deployment to Consume Secrets

We can now create a Pod on the cluster to consume the Secret. Create a new file called deployment.yaml and copy the following code into it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:latest
        env:
          - name: MYSQL_USER
            valueFrom:
              secretKeyRef:
                name: db-user
                key: username
          - name: MYSQL_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: db-user
                key: password
        ports:
        - containerPort: 3306
        volumeMounts:
          - name: mysql-persistent-storage
            mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        emptyDir: {}

Here’s an explanation of what this code is doing:

apiVersion: apps/v1 specifies the API version for the Deployment.
kind: Deployment indicates a Deployment resource.
metadata contains information about the Deployment (ex. name).
spec describes the desired state of the Deployment.
selector specifies the label selector used to identify the Pods managed by this Deployment. The Pods should have the label app: mysql.
template defines the Pod template for creating new Pods.
metadata within template specifies metadata for the Pods.
labels within metadata sets the labels for the Pods. The label app: mysql is used to match the selector defined in the Deployment.
spec within template defines the specification for the Pods.
containers lists the containers running in the Pod. In this case, there is one container named mysql with the mysql:latest image.
env specifies the environment variables for the container.
MYSQL_USER environment variable is set with the value from a Secret named db-user using secretKeyRef. The value comes from the key username in the Secret.
MYSQL_ROOT_PASSWORD environment variable is set with the value from the same Secret (db-user) using secretKeyRef. The value comes from the key password in the Secret.
ports defines the container ports that should be exposed. In this case, port 3306 is exposed, which is the default for mySQL.
volumeMounts specifies the volumes to be mounted inside the container. In this case, a volume named mysql-persistent-storage is mounted at the path /var/lib/mysql.
volumes defines the volumes available for the Pod. In this case, there is one volume named mysql-persistent-storage of type emptyDir, which is an empty volume that is tied to the Pod's lifetime.

Overall, this manifest deploys a MySQL database as a Deployment with a single container. The container has environment variables populated from a Secret, exposes port 3306, and mounts an emptyDir volume for persistent storage. Save the deployment.yaml file and apply it to the cluster using:

kubectl apply -f deployment.yaml

Give the container a few minutes to create and then check the Pods:

Confirm Secrets Were Consumed by the Pod

Remember, this is a tutorial on Kubernetes Secrets and how they get consumed by a Pod. It is not a tutorial on how to use mySQL, because I simply have not used it much. Nonetheless, we can confirm that our username and password Secrets were in-fact consumed by the mySQL Pod. In our deployment.yaml file, we used our Secrets to define the MYSQL_USER and MYSQL_ROOT_PASSWORD environment variables.

To see if the secrets got passed to the Pod, we can simply look at the environment variables within the Pod. Find the Pod name with:

kubectl get pods

Now let’s enter the container and look for its environment variables:

kubectl exec -it mysql-7b7989ffb-q84l7 -- env

We can see that the Secrets did in-fact get consumed by the Pod!

Conclusion and Clean Up

In this tutorial we discussed Kubernetes Secrets and how to create them from files. We referenced the Secrets in our deployment.yaml file, thereby allowing our mySQL Pod to consume them. We also reviewed steps on how to confirm that our Secrets were consumed.

To clean up what we just created, use the following command to delete the cluster and everything in it: