Adding persistent storage to your Kubernetes cluster

When the systems you develop require persistent storage you need to consider where your precious data is to be stored. If you get it wrong, you may be risking total loss of your data. This article deploys a persistent volume to your Kubernetes cluster.

13 min readJan 4, 2024

Whether your data is in the form of a database, configuration items or files, you need to store it somewhere that is not going to forget. This is known as persistent storage.

When you are on your development machine, your persistent storage is likely to be your disk drive. When you move to containers and on to the cloud and a Kubernetes cluster, it is a little more difficult.

This article follows my article on creating a Kubernetes cluster using Infrastructure as Code (IaC) tools such as Terraform and Ansible. It is going to extend that cluster by introducing persistent storage.

This article only introduces the basic concepts of persistent storage. If you require more detail, you can find it in the official documentation here.

Ephemeral Pods

It may be easy just to use the file system provided by your container. It is easy to develop, test and deploy. Everything is contained in the container!

Ok, so we deploy our database in a container to our Kubernetes cluster. Everything works as expected.

But along comes Kubernetes. It is in control of our pods and can start and stop them for a number of reasons. It can even start them up on a different node. We need to assume that our pod can have a short life span. This is why we refer to pods as being ephemeral. They may be stopped at any time.

So our pod containing our database is stopped by Kubernetes (or dies due to a fault). Where does our data go? It is lost with the pod. The container’s filesystem is also lost with the pod, it is ephemeral.

Using our container’s file system may no longer be the solution we want. So what it?

Persistent Volumes

Before we look at where our data is going to be held, let’s look at how Kubernetes manages persistent storage.

We start with a Persistent Volume or PV for short.

When you create a PV within your cluster, you are telling Kubernetes that you want your pods to have access to persistent storage that will outlive the pod (and possibly the cluster itself!).

You can view the PV as representing your persistent storage across the cluster. It will be given a total size, say, 10GB.

Note that a PV is not created within a namespace within your cluster and is therefore available to all pods within a cluster.

It is important to also understand that a PV may define a Storage Class. This helps ensure that the pod is getting what it needs from the PV as we shall see later.

PVs can be created manually (eg: through kubectl) or can be dynamically created by Kubernetes. In the latter case, the Storage Class is used to tell Kubernetes what type of PV the pod requires.

Persistent Volume Claims

Now you want your pods to access the PV you have just created. To do this, you need a Persistent Volume Claim or PVC, that is going to claim part of your PV for this particular pod.

This claim can claim part or all of the allocated space from the PV. If it claims part, the remainder can be allocated for other PVCs for other pods.

Note that a PVC is created within a namespace and so only pods in that namespace can mount it. However, it can be bound to any PV as these are not namespaced.

Connecting the PVC to the PV is done by Kubernetes that matches them and then binds them together. Part of that matching is based on the storage class requested by the PVC and offered by the PV. A PVC is also matched on the availability of the storage size required by the PVC and available on the PV.

It is possible to tell Kubernetes to bind a PVC to a particular PV. When you do this, Kubernetes still checks to make sure the PV is still a match for the PVC.

It is possible that Kubernetes cannot bind the PVC to a valid PV and that the PVC remains unbound until a PV becomes available.

Note that a PVC is bound 1:1 to a PV and once bound, the PV is no longer available to other PVCs. However, any number of pods can mount the PVC.

Mounting the PVC

The final part of the picture is to access the PVC in the pod by mounting the storage as a volume within the container.

Without knowing it, once your PVC is mounted by the pod, your application within the Pod’s container (or containers) now have access to the persistent storage.

Now, when the pod dies and is rescheduled, it will be reconnected to the same PV and will have access to the data it was using before it died, even if this is on another node.

Multiple Pods

You may be wondering what happens when you have multiple instances of a pod on multiple nodes.

When you create your PVC, you can specify how multiple nodes and pods are to access the PVC. Options are:

ReadWriteMany — allows any number of nodes to read and write to the PVC
ReadWriteOnce — allows any single node to read and write to the PVC but if there are multiple pods on that node, they all can
ReadOnlyMany — allows any number of nodes to read the PVC but no one can write to it
ReadWriteOncePod — allows only a single pod in the cluster to read and write to it [limited availability based on versions and technology]

Kubernetes then uses this access request in the PVC to match against the capabilities of the PV.

This access mode places no restriction on the actual access, which is managed by the underlying storage technology below the PV.

The actual storage

So far we have talked about how Kubernetes manages persistent storage within the cluster. We now need to look at how that persistent storage is provided.

The PV is configured to use a particular storage technology. Examples include:

Azure File
CephFS
CSI
iSCSI
NFS
and more

In this article we will use Network File System (NFS) which is a way of sharing a centralised filesystem across multiple nodes.

Whatever technology is used, it must be available to all nodes within the cluster. It does not have to be part of the cluster.

In this example, we will create an NFS share on a separate server within the Virtual Private Cloud (VPC) that holds our Kubernetes cluster. We will also change the Ansible configuration code to ensure that the NFS share is available to all nodes.

Creating a High Availability (HA) NFS service is beyond the scope of this article.

If you delete your NFS server, you will lose all your persisted data.

Deployment Architecture

We will now create our infrastructure by modifying the IaC code in this article. You can find the code for this article in a branch from the GitHub repository in that article.

If you have followed the previous article, you will now have deployed this architecture with the exception of the nfs-server and the NFS clients that you need on the k8s-* nodes.

If you switch to the add_nfs branch of the GitHub repo, you will find that the Terraform and Ansible scripts have been updated.

Create NFS server

First create the nfs-server with:

cd terraform
terraform init
terraform plan
terraform apply

This should create your nfs-server as an Ubuntu server within your VPC.

To use your new server, you need to bootstrap it with:

cd ../ansible/bootstrap
ansible-playbook bootstrap.yml --limit nfs_server

Now create your NFS share with:

cd ../nfs
ansible-playbook nfs.yml

Note that this creates a simple read/write share at /pv-share in this server and makes it available to all hosts. You may look at improving the security of this share after you ensure it is working for you.

You should now have a share that is accessible to all the nodes in your Kubernetes cluster.

Create NFS clients

Now you need to update the existing k8s-* servers and set up your nfs-server. I have added the nfs client configuration to the k8s Ansible playbook. To update your Kubernetes servers to use the new mount, use:

ansible-playbook k8s.yml

Testing your new share

To test your share is working, log in to your k8s-node-1 server.

sudo -i
mkdir tmp
mount <nfs-server address>:/ov-share tmp
mkdir tmp/node-1
touch tmp/node-1/hello-world

Now log in to your nfs-server and look at the share point.

sudo ls /pv-share/node-1

If all is well, you should see a hello-world file in the folder.

You can see from this that access to the shared data is open to all servers. This article is written just to get the PV working as a demonstration and is not a production ready environment.

Now we have persistent storage available to all nodes. Of course, if we destroy the nfs-server or delete the share, we will lose all our data.

Creating a PV

We will now put our persistent storage to use.

Earlier I described the process of creating a Persistent Volume (PV) and then creating a Persistent Volume Claim (PVC) on the PV.

So let’s create a PV using our NFS.

We will switch to the command line from this point rather than using automation.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  capacity:
    storage: 10Gi
  storageClassName: manual
  accessModes:
  - ReadWriteOnce
  nfs:
    path: /pv-share
    server: 10.240.0.165
  persistentVolumeReclaimPolicy: Retain

This is going to create our PV based on the nfs mount. It will make 10GB available to any bound PVC. The Retain option says what the PV should do with the data from the PVC when the PVC is removed. There are three options, each with their own pros and cons:

Retain: retain the PV and the external storage but do not make it available for another PVC to claim
Delete: delete the PV and the external storage which means you will lose your data
Recycle: effectively does the same as Delete (this option is deprecated in favour of Delete)

I recommend using Retain to make sure you do not lose data. However, be aware that if you delete your PVC (eg: by uninstalling an application) when you reinstall your application, it will not be able to claim the PV. You will need to manually delete your PV and recreate it first.

Note that Kubernetes does not apply any volume size contraints.

Apply the file to create the PV:

kubectl create -f pv.yml

You should get a message: persistentvolume/pv0001 created

You can check the status at any time with:

kubectl get pv

Note that we have created this PV in the default namespace.

Creating a PVC

Now we have a PV, we can now make a claim on it by creating a PVC. Kubernetes will match the PVC against available PV and then bind them.

Create a file, pvc.yml, and enter the following:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Now we create the PVC:

kubectl create -f pvc.yml

It should respond with: persistentvolumeclaim/my-pv-claim created

We can now check if the PVC was bound to the PV:

kubectl get pvc

This should produce something like:

NAME          STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-pv-claim   Bound    pv0001   10Gi       RWO            manual         6m30s

You can see that the PVC is now bound to the PV.

Testing

To test that our PVC is in fact available for us to use, we will use a BusyBox image and mount the PVC into the BusyBox container.

Busybox is small executable that contains a small number of tools. It is useful for testing such as this.

Tip:

You can create an interactive BusyBox pod that will be deleted when you exit. This allows you to see how a pod is operating (or not).

kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh

As we want to mount a volume using our PVC, we will create a deployment file. Call it bb.yml and add the following:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "tail -f /dev/null" ]
      volumeMounts:
      - name: volume1
        mountPath: "/mnt/volume1"
  volumes:
  - name: volume1
    persistentVolumeClaim:
      claimName: my-pv-claim
  restartPolicy: Never

You will see that this is using our PVC, my-pv-claim. It creates a volume for this called volume1, which it then mounts at /mnt/volume1.

Start the container and then enter the container with:

kubectl apply -f bb.yml 
kubectl get pods/busybox
kubectl exec -it busybox sh

Don’t run the third line until the second reports the busybox pod is Running. Once you get to a command line, try the following:

ls /mnt/volume1
echo "hello world" > /mnt/volume1/hello.txt
cat /mnt/volume1/hello.txt

You should see a hello world response.

You can now see how the retain setting works. Delete the busybox deployment and then recreate it. Go in to the pod and then check your hello.txt file. You will see that it has been retained — just as we required!

kubectl delete -f bb.yml 
kubectl apply -f bb.yml 
kubectl get pods/busybox
kubectl exec -it busybox sh
cat /mnt/volume1/hello.txt

Automatically creating PVs for PVCs

Creating PVs manually is frustrating. Kubernetes supports the concept of Operators which applications that automate a lot of manual tasks for another application. In our case there is operator for automatically creating PVs, backed by an NFS folder, whenever a relevant PVC is created.

Let’s install it now.

Updating our NFS share

First, we will create a share for our dynamic provisioning. On the nfs-server, execute the following as root. Add a new shared folder to export by adding the following line:

/etc/exports

/pv-share/auto *(rw,async,no_subtree_check)

Your exports file may have other or different entries. What is important is the additional auto path.

You also need to create the new folder and restart the NFS service:

mkdir /pv-share/auto
systemctl restart nfs-server
systemctl status nfs-server

Installing the nfs subdir external provisioner as an operator

Now, log back into your k8s-master server (I am assuming here that you this is where you are running kubectl to access your cluster. If not, use whatever access you typically use to deploy to your cluster). We will now deploy the dynamic provisioner.

We will use Helm to manage this process. If you have not installed Helm previously, you can do as follows (or follow the offical instructions here):

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Now add the required Helm repository:

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner

Now install the operator which, in this case is a provisioner, into the store namespace (replace the < > fields with your values)

helm install -n store --create-namespace nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server=<nfs IP address> --set nfs.path=/pv-share/auto

Once installed, the provisioner will listen for the creation of PVCs (in any namespace) and will automatically create the required PV. It will attach that PV to the NFS system, under the auto/ folder.

Testing the provisioner

Let’s now prove that this works as we expect. To do this, create the following file:

test.pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
  namespace: store
spec:
  storageClassName: nfs-client
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

The important bit about this PVC is that the storageClassName is set to nfs-client. This is what the provisioner looks for (by default — you can change it through the Helm chart installation) and tells it to create a PV. It also means that the provisioner can live side-by-side with your manually created PVs.

Now apply the test PVC and look at the result with:

kubectl create -f test-pvc.yml
kubectl get pv

This should show you the PV it created:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                            STORAGECLASS       REASON   AGE
...
pvc-6bc3785b-3fcf-4f18-9b4a-2aed29c9f282   1Gi        RWX            Delete           Bound    monitoring/test-pvc              nfs-client                  36s
...

So, how do you know that this is the one? Check the PVC you created:

kubectl get pvc -n store

You should now see:

NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
...
test-pvc              Bound    pvc-6bc3785b-3fcf-4f18-9b4a-2aed29c9f282   1Gi        RWX            nfs-client         15s
...

There you can see the volume name we saw earlier.

Congratulations, you now have a dynamic PV provisioner that you can use in most deployments that create a PVC.

Summary

In this article we looked at how we provide persistent storage to our Kubernetes cluster using an NFS server. The IaC configuration was updated to implement the NFS server.

Based on the share, we created a Persistent Volume (PV). We then created a persistent Volume Claim (PVC), which we bound to the PV.

Once we created the PVC, we then used a BusyBox image to show how the PVC can be mounted into a pod.

Finally, we installed a provisioner that automatically creates our PVs for us when we create a PVC.

If you found this article of interest, please give me a clap as that helps me identify what people find useful and what future articles I should write. If you have any suggestions, please add them in the comments section.