EKS — Attaching an existing volume to a Statefulset.

There may be a time when you need to attach an existing PersistentVolume (PV) to a new StatefulSet.

A common example is migrating from the Kubernetes Amazon EBS related in-tree storage plugin — called a provisioner of type “kubernetes.io/aws-ebs” — only supports Amazon EBS types io1, gp2, sc1, and st1.

If you are using the in-tree storage plugin and wish to use EBS volume types such as gp3 and io2 and take advantage of new features (like Volume Snapshots) you will need to migrate to use the Amazon EBS CSI driver.

The following AWS Blog post — Migrating EKS Clusters from gp2 to gp3 describes the steps to migrate an existing gp2 volume that is using the in-tree storage plugin to a gp3 volume using the Amazon EBS CSI driver.

The blog post provides an example of migrating the PV with a single pod.

However if you are using a Statefulset, the PV’s are dynamically created based on the number of replicas specified.

I’m going to explain how attach an existing volume(s) to pods created by the StatefulSet.

StatefulSet (Dynamic Volume Provisioning): Within a Statefulset, the volumeClaimTemplates field instructs Kubernetes to utilize Dynamic Volume Provisioning to create a new EBS Volume, a PersistentVolume (PV) and a PersistentVolumeClaim (PVC). These are automatically created based on the number of replicas that are specified in the StatefulSet.

To use an existing EBS Volume, the PV will need to be unallocated and in a pending state. To do this follow the migration blog post with steps on how to migrate a gp2 volume by creating a snapshot and creating a new gp3 volume that contains the data.

We have the following PV’s that have been migrated from gp2 to gp3 and are ready to be bound to a pod.

kubectl get pvc
NAME                                                       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0                                                  Pending                                                                        gp3            3m42s
www-web-1                                                  Pending                                                                        gp3            48s
www-web-3                                                  Pending                                                                        gp3            3s

We will now create the Statefulset, ensuring that the names of the Dynamic Volume Provisioning match the names above.

To do this we need to understand how a Statefulset will Dynamically Provisioning Volumes.

The Statefulset will create a new PVC based on the following: volumeClaimTemplates(name)-Statefulset(name)-Replica_Index.

When we create a StatefulSet, we need to ensure we have the following set:

volumeClaimTemplates(name) = www
Statefulset name = web
Replica_Index (This will change based on the number of replicas. It starts with 0).

The StatefulSet example below dynamically creates a new PVC for each replica, so www-web-0 (for the 1st volume created), www-web-1 (2nd volume created), www-web-2 (3rd volume created).

The Statefulset will Dynamically Provision a Volume for each replica Pod created, in this example we have specified three replicas.

$ touch vs-csi.yaml
$ vim vs-csi.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web # Statefulset name
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  replicas: 3 #
  minReadySeconds: 10
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: registry.k8s.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www # volumeClaimTemplates(name)
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "gp3"
      resources:
        requests:
          storage: 1Gi

If you wish to use an existing storage volume the name of the volume will need to match the naming convention that the Statefulset uses when creating a new volume as well as the storageClassName.

In this instance we have three existing volumes that are unbound and ready to be assigned to be Pod with names that match the naming convention that the StatefulSet will use to create the PV and PVC.

The StorageClass of gp3 also matches the storageClassName.

kubectl get pvc
NAME                                                       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0                                                  Pending                                                                        gp3            3m42s
www-web-1                                                  Pending                                                                        gp3            48s
www-web-3                                                  Pending                                                                        gp3            3s

We will now deploy the Statefulset:

$ kubectl apply -f statefulset.yaml

After deploying the StatefulSet, the PVC should now be bound to a pod.

We can check this by running the kubectl get pvc command.

$ kubectl get pvc 
NAME                                                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0                                                   Bound    pvc-5afa0d2e-856c-4e2a-8fbe-ee7e4b08fa9f   1Gi        RWO            gp2            143m
www-web-1                                                   Bound    pvc-ec59c796-b21c-4d4e-88ca-8cabd246b3c6   1Gi        RWO            gp2            142m
www-web-2                                                   Bound    pvc-4a93380d-f245-4c90-9b09-e22eb26c96fd   1Gi        RWO            gp2            142m

The three existing volumes are now bound to the pods that were deployed by the Statefulset.

To confirm this we can check the pod.

kubectl describe pod web-0

Volumes:
  www:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  www-web-0

The pod references the PVC claim by www-web-0.

You can also check for data on the migrated volume.

Replace the example below with the location of the volume that holds the data:

kubectl exec web-0 — sh -c “cat /usr/share/nginx/html/text.txt” | more