Azure Kubernetes Service — Next level persistent storage with Azure Disk CSI driver

Published in

01001101

4 min readOct 13, 2021

When talking about persistent storage with Kubernetes Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) are our tools of choice. We can use them with Azure Disks and Azure Files for a long time now. But now it is time to bring them to the next level!

This post mainly focuses on Azure Disks (don’t miss to review the links below to get more details on Azure Files).

What’s new?

With Azure Disk Container Storage Interface (CSI) driver for Azure Disk, we are now getting some great features that help us to run our stateful services much smoother on Azure Kubernetes Service:

Zone Redundant Storage (ZRS) Azure Disks
ReadWriteMany with Azure Disks
Kubernetes-native volume snapshots
Volume resizing
Volume cloning

Azure Disk Container Storage Interface driver

Since Kubernetes 1.21 (on Azure Kubernetes Service) we can use the new Container Storage Interface implementation which is available for Azure Disk and Azure Files.

Wait, what is Container Storage Interface?

The Container Storage Interface (CSI) is an abstraction layer that allows third-party storage providers to write plugins exposing new storage systems in Kubernetes. CSI is generally available for some time now and replaces the previous volume plugin system. With CSI third-party providers can develop and maintain their plugins outside of the Kubernetes project lifecycle which brings them more flexibility. CSI is the future for any storage integration and is already used in many scenarios.

How to use the Azure Disk CSI driver with Azure Kubernetes Service?

You will need at least Kubernetes 1.21 or newer to be able to use the Azure Disk Container Storage Interface driver. Besides this you will also need to enable some Azure features gates to be able to use all of the above-mentioned features:

az feature register --name SsdZrsManagedDisks --namespace Microsoft.Compute
az feature register --name EnableAzureDiskFileCSIDriver --namespace Microsoft.ContainerServiceaz feature list -o table --query "[?contains(name, 'Microsoft.Compute/SsdZrsManagedDisks')].{Name:name,State:properties.state}"
az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableAzureDiskFileCSIDriver')].{Name:name,State:properties.state}"az provider register --namespace Microsoft.Compute
az provider register -n Microsoft.ContainerService

You will then need to spin up an Azure Kubernetes Cluster with the CSI driver feature enabled:

az group create -g aks-1-22 -l westeuropeaz aks create -n aks-1-22 \
  -g aks-1-22 \
  -l westeurope \
  -c 3 \
  -s Standard_B2ms \
  --node-zones 1 2 3 \
  --aks-custom-headers EnableAzureDiskFileCSIDriver=trueaz aks get-credentials -n aks-1-22 -g aks-1-22

Furthermore, you will need to create a StorageClass and VolumeSnapshotClass to use all mentioned features:

cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azuredisk-csi-premium-zrs
provisioner: disk.csi.azure.com
parameters:
  skuname: Premium_ZRS
  maxShares: "3"
  cachingMode: None
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  name: azuredisk-csi-vsc
driver: disk.csi.azure.com
deletionPolicy: Delete
parameters:
  incremental: "true"
EOF

The StorageClass is configured to use the new Premium_ZRS Azure Disks that allow to mount the Disk to multiple Pods across multiple Availability Zones.

The VolumeSnapshotClass allows us to use the Kubernetes-native volume snapshot feature.

We are now ready to create a PersistentVolumeClaim based on the above StorageClass:

cat <<EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-azuredisk-csi-premium-zrs
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 256Gi
  volumeMode: Block
  storageClassName: azuredisk-csi-premium-zrs
EOF

Kubernetes-native volume snapshots

With the Azure Disk CSI driver, we are able to create volume snapshots. You will find more details on the volume snapshot feature itself here. Use the following code snippet to create a snapshot of the above create PVC:

cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: azuredisk-volume-snapshot
spec:
  volumeSnapshotClassName: azuredisk-csi-vsc
  source:
    persistentVolumeClaimName: pvc-azuredisk-csi-premium-zrs
EOF

You can now verify the snapshot with the following kubectl command:

kubectl describe volumesnapshot azuredisk-volume-snapshot

Volume cloning

Volume snapshots themselves are great but we can also use them to clone an existing PVC. For example to attach it to another workload. This can be done with the following manifest:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-azuredisk-snapshot-restored
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: azuredisk-csi-premium-zrs
  resources:
    requests:
      storage: 265Gi
  dataSource:
    name: azuredisk-volume-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
EOF

Once again you can review it with kubectl:

kubectl get pvc,pv

Volume resizing

The CSI driver also supports resizing an existing PVC. Some important notes on this one:

The PVC needs to be unmounted for resizing
The PV will be resized directly, the PVC after mounting it

You can resize the above-cloned PVC via kubectl by executing the following command:

kubectl patch pvc pvc-azuredisk-snapshot-restored --type merge --patch '{"spec": {"resources": {"requests": {"storage": "512Gi"}}}}'

ReadWriteMany across Availability Zones

As already mentioned above we are now also able to mount our PVC to multiple workloads via ReadWriteMany. With the new ZRS Azure Disks, we are even further able to use them across Availability Zones.

Let’s test these new features with the below Deployment manifest:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          volumeDevices:
            - name: azuredisk
              devicePath: /dev/sdx
      volumes:
        - name: azuredisk
          persistentVolumeClaim:
            claimName: pvc-azuredisk-csi-premium-zrs
EOF

As you see above are spinning up a deployment with three replicas across our spread Kubernetes nodes mounting the same Azure Disk with read-write access. You can review it by executing the following kubectl command:

kubectl get pods,volumeattachments -owide

That said, (as you may have already seen) ZRS only supports Block devices so far.

More details

Don’t miss to review the following links to get some more insights and further documentation: