Restore volume from AWS Snapshot and bind it to Kubernetes PVC

You may have a situation when a database got completely broken after an update, or a volume got overflown and you need to restore a volume from an AWS Snapshot to the previous state and bind it back to the Kubernetes PVC.

In this example, we will use our SonarQubesonar-db database Deployment. We will create a Snapshot from a Volume in AWS, delete all the Volume resources in Kubernetes and AWS, and then, restore the Volume and bind it to a new Kubernetes PVC.

Prepare for creating Volume Snapshot

1. Find the PersistentVolumeClaim name attached to the deployment:

namespace="your_namespace"
deployment="sonar-db"

pvc=$(kubectl get deployment "${deployment}" -o jsonpath='{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' -n "${namespace}")
echo "${pvc}"

By default, the PVC name is sonar-db in the EDP setup.

2. Save the whole PersistentVolumeClaim manifest. You will need it later.

kubectl get pvc "${pvc}" -n "${namespace}" -o yaml > pvc_manifest.yaml

Find the PersistentVolume name:

pv=$(kubectl get pvc "${pvc}" -o jsonpath='{.spec.volumeName}' -n "${namespace}")
echo "${pv}"

3. Save the PersistentVolume manifest. It will also be required.

kubectl get pv "${pv}" -n "${namespace}" -o yaml > pv_manifest.yaml

Locate the AWS volume name:

If this command does not work for you, locate the AWS volume name right in the PersistentVolume manifest.

aws_volume=$(kubectl get pv "${pv}" -o jsonpath='{.spec.csi.volumeHandle}')
echo "${aws_volume}"

Create Volume Snapshot on AWS

1. Log in to your AWS account and search for the AWS Volume you got from the command above “${aws_volume}”.
2. Check the Volume Availability Zone (for example, ‘eu-central-1b’), Volume type (for example, gp2), Volume Size, Tags.
3. Create a Snapshot from a Volume according to this guide: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html
4. Check the Snapshot status on AWS. It should be “Completed”.

Delete Volume resources

1. Scale the related Kubernetes Deployment resources to 0. In our case, they are sonar, sonar-db, and sonar-operator.

kubectl scale deployment sonar --replicas="0" -n "${namespace}"
kubectl scale deployment sonar-db --replicas="0" -n "${namespace}"
kubectl scale deployment sonar-operator --replicas="0" -n "${namespace}"

2. Delete PersistentVolumeClaim sonar-db

kubectl delete pvc "${pvc}" -n edp-delivery-yp-delivery-dev

3. Double-check if the PersistentVolume is deleted as well.

Depending on the StorageClass, the PV resource with the Volume in AWS can be deleted automatically after deleting PVC.

4. Re-check if the Volume has been deleted in AWS too.

Restore Volume

1. Go to your AWS account and restore a Volume from the created Snapshot: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-volume.html
2. Fill in the same Availability Zone, Volume type, Volume Size, Tags as it was in previous Volume.
3. Check the Volume status on AWS. It should be “Okay”.
4. Modify the “pv_manifest.yaml” PersistentVolume manifest that was saved earlier. Delete the unnecessary Kubernetes metadata, remove claimRef, nodeAffinity.
5. Change the PersistentVolume name. We will use “pv-restored”

6. Generate new “uid” and replace it in the manifest.

You can use online UUI v4 generators, like: https://www.uuidgenerator.net/version4

The manifest will look similar to this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-restored
  uid: 7f95837c-3dc0-4f27-8b32-6a6a0e2afac8
spec:
  capacity:
    storage: 1Gi
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-0f08d629e6900b96d
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1683083106347-8081-ebs.csi.aws.com
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: ebs-sc
  volumeMode: Filesystem

7. Apply the PV manifest to the cluster:

kubectl apply -f pv_manifest.yaml

8. Modify the “pvc_manifest.yaml” PersistentVolumeClaim manifest that was saved earlier. Delete the unnecessary Kubernetes metadata, change the ‘volumeName’, in our case it is ‘pv-restored’.

Here is the example (replace the namespace):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sonar-db
  namespace: <namespace>
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: pv-restored
  storageClassName: ebs-sc
  volumeMode: Filesystem

9. Apply the PVC manifest to the cluster:

kubectl apply -f pvc_manifest.yaml

10. Scale up the Deployments and check if the application volume restore was successful:

kubectl scale deployment sonar --replicas="1" -n "${namespace}"
kubectl scale deployment sonar-db --replicas="1" -n "${namespace}"
kubectl scale deployment sonar-operator --replicas="1" -n "${namespace}"

11. Check the PVC status. It should be ‘Bound’:

kubectl describe pvc "${pvc}" -n "${namespace}"