Restore volume from AWS Snapshot and bind it to Kubernetes PVC
You may have a situation when a database got completely broken after an update, or a volume got overflown and you need to restore a volume from an AWS Snapshot to the previous state and bind it back to the Kubernetes PVC.
In this example, we will use our SonarQubesonar-db
database Deployment. We will create a Snapshot from a Volume in AWS, delete all the Volume resources in Kubernetes and AWS, and then, restore the Volume and bind it to a new Kubernetes PVC.
Prepare for creating Volume Snapshot
- Find the PersistentVolumeClaim name attached to the deployment:
namespace="your_namespace"
deployment="sonar-db"
pvc=$(kubectl get deployment "${deployment}" -o jsonpath='{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' -n "${namespace}")
echo "${pvc}"
By default, the PVC name is sonar-db
in the EDP setup.
2. Save the whole PersistentVolumeClaim manifest. You will need it later.
kubectl get pvc "${pvc}" -n "${namespace}" -o yaml > pvc_manifest.yaml
Find the PersistentVolume name:
pv=$(kubectl get pvc "${pvc}" -o jsonpath='{.spec.volumeName}' -n "${namespace}")
echo "${pv}"
3. Save the PersistentVolume manifest. It will also be required.
kubectl get pv "${pv}" -n "${namespace}" -o yaml > pv_manifest.yaml
Locate the AWS volume name:
If this command does not work for you, locate the AWS volume name right in the PersistentVolume manifest.
aws_volume=$(kubectl get pv "${pv}" -o jsonpath='{.spec.csi.volumeHandle}')
echo "${aws_volume}"
Create Volume Snapshot on AWS
- Log in to your AWS account and search for the AWS Volume you got from the command above “${aws_volume}”.
- Check the Volume Availability Zone (for example, ‘eu-central-1b’), Volume type (for example, gp2), Volume Size, Tags.
- Create a Snapshot from a Volume according to this guide: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html
- Check the Snapshot status on AWS. It should be “Completed”.
Delete Volume resources
- Scale the related Kubernetes Deployment resources to 0. In our case, they are sonar, sonar-db, and sonar-operator.
kubectl scale deployment sonar --replicas="0" -n "${namespace}"
kubectl scale deployment sonar-db --replicas="0" -n "${namespace}"
kubectl scale deployment sonar-operator --replicas="0" -n "${namespace}"
2. Delete PersistentVolumeClaim sonar-db
kubectl delete pvc "${pvc}" -n edp-delivery-yp-delivery-dev
3. Double-check if the PersistentVolume is deleted as well.
Depending on the StorageClass, the PV resource with the Volume in AWS can be deleted automatically after deleting PVC.
4. Re-check if the Volume has been deleted in AWS too.
Restore Volume
- Go to your AWS account and restore a Volume from the created Snapshot: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-volume.html
- Fill in the same Availability Zone, Volume type, Volume Size, Tags as it was in previous Volume.
- Check the Volume status on AWS. It should be “Okay”.
- Modify the “pv_manifest.yaml” PersistentVolume manifest that was saved earlier. Delete the unnecessary Kubernetes metadata, remove claimRef, nodeAffinity.
- Change the PersistentVolume name. We will use “pv-restored”
6. Generate new “uid” and replace it in the manifest.
You can use online UUI v4 generators, like: https://www.uuidgenerator.net/version4
The manifest will look similar to this:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-restored
uid: 7f95837c-3dc0-4f27-8b32-6a6a0e2afac8
spec:
capacity:
storage: 1Gi
csi:
driver: ebs.csi.aws.com
volumeHandle: vol-0f08d629e6900b96d
fsType: ext4
volumeAttributes:
storage.kubernetes.io/csiProvisionerIdentity: 1683083106347-8081-ebs.csi.aws.com
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: ebs-sc
volumeMode: Filesystem
7. Apply the PV manifest to the cluster:
kubectl apply -f pv_manifest.yaml
8. Modify the “pvc_manifest.yaml” PersistentVolumeClaim manifest that was saved earlier. Delete the unnecessary Kubernetes metadata, change the ‘volumeName’, in our case it is ‘pv-restored’.
Here is the example (replace the namespace):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sonar-db
namespace: <namespace>
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
volumeName: pv-restored
storageClassName: ebs-sc
volumeMode: Filesystem
9. Apply the PVC manifest to the cluster:
kubectl apply -f pvc_manifest.yaml
10. Scale up the Deployments and check if the application volume restore was successful:
kubectl scale deployment sonar --replicas="1" -n "${namespace}"
kubectl scale deployment sonar-db --replicas="1" -n "${namespace}"
kubectl scale deployment sonar-operator --replicas="1" -n "${namespace}"
11. Check the PVC status. It should be ‘Bound’:
kubectl describe pvc "${pvc}" -n "${namespace}"