Bouncing Back: How to Fix Your StatefulSet After PVC Deletion Disaster

In this article, I’ll share our surprisingly simple recovery process after our StatefulSets lost their PVCs (volumes were still available on our cloud provider) due to an accidental deletion.

tl;dr —

1. Stop the operator if you’re using one.
2. Scale down the StatefulSets replicas to 0.
3. Generate YAML files for the PVCs from the newly created ones by the StatefulSets.
4. Remove all annotations from the YAML files and ensure all labels are correct.
5. Change the volumeName to the old volume that was previously configured (you can find the volume names in your cloud provider’s volumes section).
6. Delete the new PVCs created by the StatefulSets.
7. Apply the modified PVC YAML files — kubectl apply -f <recoveredYAMLFileName>.yaml
8. Scale the StatefulSets replicas back to their original number.
9. Restart the operator (if applicable).

Example of a recovered PVC Yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    argocd.argoproj.io/instance: clickhouse-0001
    clickhouse.altinity.com/app: chop
    clickhouse.altinity.com/chi: clickhouse
    clickhouse.altinity.com/cluster: click-0001
    clickhouse.altinity.com/namespace: clickhouse
    clickhouse.altinity.com/reclaimPolicy: Retain
    clickhouse.altinity.com/replica: "1"
    clickhouse.altinity.com/shard: "0"
  name: data-clickhouse-0001-0-1-0
  namespace: clickhouse
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti
  storageClassName: encrypted-ebs-no-delete
  volumeMode: Filesystem
  volumeName: pvc-b3579764-20x8-78hj-7254-d37107f97510

Our story begins with a ClickHouse DBMS deployed on our EKS clusters using the Altinity ClickHouse Operator.

Although our operator and ClickHouse clusters are not the latest versions, they are not extremely outdated either.
For this discussion, you don’t need to understand the intricacies of ClickHouse, only that we deploy it using its Helm charts and CRDs. Once deployed, it creates an object called ClickHouseInstallation, which manages all the standard Kubernetes objects beneath it, such as StatefulSets. Additionally, we use Argo-CD to manage all our deployments, including ClickHouse.

Occasionally, a data team member asks for our help to push a new configuration to ClickHouse. These configurations typically adjust how ClickHouse analyzes the vast amounts of data it processes. Recently, we encountered an issue where updates weren’t applied correctly.

Despite updating ClickHouse with a new configuration file, the operator failed to roll out a restart of the StatefulSets and didn’t propagate the configuration changes to the Pods.

To work around this issue, we found a risky solution: deleting the ClickHouseInstallation so that Argo-CD would recreate it with the desired configuration changes. However, this time something went wrong.
For reasons still under investigation, the PVCs were deleted along with the StatefulSets — something that usually didn’t happen, fortunately volumes were still available on AWS, so all we had to do is recover the old PVCs.

The following process involves reconnecting the PV objects, which were not deleted and represent the volumes in AWS, to their respective PVCs. We will create these PVCs with the correct configurations. I won’t delve into the details of how PVs connect to PVCs or the role of StorageClass here. For a deeper understanding of these topics, you can read this comprehensive article: Kubernetes Storage: PV, PVC, and StorageClass.

Since ClickHouseInstallation was automatically started by Argo-CD, it created new PVCs. We used these new PVCs as templates to recover the old ones.

We saved each PVC by running the following command:

kubectl get pvc -n <namespace> <pvcName> -o yaml > <pvcName>.yaml

The next step was to review and edit each of these YAML files. Here is an example of a PVC before editing it for recovery:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
    volume.kubernetes.io/selected-node: ip-10-104-5-112.eu-west-1.compute.internal
    volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
  creationTimestamp: "2024-02-07T15:39:54Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    argocd.argoproj.io/instance: clickhouse-0001
    clickhouse.altinity.com/app: chop
    clickhouse.altinity.com/chi: clickhouse
    clickhouse.altinity.com/cluster: click-0001
    clickhouse.altinity.com/namespace: clickhouse
    clickhouse.altinity.com/object-version: v19d45c48bcb6f6m3juv107264b8bd922ljjb673
    clickhouse.altinity.com/reclaimPolicy: Retain
    clickhouse.altinity.com/replica: "0"
    clickhouse.altinity.com/shard: "2"
  name: data-clickhouse-0001-2-0-0
  namespace: clickhouse
  resourceVersion: "2030492041"
  uid: 381060de-2851-86r0-869e-b83976f7w8u9
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 512Gi
  storageClassName: encrypted-ebs-no-delete
  volumeMode: Filesystem
  volumeName: pvc-198540vb-9641-32e8-741d-d29841c5b3m9
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 512Gi
  phase: Bound

This is how the YAML looks like after editing it and preparing it for the recovery:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    argocd.argoproj.io/instance: clickhouse-0001
    clickhouse.altinity.com/app: chop
    clickhouse.altinity.com/chi: clickhouse
    clickhouse.altinity.com/cluster: click-0001
    clickhouse.altinity.com/namespace: clickhouse
    clickhouse.altinity.com/reclaimPolicy: Retain
    clickhouse.altinity.com/replica: "0"
    clickhouse.altinity.com/shard: "2"
  name: data-clickhouse-0001-2-0-0
  namespace: clickhouse
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti
  storageClassName: encrypted-ebs-no-delete
  volumeMode: Filesystem
  volumeName: pvc-80530gc-8510-75r9-993c-f7539r7m7n2

We removed all annotations and managed fields such as resourceVersion, uid, object-version, and status. Additionally, we ensured the volumeName matched the actual disk volume name and verified that all labels were correct. Since we use ClickHouse, it was crucial to confirm that the clickhouse.altinity.com/shard and clickhouse.altinity.com/replica labels were accurate.

Next, we were ready to apply the recovered YAML files to our cluster. However, before doing so, we needed to stop the operator and scale down the StatefulSets. This can be done by editing the operator deployment:

kubectl edit deployment -n kube-system click-operator
under spec, search for replicasand change it to 0

or just by running:

kubectl scale --replicas=0 -n kube-system deployment/click-operator

and the same can be done for the statefulsets:

kubectl scale --replicas=0 -n clickhouse statefulset/chi-clickhouse-0001

After successfully scaling down both the operator and the StatefulSets, we deleted the newly created PVCs by running

kubectl delete pvc -n clickhouse <pvc1> <pvc2> <pvc3> ...

Now we were ready to apply the new PVCs YAML files by running:

kubectl apply -f -n clickhouse <pvcFileName>.yaml

After applying these files, we ensured the PVCs were in a Boundstate and that each PVC was correctly bound to the appropriate volume. Once verified, we began scaling back the operator and the StatefulSets.

Important: Depending on the database you are using, you may need to start the StatefulSet in a specific order. Ensure you do this rather than scaling all StatefulSets back up at once.

EDIT:

Once you apply the new PVC yaml file, most probably you’ll see that the it’s on Pending state and isn’t getting bound, to fix that you’ll have to edit the PV object: kubectl edit pv <pvName> and delete the claimRefsection, after saving the PV you’ll notice almost instantly that the PVC is in Bound state