Velero with CSI

Published in

Techlogs

3 min readSep 7, 2020

Pre-requistie Kubernetes Cluster > v1.7

2. Install the K8s SnapShot CRD’s

https://github.com/kubernetes-csi/external-snapshotter/issues/245https://github.com/kubernetes-csi/external-snapshotter/blob/master/README.md#usage--
root@k8s-storage-1:~# git clone https://github.com/kubernetes-csi/external-snapshotter.git
root@k8s-storage-1:~# cd external-snapshotter/
--
root@k8s-storage-1:~/external-snapshotter# kubectl create -f client/config/crd
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
--root@k8s-storage-1:~/external-snapshotter# kubectl create -f deploy/kubernetes/csi-snapshotter
serviceaccount/csi-snapshotter created
clusterrole.rbac.authorization.k8s.io/external-snapshotter-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-role created
role.rbac.authorization.k8s.io/external-snapshotter-leaderelection created
rolebinding.rbac.authorization.k8s.io/external-snapshotter-leaderelection created
serviceaccount/csi-provisioner created
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role created
role.rbac.authorization.k8s.io/external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg created
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-provisioner-role created
rolebinding.rbac.authorization.k8s.io/csi-snapshotter-provisioner-role-cfg created
service/csi-snapshotter created
statefulset.apps/csi-snapshotter created

3. If you want to use CSI Snapshotting feature, you need to install the VolumeSnapShotClass and VolumeSnapshot also (from Rook)

kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/csi/rbd/snapshotclass.yamlvolumesnapshotclass.snapshot.storage.k8s.io/csi-rbdplugin-snapclass createdkubectl apply -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/csi/rbd/snapshot.yamlkubectl get VolumeSnapshotClass --all-namespaces
NAME                      DRIVER                       DELETIONPOLICY   AGE
csi-rbdplugin-snapclass   rook-ceph.rbd.csi.ceph.com   Delete           67s

4. Installation Velero with CSI Enabled

velero install \
 --provider aws \
 --plugins velero/velero-plugin-for-aws:v1.0.0,velero/velero-plugin-for-csi:v0.1.1  \
 --bucket velero2  \
 --secret-file ./credentials-velero  \
 --use-volume-snapshots=true \
 --backup-location-config region=default,s3ForcePathStyle="true",s3Url=http://minio.10.131.232.223.nip.io  \
 --image velero/velero:v1.4.0  \
 --snapshot-location-config region="default" \
 --features=EnableCSI

Rest is the same as https://medium.com/techlogs/using-velero-and-restic-to-backup-kubernetes-2f0f812da4db

root@k8s-storage-1:~# k get volumesnapshot --all-namespaces
NAMESPACE     NAME                    READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS             SNAPSHOTCONTENT   CREATIONTIME   AGE
default       rbd-pvc-snapshot                     rbd-pvc                                           csi-rbdplugin-snapclass                                    6d
test-nginx    velero-ceph-ext-bcmqn                ceph-ext                                          csi-rbdplugin-snapclass                                    52m
test3-nginx   velero-ceph-ext-lfplp                ceph-ext                                          csi-rbdplugin-snapclass                                    5d23h
root@k8s-storage-1

Status — Not working. Not able to backup

time="2020-09-07T09:58:28Z" level=info msg="Waiting for CSI driver to reconcile volumesnapshot test-nginx/velero-ceph-ext-bcmqn. Retrying in 5s" backup=velero/test-nginx-csi cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:172" pluginName=velero-plugin-for-csi
time="2020-09-07T09:58:33Z" level=info msg="Waiting for CSI driver to reconcile volumesnapshot test-nginx/velero-ceph-ext-bcmqn. Retrying in 5s" backup=velero/test-nginx-csi cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:172" pluginName=velero-plugin-for-csi
time="2020-09-07T09:58:38Z" level=info msg="Waiting for CSI driver to reconcile volumesnapshot test-nginx/velero-ceph-ext-bcmqn. Retrying in 5s" backup=velero/test-nginx-csi cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:172" pluginName=velero-plugin-for-csi
...
time="2020-09-07T10:06:08Z" level=error msg="Timed out awaiting reconciliation of volumesnapshot test-nginx/velero-ceph-ext-bcmqn" backup=velero/test-nginx-csi cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:194" pluginName=velero-plugin-for-csi
time="2020-09-07T10:06:08Z" level=info msg="1 errors encountered backup up item" backup=velero/test-nginx-csi logSource="pkg/backup/backup.go:444" name=nginx-test

Bug raised https://github.com/vmware-tanzu/velero-plugin-for-csi/issues/69

root@k8s-storage-1:~# k describe volumesnapshot -n test-nginx
Name:         velero-ceph-ext-bcmqn
Namespace:    test-nginx
Labels:       velero.io/backup-name=test-nginx-csi
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1beta1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2020-09-07T09:56:08Z
  Generate Name:       velero-ceph-ext-
  Generation:          1
  Managed Fields:
    API Version:  snapshot.storage.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
        f:labels:
          .:
          f:velero.io/backup-name:
      f:spec:
        .:
        f:source:
          .:
          f:persistentVolumeClaimName:
        f:volumeSnapshotClassName:
    Manager:         velero-plugin-for-csi
    Operation:       Update
    Time:            2020-09-07T09:56:08Z
  Resource Version:  3883906
  Self Link:         /apis/snapshot.storage.k8s.io/v1beta1/namespaces/test-nginx/volumesnapshots/velero-ceph-ext-bcmqn
  UID:               64f97657-47cc-45a9-9f3e-fddee6bac425
Spec:
  Source:
    Persistent Volume Claim Name:  ceph-ext
  Volume Snapshot Class Name:      csi-rbdplugin-snapclass
Events:                            <none>

Velero with CSI

Written by Alex Punnen