Backing-up, restoring and migrating Kubernetes cluster with Velero📦 .

Velero is a light weight tool to safely backup, restore, handle cluster-failovers, and migrate Kubernetes cluster resources and persistent volumes.

All the source code used in this tutorial can be found here.

Kubernetes cluster with Kind

Kind is a light weight utility to create single-node Kubernetes cluster on a Docker container for testing purposes. Using Kind will allow us to create two test cluster of different versions quickly, thus allowing us to simulate cluster-migrations.

For installing kind:

• checkout kind-quickstart
• Or watch this video

CAUTION 🛑 🛑 :

• Make sure Docker is installed on your machine.
• If you are using Kubernetes(version 1.17), do check if coredns is working. For verifying status of coredns check this post here.

GCP service account to use with Terraform and Velero

A service account that has admin access to google cloud storage is required for Terraform to provision a bucket and for Velero to read/write backups to this bucket.

Open google cloud console and navigate to IAM & Admin>Service accounts

• Create a new service account.
• Give cloud storage admin permission to this account.
• Create a key for this account in json format and download it.
• Rename it to credentials.json and place it inside gcpServiceAccount folder.

Setting up storage plugin for Velero

Velero requires a storage site for pushing back-up files and retrieving them back in case of restoration. We’ll be using Google cloud storage bucket for this tutorial, but you can explore wide variety of storage plugin offered by Velero here.

Creating a storage bucket with Terraform

You can grab Terraform CLI from here or else use a Docker container that comes pre-installed with terraform. The infrastructure files for terraform are placed inside storage folder. Make sure your credentials.json is present inside gcpServiceAccount folder.

docker run -it --rm -v ${PWD}/storage:/storage -w /storage
akshit8/terraform

Note: akshit8/terraform is a Debian docker container installed with Terrafotm CLI (v 0.14.7).

Once the container has been created, run the following commands to create a storage bucket on GCP.

# to download gcp provider and any dependency
terraform init
# to apply changes on the cloud
terraform apply

If no error is thrown, you’ll be able to see a newly created bucket in your cloud console.

Test cluster with Kind(version 1.18)

With storage bucket in place, let us create a test-cluster with Kubernetes version 1.18.

kind create cluster --name test-cluster --image kindest/node:v1.18.0

Installing Kubectl and Velero CLI

To install both CLI’s we can use a Debian Docker container.

docker run -it --rm -v ${HOME}:/root/ -v ${PWD}:/work -w /work --net host debian:buster

mounting $HOME directory provides access to KUBE_CONFIG generated by Kind CLI.

• Installing Kubectl

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv ./kubectl /usr/local/bin/kubectl

To verify kubectl and our test-cluster, run following command

root@my-vm:/work# kubectl get nodes
NAME                         STATUS   ROLES    AGE     VERSION
test-cluster-control-plane   Ready    master   5m15s   v1.18.0

• Installing velero CLI

curl -L -o /tmp/velero.tar.gz https://github.com/vmware-tanzu/velero/releases/download/v1.5.1/velero-v1.5.1-linux-amd64.tar.gz
tar -C /tmp -xvf /tmp/velero.tar.gz
mv /tmp/velero-v1.5.1-linux-amd64/velero /usr/local/bin/velero
chmod +x /usr/local/bin/velero

Deploying Kubernetes objects in a sample namespace

Kubernetes objects used for this tutorial is located in k8s-objects folder.

kubectl create ns sample
kubectl -n sample apply -f ./k8s-objects

Configuring Velero for backing-up sample namespace

Using Velero CLI installed previously, we need to deploy some components(that velero use) inside our cluster and configure them, so that Velero can access our cloud storage bucket.

# setting the bucket name
export BUCKET=velero-akshit
# installing velero with provider gcp
velero install \
--provider gcp \
--plugins velero/velero-plugin-for-gcp:v1.1.0 \
--bucket $BUCKET \
--secret-file ./gcpServiceAccount/credentials.json

Note: it will create a new namespace velero to hold all components.

To verify above installation, run following commands

root@my-vm:/work# kubectl -n velero get pods
NAME                      READY   STATUS    RESTARTS   AGE
velero-86bb45cdfb-987ps   1/1     Running   0          23s
root@my-vm:/work# kubectl logs deployment/velero -n velero

If installation and connection to our storage bucket is successful, no error messages would be there inside deployment logs.

Backing-up the sample namespace

For adding sample namespace to Velero backup pool

velero backup create sample-namespace-backup --include-namespaces sample
velero backup describe sample-namespace-backup

If some error occurs, inspect the backup logs

velero backup logs sample-namespace-backup

Listing backups

velero get backups

Verify on Google Cloud Console

Our bucket contain backup files of all Kuberntes objects that were deployed inside sample namespace.

Deleting object inside sample namespace(cluster-failover)

kubectl -n sample delete -f ./k8s-objects

Let’s now recover the deleted objects with Velero.

velero restore create sample-namespace-backup --from-backup sample-namespace-backup

In case of any error, refer the logs

velero backup logs sample-namespace-backup

Let’s verify whether sample namespace has been restored or not

kubectl get all -n sample

Migrating cluster from version 1.18 to 1.19(cluster-migration)

As before we’ll use kind to spin another light weight cluster with version 1.19

kind create cluster --name test-cluster-2 --image kindest/node:v1.19.0

Check if the cluster is ready and accessible:

root@my-vm:/work# kubectl get nodes
NAME                           STATUS   ROLES    AGE    VERSION
test-cluster-2-control-plane   Ready    master   6m1s   v1.19.0

Installing Velero inside new cluster

• repeat the above steps to install Velero again
• make sure deployment logs displays no error
• verify all components inside namespace velero are running.

Migrating backup to new cluster

velero get backup

Starting the restore

velero restore create sample-namespace-backup --from-backup sample-namespace-backup

Verifying the restore

root@my-vm:/work/velero# velero restore describe sample-namespace-backup
Phase:  Completed
Started:    2021-02-24 09:52:47 +0000 UTC
Completed:  2021-02-24 09:52:48 +0000 UTC

Checking if all the components has been recovered

kubectl get all -n sample

Note: during migration, velero syncs with our storage bucket to get list of all backups, but it doesn’t apply or creates these backups automatically in your cluster.

Conclusion

• We have successfully simulated a cluster-failover and migration, while restoring our cluster back to original state.
• Velero can also backup stateful workloads and volumes. The focus of this tutorial was to only backup stateless workloads.

Source code for the above tutorial can be found below👇.

Akshit8/velero