ETCD : Backup and Restore 📦📦

How to backup and restore the ETCD datastore for a Kubernetes Cluster ☸☸ ??

Introduction 🚩🚩

Hi fellow Readers 👋 :))
I am trying to be better with cloud native technologies and stuff. Thus I am exploring a lot of stuff to expand my horizon. In my previous articles, I have heavily shared my knowledge about Kubernetes.

Continuing on those lines, Today I will be again writing about another major concept which is being widely used in the world of Kubernetes. This article is about how to backup and restore the ETCD datastore for a Kubernetes Cluster.

Taking Timely Backup of the ETCD datastore keeps the stable instances of the cluster safe. It can help to restore the cluster back to a good state.
Hence it is an important concept to know 😎.

So In this article, we will try to learn :

• What is ETCD datastore in Kubernetes ?
• Prerequisites for the whole process.
• How to take backup of the ETCD datastore ?
• How to restore the ETCD datastore from backup file ?

Before starting with this Article.
I just wanna say that if you have been following me in my Technical Journey and if you like my writings and want to read more from me in the future :))
Please Do clap and follow me 🙈.

Plus In case of any doubts around this article or for some general chit chat, feel free to reach out to me on my social media handles😃.

Twitter — https://twitter.com/ChindaVibhor

LinkedIn — https://www.linkedin.com/in/vibhor-chinda-465927169

It will be a small but an extremely interesting article.
So without any further delay, lets get started with it 🐵

What is ETCD datastore in Kubernetes ? 👀👀

etcd datastore

etcd is an open source distributed key-value store used to hold and manage the critical information that distributed systems need to keep running. Most notably, it manages the configuration data, state data, and metadata for Kubernetes, the popular container orchestration platform.

In simple terms :

• etcd is a Key — Value Datastore.
• Configuration data and information about the state of the cluster lives in it.
• Fault-tolerant and distributed.
• etcd is designed to be the ultimate source of truth about your cluster.
• For eg → When user runs “kubectl get pods” command, they get the answer from the values stored in the etcd.

Now we know what is an etcd datastore. 🎇 ✨
Let us now learn how to take its backup and restore from the taken backup

Prerequisites for the whole process 😐😮

Photo by Jake Hills on Unsplash

Note** : I will be using the Kubeadm Kubernetes setup for this tutorial.
I am using killercoda to run my labs. You can try to 😄

• Have the etcdctl installed :
  etcdctl: a command line tool for interacting with the etcd server

etcdctl version

If etcdctl is installed, you will see an output like below :

If not it will give a “command not found” error.

You can install etcdctl by following the steps provided in this url. You can find installation guides for Linux, MacOS, and Docker.

• Fetching endpoint and the certificate information :

Open the etcd manifest file which is located at :
/etc/kubernetes/manifests location

We can retrieve information regarding the endpoints with the following command:

cat /etc/kubernetes/manifests/etcd.yaml | grep listen

Just check the value for “— — listen-client-urls”.

We can get the certificate information with the following command.

cat /etc/kubernetes/manifests/etcd.yaml | grep file

Just check the value for “ — — cert-file”, “ — — key-file” and “ — — trusted-ca-file”.

How to take backup of the ETCD datastore ? 😈😏

Now we have all the information required. We can take backup of the etcd by just using the below command :

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save <backup-file-location>

One thing to note here is that we need to place ETCDCTL_API=3 at the beginning of the command.

API version used by etcdctl to speak to etcd may be set to version 2 or 3 via the ETCDCTL_API environment variable. However, we need to make sure it is default to the v3 API in order to take a snapshot.

Before taking snapshot of etcd, let us create a pod for verifying purposes.

Use the below command :

kubectl run podbeforesnapshot --image=nginx

Now Let’s run the snapshot save command and save the snapshot as “backup.db” :

Well done !! We now successfully have the backup of the etcd 😄
Now let us learn to restore the ETCD datastore

How to restore the ETCD datastore from backup file ? 🤔🤔

• Before taking snapshot of etcd, let us create a pod for verifying purposes.

Use the below command :

kubectl run podaftersnapshot --image=nginx

Now, we can move on to actually restoring etcd.
Imagine that etcd somehow failed and we need to revert it to the last saved state. We know that we have an backup.db which we saved earlier.

• When we use etcdctl to restore the saved snapshot, we use almost the same command and certificate files as we used when took a snapshot of the etcd.

But this time, we need to provide a new data directory (“— — data-dir”) where we will copy the cluster data from the backup.db database.

ETCDCTL_API=3 etcdctl --data-dir="/var/lib/etcd-backup" \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore backup.db

• Now we need to replace the etcd directory value in the etcd yaml file with the above location where we have restored the backup.db datastore

Replace the values in the below file “/etc/kubernetes/manifests/etcd.yaml” :

Now we have successfully chnaged all the values. Just give some time and verify the whole cluster by running the below command :

kubectl get pods

Hurray !! 🥳🥳
We have successfully restored the etcd from the backup.db datastore 😎😎

What next ?? 👀 👀

Thanks a lot for reaching till here! This is the end of this article.
But we have only scratched the surface of the K8s ecosystem :))
Much more to go, it will be a fun journey where we will learn a lot of cool stuff together. 🚀 🚀

Taking it just one article at a time 😁
Do clap and follow me 🙈 if you like my writings and want to read more from me in the future :))

In case of any doubts around this article or for some general chit chat, feel free to reach out to me on my social media handles

Twitter — https://twitter.com/ChindaVibhor

LinkedIn — https://www.linkedin.com/in/vibhor-chinda-465927169/

Previous Articles :

Metrics-Server in Kubernetes ☸☸

How to Read a Go Program 💙💙

I will still keep on coming with new articles covering a bunch of topics I am exploring.

That’s All folks !! Doodles :))