Published in


What’s inside etcd? A deep dive into the Kubernetes world

Having been away from working on K8s for a while, I decided to cut open K8s to see how it works to try falling in love with it again. This is going to be a series of blogs, with the first surgery being trying to figure out what’s stored inside the etcd cluster.

What is etcd?

Etcd is defined as a distributed, reliable key-value store for the most critical data of a distributed system. Etcd is written in Go.

How does etcd work inside kubernetes?

Our friends at Heptio have a great blog where they have shown the components involved during a simple Pod creation process. It’s a great illustration of the API Server and etcd interaction.

Installing etcd

curl -L -o etcd-v3.3.11-linux-amd64.tar.gz
tar xzvf etcd-v3.3.11-linux-amd64.tar.gz

Operate ETCD

// Set a key
./etcdctl set key1 value1
// Get a key
./etcdctl get key1

The Kubernetes Test Cluster

minikube start — vm-driver=virtualbox
$ kubectl get pods -n kube-systemNAME                               READY     STATUS    RESTARTS   AGE
coredns-5644d7b6d9-2qg54 1/1 Running 0 2d
coredns-5644d7b6d9-6mbqk 1/1 Running 0 2d
etcd-minikube 1/1 Running 0 2d
kube-addon-manager-minikube 1/1 Running 0 2d
kube-apiserver-minikube 1/1 Running 0 2d
kube-controller-manager-minikube 1/1 Running 0 20s
kube-proxy-kgt6n 1/1 Running 0 2d
kube-scheduler-minikube 1/1 Running 0 2d
storage-provisioner 1/1 Running 0 2d
kubectl exec -it etcd-minikube -n kube-system sh
etcdctl get /
# echo “$(ps aux)”USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root 1 2.9 2.3 10611260 46916 ? Ssl 10:24 7:04
etcd --advertise-client-urls= --cert-file=/var/lib/minikube/certs/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/minikube/etcd --initial-advertise-peer-urls= --initial-cluster=minikube= --key-file=/var/lib/minikube/certs/etcd/server.key --listen-client-urls=, --listen-metrics-urls= --listen-peer-urls= --name=minikube --peer-cert-file=/var/lib/minikube/certs/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/var/lib/minikube/certs/etcd/peer.key --peer-trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt
# ADVERTISE_URL=”"# ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
--cacert /var/lib/minikube/certs/etcd/ca.crt \
--cert /var/lib/minikube/certs/etcd/server.crt \
--key /var/lib/minikube/certs/etcd/server.key \
get / — prefix — keys-only
# ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
--cacert /var/lib/minikube/certs/etcd/ca.crt \
--cert /var/lib/minikube/certs/etcd/server.crt \
--key /var/lib/minikube/certs/etcd/server.key \
get / — prefix — keys-only -w json > out.json
"header": {
"cluster_id": 12197035334886545600,
"member_id": 9217530203749069991,
"revision": 19300,
"raft_term": 2
"kvs": [{
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEu",
"create_revision": 12,
"mod_revision": 12,
"version": 1
}, {
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYWRtaXNzaW9ucmVnaXN0cmF0aW9uLms4cy5pbw==",
"create_revision": 9,
"mod_revision": 9,
"version": 1
}, {
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYXBpZXh0ZW5zaW9ucy5rOHMuaW8=",
"create_revision": 10,
"mod_revision": 10,
"version": 1
# ETCDCTL_API=3 etcdctl — endpoints $ADVERTISE_URL — cacert /var/lib/minikube/certs/etcd/ca.crt — cert /var/lib/minikube/certs/etcd/server.crt — key /var/lib/minikube/certs/etcd/server.key get /registry/{
"kind": "APIService",
"apiVersion": "",
"metadata": {
"name": "v1.",
"uid": "fd15144e-ef8c-4a02–87fc-0fc72c178118",
"creationTimestamp": "2019–11–22T15:19:43Z",
"labels": {
"": "onstart"
"spec": {
"service": null,
"version": "v1",
"groupPriorityMinimum": 18000,
"versionPriority": 1
"status": {
"conditions": [{
"type": "Available",
"status": "True",
"lastTransitionTime": "2019–11–22T15:19:43Z",
"reason": "Local",
"message": "Local APIServices are always available"
  • Nodes
  • Namespaces
  • ClusterRoles
  • ClusterRoleBindings
  • ConfigMaps
  • Secrets
  • Workloads: Deployments, DaemonSets, Pods




Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store