techlog
Published in

techlog

What’s inside etcd? A deep dive into the Kubernetes world

Having been away from working on K8s for a while, I decided to cut open K8s to see how it works to try falling in love with it again. This is going to be a series of blogs, with the first surgery being trying to figure out what’s stored inside the etcd cluster.

What is etcd?

Etcd is defined as a distributed, reliable key-value store for the most critical data of a distributed system. Etcd is written in Go.

How does etcd work inside kubernetes?

Our friends at Heptio have a great blog where they have shown the components involved during a simple Pod creation process. It’s a great illustration of the API Server and etcd interaction.

Installing etcd

curl -L https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcdv3.3.11-linux-amd64.tar.gz -o etcd-v3.3.11-linux-amd64.tar.gz
tar xzvf etcd-v3.3.11-linux-amd64.tar.gz

Operate ETCD

./etcd
// Set a key
./etcdctl set key1 value1
// Get a key
./etcdctl get key1

The Kubernetes Test Cluster

minikube start — vm-driver=virtualbox
$ kubectl get pods -n kube-systemNAME                               READY     STATUS    RESTARTS   AGE
coredns-5644d7b6d9-2qg54 1/1 Running 0 2d
coredns-5644d7b6d9-6mbqk 1/1 Running 0 2d
etcd-minikube 1/1 Running 0 2d
kube-addon-manager-minikube 1/1 Running 0 2d
kube-apiserver-minikube 1/1 Running 0 2d
kube-controller-manager-minikube 1/1 Running 0 20s
kube-proxy-kgt6n 1/1 Running 0 2d
kube-scheduler-minikube 1/1 Running 0 2d
storage-provisioner 1/1 Running 0 2d
kubectl exec -it etcd-minikube -n kube-system sh
etcdctl get /
# echo “$(ps aux)”USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root 1 2.9 2.3 10611260 46916 ? Ssl 10:24 7:04
etcd --advertise-client-urls=https://192.168.99.100:2379 --cert-file=/var/lib/minikube/certs/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/minikube/etcd --initial-advertise-peer-urls=https://192.168.99.100:2380 --initial-cluster=minikube=https://192.168.99.100:2380 --key-file=/var/lib/minikube/certs/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://192.168.99.100:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://192.168.99.100:2380 --name=minikube --peer-cert-file=/var/lib/minikube/certs/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/var/lib/minikube/certs/etcd/peer.key --peer-trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt
# ADVERTISE_URL=”https://192.168.99.100:2379"# ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
--cacert /var/lib/minikube/certs/etcd/ca.crt \
--cert /var/lib/minikube/certs/etcd/server.crt \
--key /var/lib/minikube/certs/etcd/server.key \
get / — prefix — keys-only
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
/registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io
# ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
--cacert /var/lib/minikube/certs/etcd/ca.crt \
--cert /var/lib/minikube/certs/etcd/server.crt \
--key /var/lib/minikube/certs/etcd/server.key \
get / — prefix — keys-only -w json > out.json
{
"header": {
"cluster_id": 12197035334886545600,
"member_id": 9217530203749069991,
"revision": 19300,
"raft_term": 2
},
"kvs": [{
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEu",
"create_revision": 12,
"mod_revision": 12,
"version": 1
}, {
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYWRtaXNzaW9ucmVnaXN0cmF0aW9uLms4cy5pbw==",
"create_revision": 9,
"mod_revision": 9,
"version": 1
}, {
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYXBpZXh0ZW5zaW9ucy5rOHMuaW8=",
"create_revision": 10,
"mod_revision": 10,
"version": 1
}]
}
# ETCDCTL_API=3 etcdctl — endpoints $ADVERTISE_URL — cacert /var/lib/minikube/certs/etcd/ca.crt — cert /var/lib/minikube/certs/etcd/server.crt — key /var/lib/minikube/certs/etcd/server.key get /registry/apiregistration.k8s.io/apiservices/v1.{
"kind": "APIService",
"apiVersion": "apiregistration.k8s.io/v1beta1",
"metadata": {
"name": "v1.",
"uid": "fd15144e-ef8c-4a02–87fc-0fc72c178118",
"creationTimestamp": "2019–11–22T15:19:43Z",
"labels": {
"kube-aggregator.kubernetes.io/automanaged": "onstart"
}
},
"spec": {
"service": null,
"version": "v1",
"groupPriorityMinimum": 18000,
"versionPriority": 1
},
"status": {
"conditions": [{
"type": "Available",
"status": "True",
"lastTransitionTime": "2019–11–22T15:19:43Z",
"reason": "Local",
"message": "Local APIServices are always available"
}]
}
}
  • Nodes
  • Namespaces
  • ClusterRoles
  • ClusterRoleBindings
  • ConfigMaps
  • Secrets
  • Workloads: Deployments, DaemonSets, Pods

Summary

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store