Choreography Inside Kubernetes

There would probably be thousands of articles about K8s to date. You may ask — What would this story be about then? :) This story would be all about my appreciation for the architecture and system design of Kubernetes.

All the people that have started the k8s journey would have figured out the components of the master plane and the worker plane. Less do we know about

1. What the system design looks like?
2. How do the components interact with each other?
3. What kind of message exchange happens?
4. The sequence of events that would happen to complete an action (against the cluster)?

Answers to these questions has helped me understand k8s better. K8s master plane components are setup in a “Hub and Spoke” model. This would resonate well with architects and networking people. The “API server and the etcd data store” form the hub of the master plane. The controller manager (deployment controller, replica controller) and scheduler form the spokes. How does this setup work?

The spoke components do not directly interact with each other. Any message exchange would happen through the API server & etcd. This makes the flow of events inside k8s a well designed choreography (explained in detail in the next section). If you had to visualize the setup,

For Visualization purposes only, not from the official documentation

Getting back to the part why I call this a nicely designed choreography, each component has a clearly established responsibility inside K8s.

1. The execution of the tasks of any component would start when an event that the component has subscribed to is published
2. Components after completing their work (business logic in k8s world) would then perform a REST call to the API server. The API server is a powerful component that exposes REST APIs in a lot of well separated namespaces ( each namespace housing the APIs of a specific component or functionality)
3. The page mentioned below(not the latest version but is a “wealth of information”) provides a clear picture of all the API endpoints, their definitions and additional details of the parameters of the k8s apis. What if you had a swagger file of the APIs exposed from the api server ??

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#-strong-api-overview-strong-

4. The API server would now act on the https api call and after processing the request would create another event for the downstream components. This flow keeps going until the intended action is completed.

Take a look at this diagram and you would know why I used the terms “Choreography” and “Hub and Spoke” . These are sequence of actions that take place when a “Pod Deployment” is executed.

src: https://www.codetd.com/en/article/8324718

Majority of the actions in k8s work based on a timed notification mechanism called “watch”. When we think of any notification mechanism, there would of course be subscribers and publishers. You should have by now inferred that the Hub component (API server and etcd store)acts as the publisher and the spoke components, the subscribers.

When you execute a yaml template ( through deployment pipelines or from the command line), the following actions are executed

1. The yaml is converted to json and sent to the api server
2. the api server (after validating the payload) would now save the requested deployment settings to the etcd database ( this is the first step in creating the “desired state” of the cluster
3. the etcd database creates an event for the key-value data for the cluster resource that got added/updated/deleted. The consumer of this is the API sever
4. The API server in addition to hosting the REST backend also has the “watch” mechanism built into it. Now the events would be sent to the other components by the “ClientGo” component (watch the video linked to from the next section)
5. The controllers on receiving the events perform a sequence of operations ( displayed in the diagram shown below). Interesting part of this step is the the system design of the controller. You will appreciate the simplicity and beauty of it too!
6. The controllers after completing their defined tasks, send an update back to the API server. This is saved in the etcd as an update version. The controllers in each step help in getting the “actual state towards the desired state”

You can get a picture of the details of the watch mechanism, flow of events and the system design of controllers from this video. An amazing one in my opinion

The Life of a Kubernetes Watch Event — Wenjia Zhang & Haowei Cai, Google

Representation of the diagram shown in the video at 21:08

Among the plenty of things that you would learn from this design (take time to digest the information in the video), one “design decision” that is worth a mention is that — The controllers are “Level based systems” and not “Edge based systems”

Not intending to duplicate any information, I urge you to read this amazing article — resources and controller

Emphasis is on the design decision though. The controller does not work on the events that it receives. Events work just as a change notification source.Reconciliation is performed using the data in the “Local Cache”. This is the reason that the local cache is kept updated with every change that happens in the current state of the system that is of interest to the listening controller.

An excerpt from the same amazing article about the watch action( point of interest of each component in the sequence). You can correlate this with the second diagram (sequence) in this article.

• Deployment Controller watches Deployments + ReplicaSets (+ Pods)
• ReplicaSet Controller watches ReplicaSets + Pods
• Scheduler (Controller) watches Pods
• Node (Controller) watches Pods (+ Secrets + ConfigMaps)

That was all the feel good information I wanted to share with my fellow enthusiasts. Hope you enjoyed this one too.

Additional Reference

https://medium.com/@tsuyoshiushio/kubernetes-in-three-diagrams-6aba8432541c