Unveiling the Kubernetes Orchestra: A Deep Dive into Internal Workings and Communication

8 min readApr 6, 2024

Kubernetes has revolutionized containerized application deployment, but its magic lies in the intricate collaboration of its internal components. This blog post provides a comprehensive view of internal processes and communication within Kubernetes by diving deep into the heart of Kubernetes, exploring the communication symphony between components, the internal processes that orchestrate them.

The Grand Architectural Stage:

Imagine Kubernetes as a grand orchestra, with two main sections working in perfect harmony:

Control Plane: The conductor, responsible for managing the overall cluster state and issuing commands to worker nodes. Key players include:

etcd: The highly available key-value store, acting as the single source of truth for cluster data (think of it as the sheet music for the entire orchestra).
kube-apiserver: The API gateway, a RESTful interface for interacting with the cluster using tools like kubectl (the conductor’s stand, where instructions are received and relayed).
kube-scheduler: Assigns pods (containers grouped for deployment) to suitable worker nodes based on defined scheduling policies (the matchmaker, ensuring instruments are placed with the right musicians).
kube-controller-manager: Runs a set of controllers that manage cluster resources like deployments and replica sets (groups of identical pods) (the section leaders, overseeing specific sections of the orchestra).

Node Plane: The worker nodes, the physical or virtual machines that execute your containerized applications (the stage itself, where the music comes to life). Each node houses:

kubelet: The agent on each node, responsible for managing pods, fetching container images, and ensuring pod health (the stage manager on each node, ensuring everything runs smoothly).
kube-proxy: Manages network traffic by implementing service discovery (locating pods) and load balancing for pods behind a service (the network director, guiding audience traffic to the right performers).
Container Runtime: Docker, containerd, or any compliant runtime that executes containers within pods (the individual musicians, bringing the music to life).

The Communication Symphony in Exquisite Detail:

Setting the Stage (Deployment): You interact with the API server using kubectl commands or programmatic API calls to define your application resources (deployments, pods, services) (providing the sheet music and assigning roles to the musicians).
API Server — Validating the Score: The API server meticulously performs Authentication, Authorization and Admission Control on your request against defined schemas and ensures it adheres to security protocols using RBAC (Role-Based Access Control) (checking the music for accuracy and ensuring everyone has the proper permissions).
Storing the Composition (etcd): Upon successful validation, the API server stores the resource object (deployment details) in etcd, ensuring all components have access to the latest information (filing the sheet music for easy reference).
Controllers Take the Stage: Controllers like the deployment controller spring into action. They retrieve the deployment object from etcd and analyze the desired state (number of replicas and container specifications) (the section leaders reviewing the sheet music and understanding the desired performance).

The Deployment Controller’s Performance in High Definition:

Continuous Monitoring: The deployment controller enters a continuous loop, constantly watching for changes to deployment objects in etcd using the watch API (acting as a vigilant observer, ensuring everything stays in sync).
Reconciliation Symphony: This loop involves a meticulous reconciliation process:
Understanding the Score: The controller fetches the latest deployment object from etcd to understand the desired state (number of replicas, container specifications) (carefully reviewing the sheet music).
Checking the Instruments (Existing Pods): The controller queries the API server for existing pods associated with the deployment using label selectors (tags identifying pods belonging to a particular deployment) (checking which instruments are currently on stage).
Harmonizing the Orchestra (Scaling): The controller compares the desired state with the actual number of running pods. If there’s a discrepancy, it initiates corrective actions:
Adding Musicians (Scaling Up): If the desired state requires more replicas, the controller creates new pod objects with the specified container image, resource requests, and labels tying them back to the deployment. These are then submitted to the API server (adding additional musicians to the orchestra based on the conductor’s instructions).
Reducing the Ensemble (Scaling Down): If the desired state requires fewer pods, the controller selects pods for deletion based on predefined pod termination grace periods and pod anti-affinity/affinity rules (ensuring remaining pods are spread across the cluster). These selected pods are marked for deletion in the API server (reducing the number of musicians if needed).

The Scheduler: Matching Pods to Nodes — A Conductor’s Meticulous Selection Process

Tuning the Instruments (Node Health Checks): Before assigning pods, the scheduler conducts health checks on available nodes. This might involve verifying kubelet status, resource availability (CPU, memory, disk space), and taints/tolerations on the nodes (labels indicating specific requirements or limitations). (The conductor ensures each instrument is in good working order and meets the technical requirements for the piece).
Selecting the Right Stage (Scheduling Plugins): The scheduler can be extended with custom plugins for sophisticated scheduling strategies. These plugins might consider factors like node labels (genre of music the musician specializes in), resource utilization history (how much energy a musician typically expends), or pod priority levels (importance of a particular instrument in the piece) during node selection. (The conductor can leverage additional tools to ensure the best possible fit for each musician based on their skills and the demands of the music).
Accommodating Soloists (Pod Preemption — Optional): In case of critical pod placement needs or resource emergencies, the scheduler can leverage pod preemption. This involves evicting a lower-priority pod from a node to make space for a higher-priority pod (similar to how a conductor might need to temporarily replace a musician for a crucial part of the performance).

The Kubelet: The Busy Stage Manager on Each Node — A Behind-the-Scenes Look

Monitoring the Performance (Pod Status Reporting): The kubelet on each node periodically fetches the assigned pod list from the API server. It also monitors the health of running containers within each pod using techniques like liveness and readiness probes. These probes determine if a container is healthy and ready to receive traffic (ensuring each musician is performing at their best and ready for the audience).
Directing the Band (Container Runtime Integration): The kubelet interacts with the container runtime (Docker, containerd) to manage the container lifecycle. This includes creating containers from downloaded images, starting/stopping containers, and managing container logs (providing clear instructions to each musician and overseeing their performance).
Securing the Show (Secret and ConfigMap Management): Kubelet can fetch secrets (sensitive information like passwords) and config maps (configuration data) from the API server and mount them as volumes within containers, providing them with the necessary resources securely (ensuring the musicians have access to the sheet music and any confidential information they need, while keeping everything secure).
Setting Up the Backstage (CNI Integration): For pod networking, the kubelet works with the CNI plugin (e.g., Flannel, Calico) to configure network interfaces and pod IP addresses, enabling container-to-container communication within the cluster (setting up the backstage area and ensuring clear communication channels between the musicians).

Service Discovery and Network Routing: A Guide for the Audience

Defining the Stage Lights (Service Object): A service object typically defines a selector that identifies pods behind the service. It might also specify ports for exposing the service and load balancing policies (round-robin, least connections) to distribute traffic efficiently (designating which section of the audience the music will be directed to and ensuring everyone has a good chance of hearing the performance).
The Understudy (Kube-proxy): Kube-proxy on each node acts as the understudy, constantly monitoring for changes in service objects and endpoints (which list the actual pods behind the service) in the API server (staying informed about any changes to the stage setup or performer assignments).
Adapting to the Venue (Service Type):The way kube-proxy configures network rules depends on the chosen service type, which determines how external users access your application within the Kubernetes cluster. Here’s a breakdown of the three main service types and their functionalities:
LoadBalancer (External Traffic Routing): Kube-proxy might interact with a cloud provider API to provision a load balancer that routes external traffic to the service (like hiring an external company to manage ticket sales and direct the audience to the venue).
NodePort (External Access Through Specific Port): It exposes the service on a specific port on each node, allowing access from anywhere on the internet using that port (similar to displaying the show’s name and designated entrance on the outside of the venue).
ClusterIP (Internal Service Communication): This service type is like having a dedicated section within the venue accessible only to those with backstage passes. It’s primarily used for internal communication between pods within the Kubernetes cluster. Kube-proxy assigns a virtual IP address (ClusterIP) to the service, which is only routable from within the cluster network.
Guiding the Audience (Network Routing): Kube-proxy utilizes techniques like iptables or CNI plugins to manipulate routing tables and network policies. This ensures incoming traffic destined for the service reaches the appropriate pods based on the load balancing strategy (directing the audience to the correct section of the seating based on their ticket and ensuring a smooth flow of traffic).

Advanced Concepts: Delving Deeper into the Orchestra Pit

Persistent Volumes (PVs) and Persistent Volume Claims (PVCs): Imagine your musicians need sheet music stands and instrument cases. PVs act as the physical storage locations for this data, while PVCs are the requests from pods (musicians) specifying the storage they need. This ensures their music and instruments are readily available across performances.
Custom Resource Definitions (CRDs): Need a new type of performer in your orchestra? CRDs allow you to extend Kubernetes with new resource types tailored to your specific needs. This opens doors for even more creative and diverse musical experiences.
Cluster Autoscaler: Concerts can fluctuate in size. The cluster autoscaler dynamically provisions and removes nodes based on resource utilization, ensuring you have the right number of musicians for each performance, optimizing costs and efficiency.

Conclusion: The Encore — A Harmonious Performance

By understanding the intricate communication between Kubernetes components and their internal processes, you gain the power to:

Troubleshoot issues effectively: When a note is off-key, you need to diagnose the problem. This deep dive equips you to pinpoint issues within the orchestra and ensure a smooth performance.
Optimize deployments: Just like a conductor refines the performance, you can optimize deployments by understanding resource allocation and scheduling strategies.
Unlock the full potential of Kubernetes: With a deeper understanding, you can leverage Kubernetes’ flexibility and scalability to create a truly remarkable musical experience for your applications.

Feel free to share your thoughts and in the comments below!