Accelerating Open Source: Pluggable Clouds in Kubernetes
This is part I of a series of blog posts about making the cloud provider in Kubernetes pluggable. This blog post covers how the kube-controller-manager was made pluggable. For kubelet and kube-apiserver, refer to my next blog post in this series.
Kubernetes is one of the fastest growing GO projects in open source today¹. It has more than 1000 contributors and almost 50,000 commits. One of the challenges of working with such large open source projects is that it takes a non-trivial amount of time to add new code and iterate on it.
When I tried making a change in the cloud provider part of Kubernetes, thockin suggested that I figure out a way to make cloud provider pluggable instead. This was meant to address the numerous cloud vendors that wanted to integrate with Kubernetes quickly and efficiently. This would allow them to integrate with Kubernetes without adding any code upstream. This would also allow me to make the changes that I wanted. I was excited by the proposition, so I jumped on this project.
Now, 6 months later I have completed the implementation of pluggable cloud providers. I’m writing this blog post to discuss the key architectural and design decisions that went into making the changes. Specifically, I will dive into how the following components were changed:
- Kubernetes Controller Manager
- Kubelet
- Kubernetes API server
Kubernetes and Cloud: a special bond
Let’s start off by understanding how Kubernetes and the Cloud work with one another. In this section I’ll provide a brief introduction into the Kubernetes-Cloud relationship.
Kubernetes integrates with cloud providers to procure information about the nodes that are running in the cluster, provision load balancers available in the cloud, provision and configure storage volumes. In some cases, it configures the overlay network in the cloud to enable cross host communication between pods.
These actions are performed by multiple kubernetes components, namely the kube-controller-manager, kubelet and the kube-apiserver. This distributed nature of Kubernetes makes the transformation to plugin architecture interesting, because multiple integration points need to be converged into one plugin. Here’s a picture of the architecture of Kubernetes with integration points to the cloud denoted.
Kubernetes Controller Manager
As a developer, my favorite part of Kubernetes is its architecture. The various components in Kubernetes pull events from one central source of truth, and then constantly aim to match the state in the source of truth. This is in contrast to the commonly seen architecture of having a central authority that delegates tasks to the distributed components. This is not a new concept — but Kubernetes has done a really good job of implementing this architecture using Controllers.
The kube-controller-manager(or KCM) is an compendium of controllers, where multiple control loops aim to match the state in the source of truth. KCM has the highest number of integration points with the cloud, and when I started working on this project, I started by changing this component. In this section I shed light into the cloud specific duties of the KCM, and how it was made pluggable.
Let’s start off with some well known instances of the KCM’s duties:
- When a user requests a LoadBalancer service, the service controller in the KCM creates a load balancer in the cloud and configures it to balance requests among the pods selected by the service.
- When a node becomes unresponsive, the node controller within the KCM checks with the cloud to see if the node was deleted.
The following control loops in the KCM integrate with the cloud -
- Service Controller
- Route Controller
- Node Controller
- Persistent Volume Controller
The Service Controller subscribes to service create, update and delete events. In case any of those events occur, and if the service is a LoadBalancer service, then this controller talks to the cloud to provision, update or delete a load balancer (like ELB in AWS).
The Route Controller subscribes to node create events. In case any of that event occurs, then the Route Controller sets up routes in the cloud to enable containers to communicate across hosts.
The Node Controller has many responsibilities, the only pertinent one here is that if any node stops responding to health checks then it contacts the cloud to check if that node was deleted. If the node was indeed deleted from the cloud, then it deletes that node from Kubernetes.
Finally, the Persistent Volume Controller is responsible for managing the lifecycle of Persistent Volumes. It talks to the cloud to provision cloud provided volumes (like AWS EBS, and GCE PD), and to attach these volumes to the machines running the pods, or detach them if pods are moved to other machines.
Pluggable KCM: Enter Cloud Controller Manager
The various control loops communicate with the cloud using a common interface known as the Cloud Provider Interface (or CPI). When a cloud provider wants to add Kubernetes support for their cloud, they would have to write Go Structures that satisfy CPI.
Since the above mentioned controller loops are modular in nature, and they just need a CPI satisfying structure passed into their constructor, we decided to split the kube-controller-manager into two pieces.
The first piece will be called kube-controller-manager, for backward compatibility reasons, and it will house all the control loops that do not interact with the cloud.
The second piece will be called cloud-controller-manager (CCM), and it will house all the control loops(except Persistent Volume Controller) that interact with the cloud. The special property of CCM is that the CPI satisfying structures won’t reside in the Kubernetes repository. It will be maintained by the cloud vendors themselves. When the CCM is compiled, the CPI satisfying structure should be passed into the CCM through the constructor.
The Kubernetes repository will only maintain the libraries that the CCM requires, but it won’t maintain CCMs for the various clouds. This way, any cloud vendor can create their own CCM using the standard libraries from Kubernetes, and by passing in their own GO structure that satisfied CPI.
This is the new architecture of Kubernetes with the CCM and KCM.
Now that we’ve made the KCM cloud provider pluggable, this leaves the cloud provider integration points in the kube-apiserver and the kubelet
Stay tuned for my next blog post on making the Kubelet and Kube API server cloud pluggable!
