How to write Kubernetes custom controllers in Go

Photo by Patryk Grądys on Unsplash

A Kubernetes controller is an active reconciliation process. It watches the shared state of the cluster through the API server and makes changes attempting to move the current state towards the desired state.

The intention of this article is to provide an introduction on how to write Kubernetes custom controllers using the official API client library.

client-go

The client-go library provides access to Kubernetes RESTful API interface served by the Kubernetes API server. Well-known tools like kubectl use it intensively.

Choosing the right version

It is important to note that the recommended version for client-go and some of its dependent packages k8s.io/api and k8s.io/apimachinery, depend on the Kubernetes version being used. Fortunately, client-go comes with a handy compatibility matrix which tells us exactly which version combinations are fully compatible (see figure below).

client-go compatibility matrix (source: https://github.com/kubernetes/client-go)

If, for example, we would like to write a custom controller for Kubernetes 1.12 we would choose k8s.io/client-go v9.0, k8s.io/api kubernetes-1.12.X and k8s.io/apimachinery kubernetes-1.12.X

Controller structure

In the official Kubernetes GitHub repository, we can find a nice controller example, sample-controller. This repository provides a diagram which helps to understand the underlying components of a typical Kubernetes controller.

Kubernetes controller diagram(source: https://github.com/kubernetes/sample-controller/blob/master/docs/images/client-go-controller-interaction.jpeg)

As we can see in the figure above, many of the components involved (top half) are already provided by the client-go library.

We can identify two main tasks in the controller workflow:

1. Use informers to keep track of add/update/delete events for the Kubernetes resources that we want to know about. “Keeping track” involves storing them in a local cache (thread-safe store) and also adding them to a workqueue.
2. Consume items from the workqueue and process them.

client-go provides informers for standard resources, for example, deployments: k8s.io/client-go/informers/apps/v1/DeploymentInformer. These informers contain everything needed in the top half of the diagram, they provide the reflector, indexer and local storage.

Example

In order to illustrate this better let’s take a look at the sample-controller implementation. The sample-controller ensures that for every Foo resource there’s a corresponding Deployment with the number of replicas specified by the Foo resource.

Foos are custom resources. The sample-controller example also shows how to deal with custom resources when writing custom controllers.

The code in the sample controller needs to watch both Deployments and Foos to ensure both remain synced.

If we have a look at the NewController function, we can notice how a DeploymentInformer and a FooInformer are set up. Different resource handler functions are passed to deal with add, update, and delete events. These handler functions are responsible for enqueueing the key of Foo objects in a workqueue if there’s a need for some processing. Informers hold a reference to a Lister, which is a convenient way of accessing items in the local cache storage. Using local cache storage reduces the amount of traffic that the Kubernetes API server needs to handle.

The controller also runs several Goroutines which consume the workqueue by constantly calling processNextWorkItem(). The workqueue only holds the key of the object to process. The complete object is retrieved using the informer’s Lister.

func (c *Controller) runWorker() { 
    for c.processNextWorkItem() { 
    }
}

processNextWorkItem() ends up calling syncHandler() which is responsible for ensuring that there’s a Deployment associated with the Foo object being processed. It also makes sure that the number of replicas in the Deployment is the same amount specified in the Foo object.

Conclusion

client-go is a very powerful library for developing Kubernetes custom controllers. It comes with all the components needed for implementing the correct workflow in an efficient manner. However, it requires developers to learn low-level details about how Kubernetes libraries are implemented and write some boilerplate code. With this being said, there are some helpful initiatives trying to provide a better development experience, such as Kubebuilder, which may be worth exploring.