How to write Kubernetes custom controllers in Go

Anartz Nuin
Mar 28, 2019 · 3 min read
Photo by Patryk Grądys on Unsplash

A Kubernetes controller is an active reconciliation process. It watches the shared state of the cluster through the API server and makes changes attempting to move the current state towards the desired state.

The intention of this article is to provide an introduction on how to write Kubernetes custom controllers using the official API client library.


The client-go library provides access to Kubernetes RESTful API interface served by the Kubernetes API server. Well-known tools like kubectl use it intensively.

It is important to note that the recommended version for client-go and some of its dependent packages and, depend on the Kubernetes version being used. Fortunately, client-go comes with a handy compatibility matrix which tells us exactly which version combinations are fully compatible (see figure below).

client-go compatibility matrix (source:

If, for example, we would like to write a custom controller for Kubernetes 1.12 we would choose v9.0, kubernetes-1.12.X and kubernetes-1.12.X

Controller structure

In the official Kubernetes GitHub repository, we can find a nice controller example, sample-controller. This repository provides a diagram which helps to understand the underlying components of a typical Kubernetes controller.

Kubernetes controller diagram(source:

As we can see in the figure above, many of the components involved (top half) are already provided by the client-go library.

We can identify two main tasks in the controller workflow:

  1. Use informers to keep track of add/update/delete events for the Kubernetes resources that we want to know about. “Keeping track” involves storing them in a local cache (thread-safe store) and also adding them to a workqueue.
  2. Consume items from the workqueue and process them.

client-go provides informers for standard resources, for example, deployments: These informers contain everything needed in the top half of the diagram, they provide the reflector, indexer and local storage.


In order to illustrate this better let’s take a look at the sample-controller implementation. The sample-controller ensures that for every Foo resource there’s a corresponding Deployment with the number of replicas specified by the Foo resource.

Foos are custom resources. The sample-controller example also shows how to deal with custom resources when writing custom controllers.

The code in the sample controller needs to watch both Deployments and Foos to ensure both remain synced.

If we have a look at the NewController function, we can notice how a DeploymentInformer and a FooInformer are set up. Different resource handler functions are passed to deal with add, update, and delete events. These handler functions are responsible for enqueueing the key of Foo objects in a workqueue if there’s a need for some processing. Informers hold a reference to a Lister, which is a convenient way of accessing items in the local cache storage. Using local cache storage reduces the amount of traffic that the Kubernetes API server needs to handle.

The controller also runs several Goroutines which consume the workqueue by constantly calling processNextWorkItem(). The workqueue only holds the key of the object to process. The complete object is retrieved using the informer’s Lister.

func (c *Controller) runWorker() { 
for c.processNextWorkItem() {

processNextWorkItem() ends up calling syncHandler() which is responsible for ensuring that there’s a Deployment associated with the Foo object being processed. It also makes sure that the number of replicas in the Deployment is the same amount specified in the Foo object.


client-go is a very powerful library for developing Kubernetes custom controllers. It comes with all the components needed for implementing the correct workflow in an efficient manner. However, it requires developers to learn low-level details about how Kubernetes libraries are implemented and write some boilerplate code. With this being said, there are some helpful initiatives trying to provide a better development experience, such as Kubebuilder, which may be worth exploring.


Learn about all things machine learning, artificial…


Learn about all things machine learning, artificial intelligence, speech recognition, data and much more.

Anartz Nuin

Written by

Anartz Nuin is a Software Development Engineer at Speechmatics. Speechmatics is a machine learning company which specialises in automatic speech recognition.


Learn about all things machine learning, artificial intelligence, speech recognition, data and much more.