Extend your cluster behavior by writing a Kubernetes controller in Go

Published in

HungerStation

7 min readFeb 28, 2023

I recently had to extend my Kubernetes cluster by applying a custom behavior that would fit my needs and I must admit that finding out how to do it was very difficult. Kubernetes was still relatively new to me and after taking a step back on the subject, I wanted to write this article to try to demystify the concept as I would have liked to discover it when I started to be interested in it.

In this article, I will limit myself to the use of the controller-runtime framework that was recommended to me to take my first steps. The latter is based on the kubebuilder and operator-sdk projects and shares the main concepts.

WHAT KUBERNETES CONTROLLERS ARE.

A Kubernetes controller is a program that endlessly scans the resources of a Kubernetes cluster. When these are modified, it then interacts with other resources in order to reach a desired state. This state is specific to each controller and varies according to the need.

Kubernetes already runs several controllers natively such as the Replication Controller. To use our definition, when scaling a Replica Set to 5 Pods, the Replication Controller will list the pods already present in this Replica Set and create the number necessary to be at 5. It will then reach its desired state and do everything possible to maintain it.

One commonly finds controllers which facilitate the installation and the evolution of complex architectures. They then generally scrutinize CRs (Custom Resources) and adapt configurations / architectures. We quite often find this operation for the establishment of clusters for example. The CR then contains an abstract description of the cluster and the operator takes care of applying the effective changes to the Configmaps ; the Ingresses ; etc

The concept of the controller is very wide and you can use it without CRs too. We can imagine that a controller automatically modifies the configuration of a pod when an Ingress is modified for example.

SETTING UP THE CONTROLLER-RUNTIME

The controllers rely heavily on the Kubernetes API to manipulate the various resources (get, update, create …) and we’ll make these calls through a Kubernetes client . I chose to use the Go client in order to stay in the Kubernetes ecosystem, but there are some for several languages (see the official list).

A FIRST CONTROLLER

Let’s get straight to the heart of the matter. Here is a first example of a controller that scans the Ingress of a Kubernetes cluster.

Create a new folder in the directory of your choice and initiate a new project via go mod init <project_name>

Create a file main.goand put the following content in it:

package main
import (
        netv1 "k8s.io/api/networking/v1"
        "context"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "sigs.k8s.io/controller-runtime/pkg/client/config"
        "sigs.k8s.io/controller-runtime/pkg/controller"
        "sigs.k8s.io/controller-runtime/pkg/handler"
        "sigs.k8s.io/controller-runtime/pkg/manager"
        "sigs.k8s.io/controller-runtime/pkg/manager/signals"
        "sigs.k8s.io/controller-runtime/pkg/reconcile"
        "sigs.k8s.io/controller-runtime/pkg/source"
)
type ingressReconciler struct {
        client client.Client
}
func (r *ingressReconciler) Reconcile(ctx context.Context, request reconcile.Request) (reconcile.Result, error) {
        // your reconciliation logic
        res := reconcile.Result{}
        return res, nil
}
func main() {
        mg, err := manager.New(config.GetConfigOrDie(), manager.Options{}}
        if err != nil {
          // handle err
        }
        myController, _ := controller.New("awesome-controller", mg, controller.Options{
                Reconciler: &ingressReconciler{
                        client: mgr.GetClient(),
                },
        })
        myController.Watch(&source.Kind{Type: &netv1.Ingress{}}, &handler.EnqueueRequestForObject{})
        mg.Start(signals.SetupSignalHandler())
}

Install the different packages imported with go get <pkg_name>(to go faster, you can install the main package directly sigs.k8s.io/controller-runtime)

These packages contain everything needed to use a Kubernetes client; configure it to connect to a cluster; manipulate Ingresses; configure and start our controller.

Our first controller does almost nothing for the moment, but it is already a largely sufficient basis to introduce some essential notions to its good understanding.

MAIN CONCEPTS

Work Queue

The controller has a Work Queue to store the different events associated with the resources it scans. Each time an event is popped, the controller triggers a reconciliation by calling the method Reconcile().

Reconciliation

Reconciliation consists in matching the current state (before the modification of the resource) with the expected state. The method Reconcile() will contain the logic of our controller, ie the algorithm that describes how to reach this state.

It is important to understand that reconciliation is triggered each time an event occurs on the type of resource scanned. If we scan the Ingress and that IngressA and IngressBare modified, will be Reconcile() executed 2 times in a row and on each call will be an object reconcile.Request which will contain 1 resource name and 1 namespace — generally those of the resource concerned by the event — which allow us to identify this resource.

COMPONENTS

To be able to instantiate our controller, we must first create the following elements:

A manager who will provide a client to the controller and will manage the via cycle of the latter (it starts via manager.Start()in particular).
A handler that will queue reconciliation requests reconcile.Requests.
A controller that implements the Kubernetes APIs. It is he who allows you to scan resources via controller.Watch()

Manager

The manager can be configured via many options which are however not necessary for this first controller. In our example, it obtains the configuration of the client via the method config.GetConfigOrDie() which:

Tries to use $HOME/.kube/configif the controller is running outside a kubernetes cluster (another path can be specified using the argument — kubeconfig <path>)
Automatically retrieves connection information if the controller is running in a kubernetes cluster.

Attention

Check the contents of your kube config before testing your controller locally. Pay close attention to the context used to avoid possible surprises in production!

Handler

The handler supports different modes of operation that affect the values contained in reconcile.Request. More precisely, these operating modes define which objects will be reconciled when a resource is modified.

With EnqueueRequestForObject the reconciled object is the one affected by the event. If toto the namespace tata has been modified, these values will be found in reconcile.Request.
With EnqueueRequestForOwner the reconciled object will be the Owner of the object concerned by the event. It is therefore the name and namespace of the parent resource that will be in reconcile.Request.
With ‘[’ we determine a list of objects to reconcile by declaring a function. There are thus several couples {Name;Namespace} to be reconciled by events.

SCAN RESOURCES

Now let’s take a closer look at the Watch() method definition.

func Watch(src source.Source, eventhandler handler.EventHandler, predicates ...predicate.Predicate) error

It takes the following as arguments:

An event source, which can be internal to the cluster (eg pod creation, deployment modification, etc.) using source.Kind and external (ex. github hook) using source.Channel
A handler
(optional) Predicates that define which events trigger a reconciliation. For example, it is possible to disable reconciliation for events Delete if desired.

For our first controller, we will scan an internal type source Ingress and our handler will operate in mode EnqueueRequestForObject.

MANIPULATE KUBERNETES OBJECTS

To handle a Kubernetes object, you must first import the package corresponding to its Kubernetes API (ie networking for Ingresses; apps for Deployments; etc.) in order to access the different resource structures.

Let’s focus this time on the method Reconcile() in which we implement our logic. We will retrieve a resource with client.Get() which takes as argument one Namespaced Name (which simply represents a couple {Name;Namespace}).

Note

Remember that reconcile.Request is not the representation of the object being reconciled. It only contains what is needed to identify a resource.

func (r *ingressReconcilier) Reconcile(ctx context.Context, request reconcile.Request) (reconcile.Result, error) {
 // Get the ingress
 ig := &networkingv1.Ingress{}
 r.client.Get(ctx, request.NamespacedName, ig)
 // Fill Annotations and update the object
 ig.Annotations = map[string]string{
  "hey": "you",
 }
 r.client.Update(ctx, ig)
 
 return reconcile.Result{}, nil
}

Once the object has been retrieved, it can be manipulated and its status changed via the various methods — Update(); Patch(); Delete(); etc — of the client and we end the reconciliation by returning a couple (reconcile.Result{}, error).

To finalize the explanation, our controller scans the Ingresses and triggers reconciliations each time he receives an event on them. Reconciliation consists of recovering the Ingress which has been modified and ensuring that it has a label hey:you, nothing more.

Note

When the controller starts, it triggers reconciliations for all resources of the scanned type that already exist in the cluster to start from a consistent state.

All you have to do is make sure that the account used by the controller to access the resources has enough rights to function correctly.

CONCLUSION

That’s about all I need to start having fun. The rest is up to you to write. If I had to give you just one piece of advice, it would really be to explore the documentation for the different controller-runtime packages and the packages for the different Kubernetes APIs. They contain all the information you need.

Today we looked at a basic use case adding annotations to the ingress to be able to understand the structure and master how to set up the controller code, but indeed this behavior can be extended to do any type of customization we want with our cluster.

NOTE : Some of the things that you can use an operator to automate include:
- deploying an application on demand
- taking and restoring backups of that application's state
- handling upgrades of the application code alongside related changes such as database schemas or extra configuration settings
- publishing a Service to applications that don't support Kubernetes APIs to discover them
- simulating failure in all or part of your cluster to test its resilience… and many more.