Building Kubernetes Operators

This page is the second part of four-part series. Before reading it, please make sure you read the following:

1. Kubernetes Controllers, Custom Resources, and Operators Explained
2. Building Kubernetes Operators (you’re here)
3. Testing Kubebuilder Operators
4. Deploying Kubebuilder Operators on Kubernetes

The full code of this series can be found here!

For any questions you have, you can reach me via Linkedin or Twitter.

If you like my content and want to improve your skills, please look at the Devops Culture Project and get all the resources you didn’t know you needed.

Goal

On this page, I’m going to build with you a basic Kubernetes Operator using the Kubebuilder framework.

We are going to build a new CRD: AbstractWorkload .

This CRD will hold in its spec some data needed to deploy a workload to Kubernetes and which will require providing another field that will tell if we want a stateful/stateless deployment. It will then deploy by it a Kubernetes native Deployment or StatefulSet .

This is just an example of playing with Operators, this CRD is NOT a good idea to implement.

You might be very confused, but I promise you that through writing this code on your own and reading the provided links, you will have an excellent base for developing your own, real use-case operator.

Prerequisites

GitHub

Please make sure you have a GitHub account and the gh CLI installed:

brew install gh

Docker

Docker Desktop is an easy way of installing Docker.

kind

kind is a tool for running local Kubernetes clusters using Docker container “nodes”.

Install kind CLI through your OS package manager (installation docs here). For brew:

brew install kind
# Create cluster
kind create cluster
# Verify that everything is running:
kubectl get -A pods --context kind-kind

We’re going to develop our controller with Golang so make sure to install it:

brew install go

Kubebuilder is a framework for building Kubernetes Controllers. It provides a built-in CRDs generation and testing environment, generating for you the basic general code for defining your CRD’s API and for building and registering your controller to the manager.

brew install kubebuilder

Developing an Operator

Setting up a new project

Let's first create a new GitHub repository. This command will create a public repository under your configured GitHub user and will clone it to your current directory.

gh repo create abstract-workload -c --public

We can now create our Kubebuilder project:

kubebuilder init --domain itamar.marom --repo itamar.marom/abstractworkload

Take a look at the generated project. Kubebuilder generated an entire project including Makefile, Dockerfile, README, and a whole lot of directories and files. We don’t need to understand it now so let's move on.

We want to create a new API for our controller — the AbstractWorkload :

kubebuilder create api --group examples --version v1alpha1 --kind AbstractWorkload

Answer y for creating a resource and for creating a controller. You can see that two important things happened:

• api/v1alpha1/abstractworkload_types.go — this file holds a Golang representation of our Kubernetes new CRD.
• controllers/abstractworkload_controller.go — this file holds the core logic of our operator.

Designing our API

We will start with defining our API since the logic of the controller will interact with it. Please read about the spec and status section on the Kubernetes community API conventions page to understand what you should put in those fields.

Because we are building a simple abstraction on top of existing resources, we can take the mandatory specs from those objects. You can look at the Kubernetes API reference for examples.

The fields I chose for the spec API are:

replicas # Number of pods
containerImage # The image of the container the pod will hold
workloadType # Whether this deployment is stateless or stateful

All other fields required by the Kubernetes workloads are going to be filed by generated values.

The fields I chose for the status API are:

workload # A reference to the created workload

Building the API

As I said earlier, our API is described in the api/v1alpha1/abstractworkload_types.go file. You can see that Kubebuilder already generated our struct:

//+kubebuilder:resource:shortName="aw"
//+kubebuilder:printcolumn:name="Replicas",type=string,JSONPath=`.spec.replicas`
//+kubebuilder:printcolumn:name="WorkloadType",type=string,JSONPath=`.spec.workloadType`
//+kubebuilder:printcolumn:name="WorkloadKind",type=string,JSONPath=`.status.workload.kind`
type AbstractWorkload struct {
   metav1.TypeMeta   `json:",inline"`
   metav1.ObjectMeta `json:"metadata,omitempty"`

   Spec   AbstractWorkloadSpec   `json:"spec,omitempty"`
   Status AbstractWorkloadStatus `json:"status,omitempty"`
}

I’ve added some annotations on top of the structure, these specific annotations will:

• create a shortcut name for the CRD: aw
• add some objects to the CRD’s printer columns

Leaving us to only declare our API under the AbstractWorkloadSpec and the AbstractWorkloadStatus structs.

Since we already know what our API looks like, let's fill those objects:

// AbstractWorkloadSpec defines the desired state of AbstractWorkload
type AbstractWorkloadSpec struct {
   // +required
   Replicas *int32 `json:"replicas"`
   // +required
   ContainerImage string `json:"containerImage"`
   // +required
   WorkloadType WorkloadType `json:"workloadType"`
}

// AbstractWorkloadStatus defines the observed state of AbstractWorkload
type AbstractWorkloadStatus struct {
   Workload CrossNamespaceObjectReference `json:"workload"`
}

Notice that I’m using some custom types and structs that I’ve built, they are in the full code on Github.

OK, so after defining our API, let's generate our CRDs! Kubebuilder has a great Makefile, you should read it and look at the different commands. This time we will use the manifests command:

make manifests

After running this command, our CRD file is created under the path: config/crd/bases/examples.itamar.marom_abstractworkloads.yaml take a look at its spec and status, and see how our Go structs definitions are implemented in this CRD.

Adding Controller Logic

We have our API ready, we can now develop our controller and we’ll do it through our generated file: controllers/abstractworkload_controller.go

This file includes:

• AbstractWorkloadReconciler — a struct that is responsible for reconciling the AbstractWorkload object.
• Reconcile function — this function will include our core reconciliation logic.
• SetupWithManager function — responsible for registering this controller to the manager. We will cover it later.

Let’s focus on our Reconcile function. This function gets the context and a request that holds a name of an object that we need to reconcile.

The name we’re getting from the request is a name of an object stored in Etcd. That means that It can be an old/existing object. The first thing we will try to do is to get this object, if it does not exist, we will release this req since we can’t do anything in the reconciliation run. After some logging additions our function will look like this:

func (r *AbstractWorkloadReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
   // Add a uuid for each reconciliation
   log := log.FromContext(ctx).WithValues("reconcileID", uuid.NewUUID())

   // Add the controller logger to the context
   ctx = ctrl.LoggerInto(ctx, log)

   obj := &examplesv1alpha1.AbstractWorkload{}
   if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
      log.Error(err, "unable to fetch AbstractWorkload")
      return ctrl.Result{}, client.IgnoreNotFound(err)
   }

   return ctrl.Result{}, nil
}

Our next step is to understand what we need to create — Deployment / StatefulSet. Well we can know that by our spec:

var workloadKind string
var workloadAPIVersion string
labels := map[string]string{
   "name": req.NamespacedName.Name,
   "type": obj.Spec.WorkloadType.String(),
}
switch obj.Spec.WorkloadType.String() {
case examplesv1alpha1.StrStateless:
   log.Info("Stateless application, deploying Deployment")
case examplesv1alpha1.StrStateful:
   log.Info("Stateful application, deploying StatefulSet")
}

We also want to save the Kind and APIVersion of the created object so we can update out AbstractWorkload status.
String() is also a function I implemented myself. Again, everything is in the full code!

In both cases, we will try to get the current object, if it doesn’t exist, we will create it, if it exists — we will update its spec. By telling the Kubernetes client to create this object, it will be stored in the Etcd, then the relevant controller will take care of updating the created workload status.

In real life, we will split the reconcile code into functions, but for readability and simplicity for you, it will all be in the main Reconcile function.

Deployment

log.Info("Stateless application, creating Deployment")
deployment := &v1.Deployment{}
if err := r.Get(ctx, req.NamespacedName, deployment); err != nil {
   deployment = &v1.Deployment{
      ObjectMeta: metav1.ObjectMeta{
         Name:      req.NamespacedName.Name,
         Namespace: req.NamespacedName.Namespace,
         Labels:    labels,
      },
      Spec: v1.DeploymentSpec{
         Replicas: obj.Spec.Replicas,
         Selector: &metav1.LabelSelector{MatchLabels: labels},
         Template: v12.PodTemplateSpec{
            ObjectMeta: metav1.ObjectMeta{Labels: labels},
            Spec: v12.PodSpec{
               Containers: []v12.Container{{
                  Name:  req.NamespacedName.Name,
                  Image: obj.Spec.ContainerImage,
               }},
            },
         },
      },
   }

   if err := r.Client.Create(ctx, deployment); err != nil {
      log.Error(err, "Creating Deployment object")
      return ctrl.Result{}, err
   }
} else {
   deployment.Spec.Replicas = obj.Spec.Replicas
   deployment.Spec.Template.Spec.Containers = []v12.Container{{
      Name:  req.NamespacedName.Name,
      Image: obj.Spec.ContainerImage,
   }}

   if err := r.Client.Update(ctx, deployment); err != nil {
      log.Error(err, "Updating Deployment object")
      return ctrl.Result{}, err
   }
}

workloadKind = deployment.Kind
workloadAPIVersion = deployment.APIVersion

StatefulSet

log.Info("Stateful application, deploying StatefulSet")
statefulSet := &v1.StatefulSet{}
if err := r.Get(ctx, req.NamespacedName, statefulSet); err != nil {
   statefulSet = &v1.StatefulSet{
      ObjectMeta: metav1.ObjectMeta{
         Name:      req.NamespacedName.Name,
         Namespace: req.NamespacedName.Namespace,
         Labels:    labels,
      },
      Spec: v1.StatefulSetSpec{
         Replicas: obj.Spec.Replicas,
         Selector: &metav1.LabelSelector{MatchLabels: labels},
         Template: v12.PodTemplateSpec{
            ObjectMeta: metav1.ObjectMeta{Labels: labels},
            Spec: v12.PodSpec{
               Containers: []v12.Container{{
                  Name:  req.NamespacedName.Name,
                  Image: obj.Spec.ContainerImage,
               }},
            },
         },
         VolumeClaimTemplates: []v12.PersistentVolumeClaim{{
            ObjectMeta: metav1.ObjectMeta{Name: req.NamespacedName.Name},
            Spec: v12.PersistentVolumeClaimSpec{
               AccessModes: []v12.PersistentVolumeAccessMode{v12.ReadWriteOnce},
               Resources: v12.ResourceRequirements{
                  Requests: map[v12.ResourceName]resource.Quantity{
                     v12.ResourceStorage: *resource.NewQuantity(1, resource.BinarySI),
                  },
               },
               StorageClassName: nil,
            },
         }},
      },
   }

   if err := r.Client.Create(ctx, statefulSet); err != nil {
      log.Error(err, "Creating StatefulSet object")
      return ctrl.Result{}, err
   }
} else {
   statefulSet.Spec.Replicas = obj.Spec.Replicas
   statefulSet.Spec.Template.Spec.Containers = []v12.Container{{
      Name:  req.NamespacedName.Name,
      Image: obj.Spec.ContainerImage,
   }}

   if err := r.Client.Update(ctx, statefulSet); err != nil {
      log.Error(err, "Updating StatefulSet object")
      return ctrl.Result{}, err
   }
}

workloadKind = statefulSet.Kind
workloadAPIVersion = statefulSet.APIVersion

Great, The last thing we need to do is update our AbstractWorkload status:

obj.Status.Workload = examplesv1alpha1.CrossNamespaceObjectReference{
   APIVersion: workloadAPIVersion,
   Kind:       workloadKind,
   Name:       req.NamespacedName.Name,
   Namespace:  req.NamespacedName.Namespace,
}

if err := r.Client.Status().Update(ctx, obj); err != nil {
   log.Error(err, "Updating workload status")
   return ctrl.Result{}, err
}

return ctrl.Result{}, nil

RBAC

From writing this code, we understand that this controller will need access to other resources (Deployment and StatefulSet). To give this controller the right permissions, we need to add some Kubebuilder RBAC markers on top of the Reconcile() function. There are already some RBAC markers related to our CRD, after change it should look like this:

//+kubebuilder:rbac:groups=examples.itamar.marom,resources=abstractworkloads,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=examples.itamar.marom,resources=abstractworkloads/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=examples.itamar.marom,resources=abstractworkloads/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete

Owner Reference

What will happen then we delete an AbstractWorkload? I’ll answer that for you — It will be deleted from the Etcd just like we expect. But, it won’t delete the Workload we created! We need to take care of that. Sure, we can add a deletion logic, but Kubernetes gives us a great feature for that use case.

OwnerReference is a Kubernetes mechanism for creating relations between objects. It is used by a lot of Kubernetes objects like Deployment, ReplicaSet, CronJob and more. Setting an Owner reference to an object will cause this object to be dependent on its owner’s existence.

If you have a more complicated use case that includes more actions other than object deletion, read about Finalizers.

Let’s set this for our created objects. Change the object’s meta inside each workload:

ObjectMeta: metav1.ObjectMeta{
   Name:      req.NamespacedName.Name,
   Namespace: req.NamespacedName.Namespace,
   Labels:    labels,
   OwnerReferences: []metav1.OwnerReference{{
      APIVersion: obj.APIVersion,
      Kind:       obj.Kind,
      Name:       obj.Name,
      UID:        obj.UID,
   }},
},

We will also add to our SteupWithManager function an Owns configuration:

func (r *AbstractWorkloadReconciler) SetupWithManager(mgr ctrl.Manager) error {
   return ctrl.NewControllerManagedBy(mgr).
      For(&examplesv1alpha1.AbstractWorkload{}).
      Owns(&v1.Deployment{}).
      Owns(&v1.StatefulSet{}).
      Complete(r)
}

Watching Other Resources

Currently, when deleting a Deployment/StatefulSet manually from the cluster, AbstractWorkload won’t create it again since it’s not asking for their state. We need to add a watching mechanism to those objects.

Kubebuilder has a solution for that — the Watch() function that gets three properties:

• The object to watch for
• A function that converts a Deployment/StatefulSet event into an AbstractWorkload request.
• A list of options for watching those objects (We don’t need that in our use case)

Let’s start with the conversion functions. It can be the same function for both Deployment and StatefulSet:

func (r *AbstractWorkloadReconciler) getAWForChildObject(workload client.Object) []reconcile.Request {
   requests := []reconcile.Request{{
      NamespacedName: types.NamespacedName{
         Name:      workload.GetName(),
         Namespace: workload.GetNamespace(),
      }}}
   return requests
}

Now we can add the Watch() function to the SetupWithManager function:

func (r *AbstractWorkloadReconciler) SetupWithManager(mgr ctrl.Manager) error {
   return ctrl.NewControllerManagedBy(mgr).
      For(&examplesv1alpha1.AbstractWorkload{}).
      Owns(&v1.Deployment{}).
      Owns(&v1.StatefulSet{}).
      Watches(
         &source.Kind{Type: &v1.Deployment{}},
         handler.EnqueueRequestsFromMapFunc(r.getAWForChildObject)).
      Watches(
         &source.Kind{Type: &v1.StatefulSet{}},
         handler.EnqueueRequestsFromMapFunc(r.getAWForChildObject)).
      Complete(r)
}

The Main program and the Manager

Every operator has one manager. The manager's goal is to wrap the controllers with the general logic of integrating with Kubernetes components. It's the component that is responsible for calling the controllersReconcile function with the relevant object requests.

“Every journey needs a start, every program needs a main” — Kubebuilder

The main.go is the operator's entry point. It includes the initial setup requirements such as:

• Registering object (like our CRD) to the operator's scheme
• Initialize the Manager
• Register the Controllers with the managers
• Adding health checks and metrics

Running The Operator Locally

Let’s try to integrate our operator with our Kind Kubernetes cluster. We’ll make sure our kubectl context is correct, and then use another Makefile command to deploy our CRDs:

# Check context
$ kubectl config current-context
kind-kind
# Deploy our CRD
$ make install
# See the deployed CRD
$ kubectl get crd abstractworkloads.examples.itamar.marom

We can now run the controller, and do it in a new terminal:

make run

We can now see how the controller behaves in our Kubernetes cluster. We will create two objects — stateless and stateful:

Check stateless

# examples/stateless.yaml
apiVersion: examples.itamar.marom/v1alpha1
kind: AbstractWorkload
metadata:
  name: test-stateless
spec:
  containerImage: "nginx:latest"
  replicas: 2
  workloadType: stateless

Run the following:

kubectl apply -f examples/stateless.yaml
# aw is the shortcut we created with Kubebuilder mark
kubectl get aw test-stateless
kubectl get deploy test-stateless -w

It might take some time for the actual workload to be created due to pulling the container image.

Delete the deployment and see if it is created again (It should be super quick):

kubectl delete deploy test-stateless
kubectl get deploy -w

Check stateful

# examples/stateful.yaml
apiVersion: examples.itamar.marom/v1alpha1
kind: AbstractWorkload
metadata:
  name: test-stateful
spec:
  containerImage: "nginx:latest"
  replicas: 1
  workloadType: stateful

Run the following:

kubectl apply -f examples/stateful.yaml
# aw is the shortcut we created with Kubebuilder mark
kubectl get aw test-stateful
kubectl get sts test-stateful -w

I hope it works, if not you can ping me or try the source code.

Clean testing YAML

To clean the environment we’ll need to delete our AbstractWorkload objects:

kubectl delete -f examples/stateless.yaml
kubectl delete -f examples/stateful.yaml

You can check that the Deployment/StatefulSet is getting deleted due to OwnerReference.

Clean your cluster environment

Delete the CRD from your cluster with the Kubebuilder Makefile command:

make uninstall

Summary

We had a lot of fun, but now it's time to say goodbye.

In this part, we transformed our concepts learned in part 1 into existence. We developed our Kubernetes operator and tried it against our local Kubernetes cluster.

The AbstractWorkload controller might be not a great idea, but the concept of abstraction is. In the end, we want our developers to work only on their business logic and forget about the infra.

In the next parts, I will help you to set up a local integration testing environment with the Kubebuilder framework, and deploy your controller to run on your Kubernetes cluster.