Build Kubernetes Operator using Kubebuilder

Introduction

Published in

Kubernetes

7 min readDec 25, 2019

Building Kubernetes API involves writing a lot of boiler plate codes and CRDs. There are many ways to implement the Operators for kubernetes, Kubebuilder framework is one of such tool to easily create required boiler plate codes and necessary CRDs. Other tools include KUDO, Operator Framework,and Metacontroller. This article provides a walk through of a sample operator implementation using Kubebuilder framework.

Operator

Operator is a type of controller that implements a specific operational logic to manage a group of resources. Operator makes use of the control loop to manage the resources’ state. The control loop logic brings the state of the target resource to from the current state to the desired state.

CRD

CRD — Custom Resource Definition — defines a Custom Resource which is not available in the default Kubernetes implementation.

E.g.

---
#Defines the CRD 'Foo'apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: foos.samplecontroller.k8s.io
spec:
  group: samplecontroller.k8s.io
  version: v1alpha1
  names:
    kind: Foo
    plural: foos
  scope: Namespaced---
# Foo KindapiVersion: samplecontroller.k8s.io/v1alpha1
kind: Foo
metadata:
  name: example-foo
spec:
  deploymentName: example-foo
  replicas: 1

The CRD alone doesn’t do anything, a controller/operator needs to be implemented to create and manage the resources for the CRD (‘Foo’ in the example above).

Kubebuilder

Kubebuilder is a framework for building Kubernetes APIs / Operators, which helps to generate a set of boiler plate codes for the Controller, and related CRDs.

Kubernetes API Terminology

Groups and Version —An API Group in Kubernetes is a collection of related functionality. Each group has one or more versions, which allow to change how an API works over time.

e.g. When we define “apiVersion: webapp.demo.my-watchlist.io/v1” in a CRD yaml file, the ‘webapp.demo.my-watchlist.io’ part indicates the group name, and ‘v1’ indicates the version.

Kind and Resources — Each API group-version contains one or more kubernetes API types, which is called as a ‘Kind’.

A Kind consists of Metadata + Spec + Status + List. Typically the spec contains the desired state and the status contains the observed state.

Example:

apiVersion: webapp.demo.my-watchlist.io/v1
kind: MyWatchlist
metadata:
  creationTimestamp: "2019-12-24T20:53:08Z"
  generation: 1
  name: mywatchlist-samplename
  space: default
  resourceVersion: "194152"
  uid: e9d8ac53-c956-4c98-99ec-9c2e367875f8
spec:
  frontend:
    replicas: 1
    resources:
      requests:
        cpu: 55m
    servingPort: 8080
  redisName: redis-sample
status:
  url: ""

MyWatchlist Operator Demo

The goal of this demo is to deploy the my-watchlist-go-redis-demo using operators and CRDs. A high level view of the deployment is depicted in the diagram below.

Setup

Install the kubebuilder as per the instructions
Access to a kubernetes 1.16+ cluster — e.g. kind cluster

Init (go mod, kubebuilder)

Use the following commands to scaffold the basic project structure.

$ go mod init my-watchlist.io# domain will be used as suffix to the API groups, should be unique. 
$ kubebuilder init --domain demo.my-watchlist.io

The above command will generate the main.go file and a set of scaffold configs under the ‘config’ directory.

Create API types and Controllers

Create a new Kind ‘MyWatchlist’ and its controller using the following command.

$ kubebuilder create api --group webapp --kind MyWatchlist --version v1
Create Resource [y/n]
y
Create Controller [y/n]
y

The intended yaml structure for the ‘MyWatchlist’ deployment is as follows, note that this structure will drive how we want to implement the controller.

/config/samples/webapp_v1_mywatchlist.yamlapiVersion: webapp.demo.my-watchlist.io/v1
kind: MyWatchlist
metadata:
  name: mywatchlist-sample
spec:
  # Add fields here
  redisName: redis-sample # points to the DB to be used
  #Frontend details, we could even specify the container image name, if needed, to give more flexibility
   frontend:
     resources: 
       requests:
        cpu: 55m

Implement the ‘MyWatchlist’ types to match the above desired structure. Note the // +kubebuilder code generation markers are used for automatic CRD generation, and validations (https://book.kubebuilder.io/reference/markers.html).

/api/v1/mywatchlist_types.go// MyWatchlistSpec defines the desired state of MyWatchlist
type MyWatchlistSpec struct {
	Frontend  FrontendSpec `json:"frontend"`
	RedisName string       `json:"redisName,omitempty"`
}

// FrontendSpec speficies the frontend container spec
type FrontendSpec struct {
	// +optional
	Resources corev1.ResourceRequirements `json:"resources"`

	// +optional
	// +kubebuilder:default=8080
	// +kubebuilder:validation:Minimum=0
	ServingPort int32 `json:"servingPort"`

	// +optional
	// +kubebuilder:default=1
	// +kubebuilder:validation:Minimum=0
	Replicas *int32 `json:"replicas,omitempty"`
}

// MyWatchlistStatus defines the observed state of MyWatchlist
type MyWatchlistStatus struct {
	URL string `json:"url"`
}

Since there is a new struc ‘FrontendSpec’ added, use the following to auto generate the required deepcopy methods.

kubebuilder create api --group webapp --kind Frontend --version v1
Create Resource [y/n]
n
Create Controller [y/n]
n

The intended frontend deployment for the MyWatchlist Kind is similar to https://github.com/hmanikkothu/my-watchlist-go-redis-demo/blob/master/watchlist-app-deployment.yaml. So implement the controller logic in the ‘Reconcile’ method to create a ‘Deployment’ kind and ‘Service’ kind in the cluster that matches desired structure.

/controllers/mywatchlist_controller.go

// +kubebuilder:rbac:groups=webapp.demo.my-watchlist.io,resources=mywatchlists,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=webapp.demo.my-watchlist.io,resources=mywatchlists/status,verbs=get;update;patch

// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=list;watch;get;patch;create;update
// +kubebuilder:rbac:groups=core,resources=services,verbs=list;watch;get;patch;create;update

// Reconcile reconciles the request
func (r *MyWatchlistReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
	ctx := context.Background()
	log := r.Log.WithValues("mywatchlist", req.NamespacedName)

	log.Info("reconciling mywatchlist")

Note the +kubebuilder markers, these are required for the operator to be able to update the ‘mywatchlist’ resources. Second set of markers is required for the operator to be able to create the Deployment and Service.

Next step is to retrieve the ‘MyWatchlist’ object, and related ‘Redis’ object. Note that the MyWatchlist has a dependency on redis object, so the redis kind should be deployed for the ‘MyWatchlist’ reconciler to complete successfully.

// Gets the 'MyWatchlist' object
	var watchlist webappv1.MyWatchlist
	if err := r.Get(ctx, req.NamespacedName, &watchlist); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}// Retrieves the redis
	var redis webappv1.Redis
	redisName := client.ObjectKey{Name: watchlist.Spec.RedisName, Namespace: req.Namespace}
	if err := r.Get(ctx, redisName, &redis); err != nil {
		log.Error(err, "didn't get redis")
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	log.Info("got redis", "redis", redis.Name)

The following code creates the ‘watchlist’ frontend Deployment and related Service.

deployment, err := r.createDeployment(watchlist, redis)
	if err != nil {
		return ctrl.Result{}, err
	}
	svc, err := r.createService(watchlist)
	if err != nil {
		return ctrl.Result{}, err
	}

	applyOpts := []client.PatchOption{client.ForceOwnership, client.FieldOwner("watchlist-controller")}

	err = r.Patch(ctx, &deployment, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}

	err = r.Patch(ctx, &svc, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}

Next step is to update the MyWatchlistSatus.URL with the value of the service url.

watchlist.Status.URL = getServiceURL(svc, watchlist.Spec.Frontend.ServingPort)

	err = r.Status().Update(ctx, &watchlist)
	if err != nil {
		return ctrl.Result{}, err
	}

	log.Info("reconciled watchlist")

	return ctrl.Result{}, nil
}

Note the service created in this example is of type ‘LoadBalancer’. The Status.URL will be displayed with the ‘kubectl get mywatchlist’, and the url can be used to access the frontend UI. Some dev clusters (e.g. Kind cluster) does not support the service type ‘LoadBalancer’, so some kind of port forwarding commands can be used to view the UI. e.g. kubectl port-forward svc/mywatchlist-sample 7000:8080

Entire code for the MyWatchlist controller implementation can be found here.

Follow similar approach to implement the ‘Redis’ controller as well — a completed code can be found here.

Generate Manifests

Use the following to generate manifests

$ make manifests

An updated set of manifest for the CRDs will be generated at the folder ‘config/crd/base’ as per the API Types and kubebuilder markers.

Run locally

Now it’s time to apply the CRDs and run controller locally to test.

#Create CRDs for 'MyWatchlist' and 'Redis'
kubectl create -f config/crd/bases/#Create Redis Kind
kubectl create -f config/samples/webapp_v1_redis.yaml#Create MyWatchlist Kind
kubectl create -f config/samples/webapp_v1_mywatchlist.yaml

Use the following command to run and test the controller locally.

$ make run

Now, if everything is fine, the output will show a bunch of messages indicating successful completion of the Reconcile loop.

2019-12-24T20:53:08.140Z INFO controllers.MyWatchlist reconciling mywatchlist {"mywatchlist": "default/mywatchlist-sample"}2019-12-24T20:53:08.140Z INFO controllers.MyWatchlist got redis {"mywatchlist": "default/mywatchlist-sample", "redis": "redis-sample"}time="2019-12-24T20:53:08Z" level=info msg="HK - WR-createDeployment :###: RedisServiceName=redis-sample-watchlist-db\n" source="mywatchlist_controller.go:142"time="2019-12-24T20:53:08Z" level=info msg="urlForService: LoadBalancer.Ingess not set" source="mywatchlist_controller.go:105"2019-12-24T20:53:08.183Z INFO controllers.MyWatchlist reconciled watchlist {"mywatchlist": "default/mywatchlist-sample"}2019-12-24T20:53:08.183Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "mywatchlist", "request": "default/mywatchlist-sample"}

Publish to Docker Repo

Use the following command in order to build a docker image, and push it to the repo

$ make docker-build docker-push IMG=<repo-name>/mywatchlist-operator:v1

RBAC

Notice the autogenerated files in the folder ‘config/rbac’. role.yaml defines the manager-role role with necessary permissions for mywatchlist and redis resources as per the info provided through the kubebuilder markers. Additionally, for this sample it requires permissions to manage deployment and services to deploy the necessary pods and services.

Deploy from Docker Repo

Now that we have the operator available as a docker image, use the following to directly deploy the controller from docker repo.

$ make deploy IMG=<rep-name>/mywatchlist-operator:v1

This would create necessary controller in the namespace provided in the ‘/config/default/kustomization.yaml’, e.g. ‘watchlist-operator-system’

Now, use the following to deploy Redis and MyWatchlist CRDs

$ kubectl create -f config/samples/webapp_v1_redis.yaml 
$ kubectl create -f config/samples/webapp_v1_mywatchlist.yaml

Verify

Verify that the controller is deployed in ‘watchlist-operator-system’ namespace

$ kubectl get all -n watchlist-operator-system

Verify that the pods and services for frontend and Redis are created in the default namespace

$ kubectl get all

Get the URL to test the UI

$ kubectl get mywatchlistNAME                 URL                       DESIREDmywatchlist-sample   http://32.225.23.xxx:8080      1

Or use port-forward option with kubectl

$ kubectl port-forward svc/mywatchlist-sample 7000:8080

Navigate to the frontend UI