Go Kubernetes Operator

A simple Operator, but not too simple

Published in

Techlogs

6 min readAug 23, 2022

Here is a Go operator based the on Kubebuilder framework; for those who can follow Kubebuilder too simple example and cannot follow the next too complex one.

This is a pretty simple Kubernetes Operator; It reads a Custom CRD and creates a deployment from the Image name present in that YAML

That’s it. (the Image name is from this Go-GRPC Sample project -https://github.com/alexcpn/go_grpc_2022)

No point in duplicating good Tutorials; So follow these two references first

https://book.kubebuilder.io/quick-start.html

The second is bit complex https://book.kubebuilder.io/cronjob-tutorial/cronjob-tutorial.html

Read till Section 1.7 https://book.kubebuilder.io/cronjob-tutorial/controller-implementation.html

All code here- https://github.com/alexcpn/go_operator_2022

The easiest way to understand what a Kubernetes Operator can do is by building one. We will use the Kubebuilder framework to build one in Go language. The other framework for this is operator-sdk. Operator-SDK also uses Kubebuilder in the backend. We will create a simple operator that reads a custom CRD that we create (read a Yaml of a custom type similar to a Deployment yaml); and creates a Deployment out of that via code.

Step 1 : Install Kubebuilder

Follow the make file to install Kubebuilder

make once

Step 2: Init the project

make init_project

Note that we are giving DOMAIN and Project name as below in the make file.

DOMAIN = mytest.io
PROJECT = testoperator

The Init will create a child folder of the name PROJECT and fill with Bolier plate code and files

Step 3 : Create the API

make create_crd

Select y for both options Resources and Controller

cd testoperator && kubebuilder create api --group grpcapp --version v1 --kind Testoperartor && make manifests
Create Resource [y/n]
y
Create Controller [y/n]
y

This will create the CRD and Controller files. Out of the generated files three are important — The Controller, The Spec and the Yaml

testoperator_controller.go
testoperator_types.go
grpcapp_v1_testoperator.yaml

You can see all the generated files here in the two commits in this branch https://github.com/alexcpn/go_operator_2022/compare/master...generated-code

Step 4: Implement the logic

In this simple Operator, we are going to read the CRD testoperator/config/samples/grpcapp_v1_testoperartor.yaml and create a deployment via code.

Step 4.1: Adding custom fields to CRD yaml

For this, the minimum is the Pod Image needed to create a deployment. We will add that to the above file

testoperator/config/samples/grpcapp_v1_testoperartor.yaml

# This is a sample Operator that will create a deployment with the name of the 
# podImage and also create a service with the given port and name
apiVersion: grpcapp.mytest.io/v1
kind: Testoperartor
metadata:
  name: testoperartor-sample
spec:
  # TODO(user): Add fields here
  # 1 ADDED
  podImage: alexcpn/run_server:1.2

Step 4.2: Adding custom types for added fields to the Go code

Before we Apply this to the cluster we need to add the PodImage field to the controller types file /testoperator/api/v1/testoperartor_types.go

// TestoperartorSpec defines the desired state of Testoperartor
type TestoperartorSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file	// Foo is an example field of Testoperartor. Edit testoperartor_types.go to remove/update
	Foo string `json:"foo,omitempty"`	// Let's create a service with this operator
	PodImage string `json:"podImage,omitempty"` //2 ADDED
}

Every time a new field is added, re-run the make file

cd testoperator
make

(The Make file for Kubebuilder is in the child PROJECT folder, generated by Kubebuilder)

Step 4.3: Apply CRD to Cluster

With the above step, we will be able to successfully deploy the Yaml to the cluster with first Make install — which will install all the needed CRD’s and then apply our modified file

cd testoperator
make install
kubectl apply -f ./config/samples/grpcapp_v1_testoperartor.yaml

Ouput

kubectl get testoperartor
NAME                   AGE
testoperartor-sample   46s
kubectl get crds
NAME                               CREATED AT
testoperartors.grpcapp.mytest.io   2022-08-22T11:55:11Z

Note — There is a typo in the operator name above (testoperartor); To keep the commits easy to follow I am not correcting it now.

Step 4.4: Adding reconcile logic — here create a deployment with data in CRD

Add the controller logic in testoperartor_controller.go

We first Get the PodImage name from the deployed Kind (Step 4.2 & 4.3) and add the code to create a Deployment based on the retrieved Image name in the controller Reconcile loop

Since we are creating a deployment we need to add the following Imports

appsv1 "k8s.io/api/apps/v1"                   //ADDED
	corev1 "k8s.io/api/core/v1"                   //ADDED
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" //ADDED

To instruct the Kubebuilder to add RBAC for this operation we add the following too in the Reconcile function comments

//ADDED extra for creating deployment
// generate rbac to get,list, and watch pods // 3 ADDED
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch
// generate rbac to get, list, watch, create, update, patch, and delete deployments
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete

Now add the logic; Basically, first GET the STATE you want; that is in our case a deployment with a particular Pod Image name, and then in reconcile controller method add the logic to REACH that STATE. Basically in our case, create a deployment Note — That particular Deployment code boilerplate was generated by GitHub AI Co-pilot

You can find just the added code here https://github.com/alexcpn/go_operator_2022/pull/1/files

Especially these lines https://github.com/alexcpn/go_operator_2022/pull/1/files#diff-aa914aaeb9f16af8dc5d9cea70c2eb25660b552dc7b2ffe4c726134002d1fe5eR62-R114

Step 4.5 — Running the operator in Place

Test if there are any code errors by ‘makeby deploying to cluster via make run`; which will run the controller in the terminal

cd testoperator
make
make runalex@pop-os:~/coding/app_fw/go_operator/testoperator$ make run
test -s /home/alex/coding/app_fw/go_operator/testoperator/bin/controller-gen || GOBIN=/home/alex/coding/app_fw/go_operator/testoperator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.2
/home/alex/coding/app_fw/go_operator/testoperator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/alex/coding/app_fw/go_operator/testoperator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
1.661231577844857e+09   INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
1.6612315778451552e+09  INFO    setup   starting manager
1.6612315778454158e+09  INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6612315778454406e+09  INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6612315778456118e+09  INFO    Starting EventSource    {"controller": "testoperartor", "controllerGroup": "grpcapp.mytest.io", "controllerKind": "Testoperartor", "source": "kind source: *v1.Testoperartor"}
1.661231577845636e+09   INFO    Starting Controller     {"controller": "testoperartor", "controllerGroup": "grpcapp.mytest.io", "controllerKind": "Testoperartor"}
1.66123157794669e+09    INFO    Starting workers        {"controller": "testoperartor", "controllerGroup": "grpcapp.mytest.io", "controllerKind": "Testoperartor", "worker count": 1}
1.6612315779469857e+09  INFO    Reconciling Test Operator       {"Test Operator": {"kind":"Testoperartor","apiVersion":"grpcapp.mytest.io/v1","metadata":{"name":"testoperartor-sample","namespace":"default","uid":"4c81b1d1-5e0e-42c3-a352-bce980542cd3","resourceVersion":"555018","generation":1,"creationTimestamp":"2022-08-22T12:10:26Z","annotations":..."alexcpn/run_server:1.2"}...
1.6612315779471476e+09  INFO    Pod Image is set        {"PodImageName": "alexcpn/run_server:1.2"}
1.661231577953225e+09   INFO    Created Deployment      {"Deployment.Namespace": "default", "Deployment.Name": "testoperartor-sample-deployment"}

In the cluster

$ kubectl get deployment
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
testoperartor-sample-deployment   1/1     1            1           31s$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
testoperartor-sample-deployment-55645ff5cb-m4rjr   1/1     Running   0          44skubectl get testoperartor
NAME                   AGE
testoperartor-sample   17h$ kubectl get testoperartor -o yaml
apiVersion: v1
items:
- apiVersion: grpcapp.mytest.io/v1
  kind: Testoperartor
  metadata:
    annotations:
                  ....
    creationTimestamp: "2022-08-22T12:10:26Z"
    generation: 1
    name: testoperartor-sample
    namespace: default
    resourceVersion: "555018"
    uid: 4c81b1d1-5e0e-42c3-a352-bce980542cd3
  spec:
    podImage: alexcpn/run_server:1.2
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Step 4.5 -Create a Docker Image for the operator and install it

As per the guideline here https://book.kubebuilder.io/cronjob-tutorial/running.html

cd testoperator
make docker-build docker-push IMG=alexcpn/testoperator:1
make deploy IMG=alexcpn/testoperator:1

Output

You can see that testoperator-controller-manager is configured and running

kubectl get deployment -A
NAMESPACE             NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
kube-system           coredns                           2/2     2            2           5d18h
local-path-storage    local-path-provisioner            1/1     1            1           5d18h
testoperator-system   testoperator-controller-manager   1/1     1            1           2m17skubectl get pods  -n testoperator-system
NAME                                               READY   STATUS    RESTARTS   AGE
testoperator-controller-manager-66c9fcdc58-556w4   2/2     Running   0          4m12s

Let’s now delete the older deployment and apply fresh while watching the values

kubectl apply -f testoperator/config/samples/