Kubernetes Operator Demystified

Gayatri S Ajith
Aug 6 · 8 min read
The mystery behind Kubernetes Operators!

Let’s get one thing off the plate — Kubernetes Operator is not an object. It's a design pattern. With that, let's understand what is Kubernetes Operator.

Operator Pattern

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

So, what does this mean? It means a Kubernetes operator is a design pattern that combines custom resources & custom controllers.

Again, what does this mean?

Custom Resource + Custom Controller = Kubernetes Operator
Custom Resource + Custom Controller = Kubernetes Operator
Kubernetes Operator is a design pattern.

Custom Resource

In the same way, a custom resource is a custom endpoint in the Kubernetes API that isn’t a part of the default installation. Once you create a custom resource definition (CRD) in the cluster, you can perform all the basic REST API calls — GET, PUT, POST, DELETE, PATCH and so on — using the custom endpoint created to manage the data of the custom resource objects.

Controller or Control Loop

Custom Resource Definition will just create a custom endpoint to blindly store and retrieve data related to the custom object. But, what should this custom resource do? For example, ReplicaSet resource creates pods, Pod resource creates containers — what will your resource do? This is where custom controllers come into the picture.

A controller is a control loop/non-terminating process that is constantly checking the state of at least one Kubernetes object (built-in or custom). This is the script that is notified when an object of a particular resource type is added, deleted or modified. You can then decide what you want to do with the event — create containers, call external URLs, whatever you fancy.

To create a control loop script all you need is a way to detect changes happening in the cluster — like, a new object of a particular type added, deleted or modified. Kubernetes has a built-in feature for this called Watch.

Kubernetes API Watch

To enable clients to build a model of the current state of a cluster, all Kubernetes object resource types are required to support consistent lists and an incremental change notification feed called a watch. Every Kubernetes object has a resourceVersion field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a resourceVersion value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied resourceVersion. This allows a client to fetch the current state and then watch for changes without missing any updates. If the client watch is disconnected they can restart a new watch from the last returned resourceVersion, or perform a new collection request and begin again.

Let’s try to understand this with an example.

This is a sample API call to GET the list of Pods in the default namespace.
The response returns 3 Pods. Notice the resourceVersion in the metadata of each Pod. resourceVersion is like an ID for the object in the database.

curl --insecure -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/default/pods
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"resourceVersion": "1291"
},
"items": [
{
"metadata": {
"name": "test-ccdc9b768-nxtcw",
"namespace": "default",
"resourceVersion": "918",
...
},
"spec": {
...
},
"status": {
...
}
},
{
"metadata": {
"name": "test-ccdc9b768-vwsxn",
"namespace": "default",
"resourceVersion": "953",
...
},
"spec": {
...
},
"status": {
...
}
},
{
"metadata": {
"name": "test-ccdc9b768-vxhds",
"namespace": "default",
"resourceVersion": "987",
...
},
"spec": {
...
},
"status": {
...
}
}
]
}

You can now initiate a Watch for Pods in the default namespace by sending watch=1 and resourceVersion=<resourceVersion> in the API call. What this will do is,

  • return all the events for the object type Pod in the default namespace after this resourceVersion.
  • keep the connection open (using long-polling)so that you can continue getting the events.

For example, if I created a new Pod named “my-new-test-pod” and then, started a Watch using the latest resourceVersion from the previous call (which is 987 in our example) then I would get a response like this…

curl --insecure -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/default/pods?watch=1&resourceVersion=987{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"my-new-test-pod","resourceVersion":"11046"...},"spec":{...}...}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"resourceVersion":"11047"...},"spec":{...}...}}

Every time an event associated with a Pod in the default namespace occurs, I will get the event type and the object information that has the resourceVersion and the specs. And this connection will not close — so, I will continue getting the list of events. I can now decide what I want to do with these events!

Let’s Understand the Flow with an Example.

These are the steps we will follow:

  1. Create a custom resource
  2. Create a Pod that will run our control loop script
  3. Decide what our script will do with the events

Step 1: Create a Custom Resource

CRD will describe the schema of our new resource (schema means what fields can be used in our new resource, what is the data type of the values, default values, validations etc).

mycrd.yamlapiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: mywebservers.k8sobjects.gsa.com
spec:
group: k8sobjects.gsa.com
scope: Namespaced
names:
kind: MyWebserver
singular: mywebserver
plural: mywebservers
shortNames:
- myweb
- mws
versions:
- name: v2
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
image:
type: string
port:
type: integer
replicas:
type: integer

Above CRD defines the schema for a new object called ‘MyWebserver’. It can be referred to with the names mywebserver, mywebservers, myweb, mws.

kubectl apply -f mycrd.yaml

This custom resource definition will create new endpoints in the Kubernetes API

# Get the list of API groups available. You should see your custom resource listed here
curl --insecure -X GET -H "Authorization: Bearer $TOKEN" https://kubernetes/apis/
{
"kind": "APIGroupList",
"apiVersion": "v1",
"groups": [
...
{
"name": "apps",
"versions": [
{
"groupVersion": "apps/v1",
"version": "v1"
}
],
"preferredVersion": {
"groupVersion": "apps/v1",
"version": "v1"
}
},
...
{
"name": "k8sobjects.gsa.com",
"versions": [
{
"groupVersion": "k8sobjects.gsa.com/v2",
"version": "v2"
}
],
"preferredVersion": {
"groupVersion": "k8sobjects.gsa.com/v2",
"version": "v2"
}
}
]
}
# Get list of mywebserver objects in any namespace
curl --insecure -X GET -H "Authorization: Bearer $TOKEN" https://kubernetes/apis/k8sobjects.gsa.com/v2/mywebservers
{
"apiVersion":"k8sobjects.gsa.com/v2",
"items":[....],
"kind":"MyWebserverList",
"metadata":{"continue":"","resourceVersion":"16800"}
}

So, the YAML to create my new object based on this custom resource definition will look like this

myobj.yamlapiVersion: "k8sobjects.gsa.com/v2"
kind: MyWebserver
metadata:
name: webapp
spec:
image: nginx
port: 80
replicas: 2
kubectl apply -f myobj.yaml

apiVersion is your CRD <group name>/<version name>
the object has a property spec which has the sub-properties image of the type string, port of the type integer and replicas of the type integer.
You can now use kubectl to manage the object.

kubectl get mws
NAME AGE
webapp 67s
kubectl describe mws webapp
Name: webapp
Namespace: default
Labels: <none>
Annotations: <none>
API Version: k8sobjects.gsa.com/v2
Kind: MyWebserver
Metadata:
Creation Timestamp: 2021-08-06T18:57:09Z
Generation: 1
Managed Fields:
API Version: k8sobjects.gsa.com/v2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:image:
f:port:
f:replicas:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-08-06T18:57:09Z
Resource Version: 17033
UID: ce87c872-ad71-4092-a0ec-a10f65a4b132
Spec:
Image: nginx
Port: 80
Replicas: 2
Events: <none>
# List mywebserver objects in the default namespace
curl --insecure -X GET -H "Authorization: Bearer $TOKEN" https://kubernetes/apis/k8sobjects.gsa.com/v2/namespaces/default/mywebservers
{
"apiVersion":"k8sobjects.gsa.com/v2",
"items":[
{
"apiVersion":"k8sobjects.gsa.com/v2",
"kind":"MyWebserver",
"metadata":{
"annotations":{...},
"creationTimestamp":"2021-08-06T18:57:09Z",
"generation":1,
"managedFields":[...],
"name":"webapp",
"namespace":"default",
"resourceVersion":"17033",
"uid":"ce87c872-ad71-4092-a0ec-a10f65a4b132"
},
"spec":{
"image":"nginx",
"port":80,"replicas":2
}
}
],
"kind":"MyWebserverList",
"metadata":{
"continue":"",
"resourceVersion":"17265"
}
}

Step 2: Create a Pod that will run our Control Loop script

Why are we running this Control Loop script inside a Pod?

  • It's easier to grant the necessary permissions to our script using Service Accounts.
  • You can run the pod as a daemonset to ensure that it's always running.

To keep things simple, I am going to use PHP to run our control loop script. Feel free to use any programming language you like. It's a misconception that control loop scripts can be written only in Go, Python or Java!

Here is the Service Account to give our script the necessary permissions. We are permitting ourselves to work with our Custom Resource (MyWebserver), Deployments and Pods in any namespace.

permissions.yaml---
apiVersion: v1
kind: ServiceAccount
metadata:
name: mywebservers-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: mywebservers-cr
rules:
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- list
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ['*']
- apiGroups:
- ""
resources:
- pods
verbs:
- '*'
- apiGroups:
- "k8sobjects.gsa.com"
- "k8sobjects.gsa.com/v2"
resources:
- mywebservers
verbs:
- '*'
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mywebservers-crb
subjects:
- kind: ServiceAccount
name: mywebservers-sa
namespace: default
roleRef:
kind: ClusterRole
name: mywebservers-cr
apiGroup: rbac.authorization.k8s.io

This Deployment will run my control loop script. It is of course in the custom Docker image.

apiVersion: apps/v1
kind: Deployment
metadata:
name: php-client
spec:
replicas: 1
selector:
matchLabels:
app: php-client
template:
metadata:
labels:
app: php-client
spec:
serviceAccountName: mywebservers-sa
containers:
- name: n1
image: gayatrisa/learn-k8s:client
ports:
- containerPort: 80

Step 3: Decide what you want to do inside your script

My control loop script looks like this. (I have hidden the PHP coding complexities of streams etc. A working code is available on GitHub.)

<?php
$myop = new K8sClient();
$resourceVersion = '';
do {
$events = $myop->watchObject($resourceVersion);
if (!empty($events) && is_array($events)) {
$latestResourceVersion = $events[sizeof($events)-1]['object']['metadata']['resourceVersion'];
if ($resourceVersion >= $latestResourceVersion) {
break;
}
$resourceVersion = $latestResourceVersion;
}
foreach ($events as $event) {
switch ($event['type']) {
case 'ADDED':
echo '----- EVENT TYPE: ADDED -----' . "\n";
// Create a deployment and service based on the object data we got from the event
break;
case 'DELETED':
echo '----- EVENT TYPE: DELETED -----' . "\n";
// Delete the deployment and service we created based on the object data we got from the event
break;
}
}

} while (!empty($events));

So for every mywebserver object that is added 1 Deployment and 1 Service object is created by my controller (control loop script). Similarly, for every mywebserver object, you delete the corresponding 1 Deployment and 1 Service object will get deleted by my controller.

Now, you have total power — you can choose to even trigger an external action!

Conclusion

  • A Custom Resource Definition, to create your custom API endpoints
  • A Custom Controller/Control Loop script, that will get the events related to your custom resource
  • A Pod, to run your control loop script
  • A Service Account, which gives your control loop script the necessary permissions

Many wonderful SDKs hide these complexities and steps, making it all smooth and easy for you. One of the most popular ones is OperatorSDK. Once this flow is clear I would recommend using an SDK instead of doing it all yourself.

Nerd For Tech

From Confusion to Clarification

Nerd For Tech

NFT is an Educational Media House. Our mission is to bring the invaluable knowledge and experiences of experts from all over the world to the novice. To know more about us, visit https://www.nerdfortech.org/.

Gayatri S Ajith

Written by

Corporate Trainer for DevOps/Docker/Kubernetes/Ansible/Splunk/Jenkins/Chef/Puppet/AWS

Nerd For Tech

NFT is an Educational Media House. Our mission is to bring the invaluable knowledge and experiences of experts from all over the world to the novice. To know more about us, visit https://www.nerdfortech.org/.