Extending Kubernetes: Create Controllers for Core and Custom Resources
Kubernetes is powerful and ships with a lot of out-of-the-box functionality. But as we start to think of new ways to use Kubernetes, we may want to have the ability to create our very own “Kubernetes logic” inside of our cluster. This is where the ability to create controllers and custom resources can help: It allows us to extend Kubernetes.
What is this?
This post can be broken down into sub-topics:
- Controllers overview
- Controller event flow
- Controller with core resources
- Controller with custom resources
- Defining custom resources
- Generating custom resource code
- Wiring up the generated code to the controller
- Creating Custom Resource Definitions
- Running the controller
What is this NOT?
This post is not a discussion about when you should use custom resources (and controllers). This assumes that you are looking for the knowledge on how to create them, whether out of curiosity or a requirement.
For a really good summary of when you should or shouldn’t create custom resources and controllers, please refer to the official Kubernetes documentation on the topic.
Kubernetes has a very “pluggable” way to add your own logic in the form of a controller. A controller is a component that you can develop and run in the context of a Kubernetes cluster.
Controllers are an essential part of Kubernetes. They are the “brains” behind the resources themselves. For instance, a Deployment resource for Kubernetes is tasked with making sure there is a certain amount of pods running. This logic can be found in the deployment controller (GitHub).
You can have a custom controller without a custom resource (e.g. custom logic on native resource types). Conversely, you can have custom resources without a controller, but that is a glorified data store with no custom logic behind it.
Controller event flow
Working backwards (as far as event flow goes), the controller “subscribes” to a queue. The controller worker is going to block on a call to get the next item from the queue.
An event is the combination of an action (create, update, or delete) and a resource key (typically in the format of namespace/name).
Before we talk about how the queue is populated for the controller, it is worth mentioning the idea of an informer. The informer is the “link” to the part of Kubernetes that is tasked with handing out these events, as well as retrieving the resources in the cluster to focus on. Put another way, the informer is the proxy between Kubernetes and your controller (and the queue is the store for it).
Part of the informer’s responsibility is to register event handlers for the three different types of events: Add, update, and delete. It is in those informer’s event handler functions that we add the key to the queue to pass off logic to the controller’s handlers.
See below for an illustration of the event flow…
Controller: Core resources
There are two types of resources that controllers can “watch”: Core resources and custom resources. Core resources are what Kubernetes ship with (for instance: Pods).
To work with core resources, when you define your informer you specify a few components…
- ListWatch — the
WatchFuncshould be referencing native APIs to list and watch core resources
- Controller handlers — the controller should take into account the type of resource that it expects to work with
In the case of the example, this informer (GitHub) is defined to list and watch pods…
This could just as easily been programmed to work with deployments, daemon sets, or any other core resource that ships with Kubernetes.
For a more detailed look into how a controller for core resources would work, please refer to the GitHub repo showing an example of this. A few things to note, I specifically wrote this code to be read as easily as possible. This includes everything in a single package, as well as extremely verbosely commented code. So hopefully it reads like a book! The significant source code files are…
- main.go — this is the entry point for the controller as well as where everything is wired up. Start here
- controller.go — the Controller struct and methods, and where all of the work is done as far as the controller loop is concerned
- handler.go — the sample handler that the controller uses to take action on triggered events
Controller: Custom resources
Handling core resource events is interesting, and a great way to understand the basic mechanisms of controllers, informers, and queues. But the use-cases are limited. The real power and flexibility with controllers is when you can start working with custom resources.
You can think of custom resources as the data, and controllers as the logic behind the data. Working together, they are a significant component to extending Kubernetes.
The base components of our controller will remain mostly the same as when working with core resources: We will still have an informer, a queue, and the controller itself. But now we need to define the actual custom resource and inject that into the informer.
Define custom resource
When developing a custom resource (and controller) you will undoubtedly already a requirement. The first step in defining the custom resource is to figure out the following…
- The API group name — in my case I’ll use
trstringer.combut this can be whatever you want
- The version — I’ll use “v1” for this custom resource but you are welcome to use any that you like. For some ideas of existing API versions in your existing Kubernetes cluster you can run
kubectl api-versions. Some common ones are “v1”, “v1beta2”, “v2alpha1”
- Resource name — how your resource will be individually identified. For my example I’ll use the resource name
Before we create the resource and necessary items, let’s first create the directory structure:
$ mkdir -p pkg/apis/myresource/v1.
Create the group name const in a new file:
$ touch pkg/apis/myresource/register.go…
Our new package has the same name as the resource and defines the group name for future reference.
Create the resource structs:
$ touch pkg/apis/myresource/v1/types.go…
For all intents and purposes, this is the data structure of our custom resource. We include Kubernetes resource components like metadata, but this is our skeleton resource that we expect to use.
Hopefully the comments explain most things here, but you’ll also see a few comments in the format of
// +<tag_name>[=value]. These are “indicators” for the code generator (usage of the generator is explained with a walk-through below) that direct specific behavior for code generation…
- +genclient — generate a client (see below) for this package
- +genclient:noStatus — when generating the client, there is no status stored for the package
- +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object — generate deepcopy logic (required) implementing the runtime.Object interface (this is for both
Create a doc source file for the package:
$ touch pkg/apis/myresource/v1/doc.go…
types.go, we have a couple of comment tags for the code generator. When defined in
doc.go for the package, these settings take effect for the whole package. Here we set deepcopy should be generated for all types in the package (unless otherwise turned off). And we tell the generator what the API group name is with the
The client requires a particular API surface area for custom types, and the package needs to include
Resource. These functions handle adding types to the schemes. Create the source file for this functionality in the package:
$ touch pkg/apis/myresource/v1/register.go…
At this point we should have all of the boilerplate to run the code generator to do a lot of the heavy lifting to create the client, informer, and lister code (as well as the deepcopy functionality that is required).
Run the code generator
There is a little bit of setup to run the code generator. I’ve included the shell commands below that you need to run. It’s the
k8s.io/code-generator package that contains the
generate-groups.sh shell script which we will use to do all of the heavy lifting (this shell script directly invokes the
After running the code generator we now have generated code that handles a large array of functionality for our new resource. Now we need to tie a lot of loose ends together for our new resource.
Wire up the generated code
There are a couple of changes we need to make. First, in our helper function that gets the Kubernetes client, we need to now also return a an instance of a configured client that can interact with
We also need to now store the custom resource client, and we can utilize the generated helper function to return an informer tailored to the custom resource…
Custom Resource Definition
Now that we’ve created the custom logic part of the custom resource (through the controller), we need to actually create the data part of our custom resource: the Custom Resource Definition.
I put my CRD in a separate dir at the root of the repo:
$ mkdir crd. And then I create my definition:
$ touch crd/myresource.yaml…
This should appear straightforward, as we’re using this CRD to define the API group, version, and name of the custom resource.
Create the CRD in your cluster by running
$ kubectl apply -f crd/myresource.yaml.
The full code for this example can be found on this repo (GitHub).
Running the controller
To run the controller, in the root of the repo run
$ go run *.go. And then in a separate shell, create an object that is of type
MyResource. I did this by creating an example configuration in my root repo:
$ mkdir example && touch example/example-myresource.yaml…
And then I created this in my cluster:
$ kubectl apply -f example/example-myresource.yaml and the output from my controller logging shows that my custom controller did indeed pick up this create event for this resource (and could have handled it however it needed to be handled)…
Kubernetes is an exciting platform, and one of the really great features of it is the ability to extend it. The sky is the limit, and hopefully with this additional knowledge it’ll be easier to understand how controllers work and how to create your own.
- https://kubernetes.io/docs/concepts/api-extension/custom-resources/ (official Kubernetes documentation on custom resources)
- https://github.com/kubernetes/sample-controller (example controller)
- https://github.com/kubernetes/code-generator (code-generator source repo)
- https://blog.openshift.com/kubernetes-deep-dive-code-generation-customresources/ (great description from the OpenShift blog on how to use the code-generator and explanation of the tagging system <client|informer|lister>-gen expects)
- https://engineering.bitnami.com/articles/kubewatch-an-example-of-kubernetes-custom-controller.html (overview and implementation of core resource watching controller from Bitnami)
- https://github.com/trstringer/k8s-controller-core-resource (sample of custom controller with a core resource)
- https://github.com/trstringer/k8s-controller-custom-resource (sample of custom controller with a custom resource)