Programming a Kubernetes Extension API Server — Part 1

Introduction:

In some situations you may want to use a Custom Resource Definition to add new resources to your system or add new functionalities to kubernetes to better fit your needs through custom controllers. The problem with CRDs is that they can be limited in some situations like limited Validation, no graceful deletion, limited number of Subresources and significant performance cost to the Kubernetes API server.

Prerequisites:

  • Basic Kubernetes knowledge.
  • Basic Docker knowledge.
  • Basic Golang knowledge.
  • Basic Minikube knowledge.

Kubernetes API concepts:

Before we start to write our code, it will be useful to define some concepts.

  • Client Sets: Client sets give the programmer access to multiple API groups and resources. With a ClientSet for a specific API group you will be able to manage resources in your cluster, like deployments, pods and whatever resource you may think just via code.
  • Listers: Listers are objects that allow the programmer to get and list a set of resources registered inside the cluster. They can be viewed as the programmer’s way of calling “kubectl list” and “kubectl get”.
  • Informers: As the name suggests, informers are objects that can inform about the lifecycle of a resource. With informers you can execute functions when a resource is created, updated and deleted. They are very useful when writing custom Controllers.
  • Codegen: When adding custom resources to Kubernetes, a lot of work may have to be done by the programmer, like creating all the listers, informers, client sets and the functions to set default values to the resource fields and conversions between versions. Since all of those objects share a big chunk of code for all the different types inside our packages, writing those objects will be a very easy task with the help of templates, but since this isn’t a feature of the golang, codegen is generally used in its place. Instead of manually writing the boilerplate common code, we are going to tell a software how the code should be written and let the software write that boilerplate code for us.

API Versioning and Resources

A given API group may have different versions like v1beta1, v1alpha1, v1, v2 … etc. All of those versions may add new resources to the API and handle data in a different way. The problem with this is that to handle those resources in different versions, the programmer would have to write the same business logic but for all the same types inside different versions. To avoid that burden, the API resources are converted between each other, so if you have a piece of code that runs over a specific version, it can be executed for other versions using the resources conversion.

Defining the resources:

Let’s now think about what group and resources we want to add to the kubernetes API. Because this is just for educational purposes, we are not going to do anything fancy. We will add two resources of Kind Foo and Bar respectively on the baz group. An example of the yaml manifest files that we will use to create our resources inside the cluster will look something like this:

Manifest for the Bar kind example
Manifest for the Foo kind example

Introduction to Codegen

Before we start writing the code, we’re going to discuss a bit about the codegen needed to write the custom server.

DeepCopyObject example:

The DeepCopyObject() method does nothing more than calling the generated DeepCopy method. The signature of the latter varies from type to type (DeepCopy() *T depends on T).

  • +genclient:nonNamespaced — all verb functions are generated without namespace.
  • +genclient:onlyVerbs=create,get — only listed verb functions will be generated.
  • +genclient:skipVerbs=watch — all default client verb functions will be generated except watch verb.
  • +genclient:noStatus — skip generation of updateStatus verb even though the Status field exists.
  • +k8s:deepcopy-gen:interfaces — tag can and should also be used in cases where you define API types that have fields of some interface type. Then “+k8s:deepcopy-gen:interfaces=example.com/pkg/apis/example.SomeInterface” will lead to the generation of a “DeepCopySomeInterface() SomeInterface” method. This allows it to deepcopy those fields in a type-correct way.
  • +groupName=example.com defines the fully qualified API group name. If you get that wrong, client-gen will produce wrong code. Be warned that this tag must be in the comment block just above the package name.

Coding

Setup:

Let’s define the golang project. I’m going to create a package that will be stored on my github account, you can define your package as you want.

The first packages:

We now have to define our package folder structure, the more common way of defining the api types is inside a pkg/apis folder. We are going to start with a folder structure that looks like this:

Configuring the Codegen:

Let’s define a dockerfile and a script that we can use to call the codegen.

  • zz_generated.deepcopy.go
  • zz_generated.defaults.go
https://gist.github.com/Marcos30004347/1545274099aeaaa3890702ad1ad8c1e8

Defaulting:

You may want to define default values for some of the resources fields, if this is the case, inside the pkg/apis/baz/v1alpha1 folder, or inside the folder of whatever external version you want to define the default values, create a file called defaults.go and place the following content:

Last remarks

If you want to take a look at the complete code of this series, it can be found on the following github repository: