Intro to Kubernetes API Extensions
In this blog, we will discuss two ways of extending Kubernetes API; Custom Resource Definition and API Aggregation Layer
Operators
Operators are one of the 7 ways to extend Kubernetes which ranges from having a minimal scope of running a simple script to managing multiple clusters from the same console.
Their base concept comes from Controllers. Controllers are control loops that can watch the state of resources on the cluster and perform required operations when needed. Kubernetes natively run many controllers, one example could be a controller watching Deployments, whose job is to spin up separate ReplicaSet resource whenever there’s an update to important parts of Deployments. Then ReplicaSet controller would create Pods according to the specifications in which your application will run as Containers. All these native Controllers are bundled up as one called Kube Controller Manager.
Reloader can be considered a good example of Controllers, as it watches for changes in secrets and config maps, to do rolling upgrades to the pods wherever needed.
While Controllers work on native k8s resources, Operators use the same control flow but are more focused on Custom Resources (CR) which are created under a specific schema using Custom Resource Definitions (CRD). CRs are user defined resources to tackle a specific challenge that is outside the scope of native resources.
Cert Manager is a good example, focusing on managing certificates for the cluster and user applications.
Operator Framework can be a good start for people new to the field, which provides proper guides on how to start and mentions considerations while writing an operator. It uses Kubebuilder underneath to scaffold operators using best practices.
API Aggregation Layer
There is another way to extend Kubernetes API, which is by creating an API Aggregation Layer. This works by the combination of extension api-server and custom controllers. A resource called APIService is created, which is used to redirect the request to your extension api-server whenever the registered API path is hit.
The main differences in API Aggregation Layer and CRDs are that, CRDs are easier to handle and doesn't require an additional service while Aggregation Layer being programmatically expensive has more control over the resources plus the burden of running extension-apiserver. Both, however, have controllers running underneath.
One good example of Aggregation Layer is how RedHat OpenShift handles Project, which is a “shadow” over Namespaces providing better fine-grained control which doesn't break in multi-tenant conditions implemented over RBAC.
Another is the Metrics Server, which has the responsibility of collecting resource metrics and exposing it over the Metrics API for better resource management.
Example-APIServer can be a good start for understanding and building extension-apiserver.
APIServer-Builder has more examples to take inspiration from and understanding how things fall in place for API Aggregation to work.
Selection between the two can be made considering the Pros/Cons of both approaches and your use-case. Official Kubernetes docs provide a good comparison between these two extension strategies.
Conclusion
While both approaches have their differences, no one’s a clear winner here and it’d depend on your use case to select which way to go forward. Operator pattern, however, is more commonly accepted and implemented in the community because of its ease of use and development.
References
- https://kubernetes.io/docs/concepts/extend-kubernetes/
- https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
- https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
- https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/
- https://sdk.operatorframework.io/
- https://github.com/openshift/kube-projects/
