Kubernetes is Your Platform: Design Patterns For Extensible Controllers-My KubeCon Notes
Hi everyone,
I want to talk about my KubeCon Europe 2022 notes, Kubernetes is Your Platform: Design Patterns For Extensible Controllers session.
We know that, developing Kubernetes extensions is quickly becoming a mainstream practice to solve problems in a cloud native way.
But, when the business problem is complex, using a single controller quickly becomes a limit that developers are required to cross.
Let we learn that reusable design patterns derived from the concrete experience and the hard lessons learned on the field.
Let’s clarify some terms…
Controller: A single reconcile loop (such as the ReplicaSet controller)
Manager/Operator: A component that runs one or more controllers, like the kube-controller-manager
Custom Resource: an instance of a Kubernetes type defined by using Custom Resource Definitions. A custom resource is formed by metadata, spec and status.
Owner: Controller responsible for driving a custom resource from the observed state to the desired state. (Such as ReplicaSetController owns the ReplicaSet resources.)
Watch: Action of a controller that looks for resource under your control and other resources as well. (Such as ReplicaSetController watch for Pod resources.)
Orchestrate Many Controllers
This set of design patterns allows developer to break down composable, smaller controllers.
Controllers Watching The Same Object
A controller owns the CR, the second controller dynamically attach additional behaviours, eventually controllers can synchorize using finalizers or annotations.
USE WHEN…
- You want to make progress after a precondition is met, such as machine is being deleted.
PROS…
- Responsibility is well defined. Each controller does one task.
- It is a battle tested pattern, such as garbage collector
CONS…
- Limited orchestration options, such as finalizers
- Behavioral dependencies between controllers are not well documented, they don’t surface in API.
Controller Watching Related Objects
A controller owns a parent CR and creates/deletes child CRs, another controller owns the child CR.
USE WHEN…
- There are parent-child semantic relations between the objects you control.
PROS…
- Proved effectiveness, it is in Kubernetes core.
- Support sophisticated variants, such as history of changes, rollbacks
CONS…
- Not trivial to debug, why is the resource not progressing…
- Not trivial to evolve, such as change in a resource might impact more that one controller.
API Contracts
It is a variant of the controller watching related objects pattern where the second CR is pluggable and it is expected to have a set of well known fields (the API contract).
USE WHEN…
- There is a semantic relation between an object and a set of related objects (machine-infrastructure machines).
- It is possible to identify a set of common API fields across all the related object implementations (status.infrastructureReady).
PROS…
- It allows swappable implementations.
- Extensively used in Cluster API.
CONS…
- It requies all the stakeholders to agree on an API contract.
- Evolving the contract requires coordination across all stakeholders.
- It requires to use unstructured/generic client for accessing field defined in the API contract.
Extend Capabilities of a Single Controller
This set of patterns explore how developers can implement plug-in systems that can bring to a next level the number of use cases a single controller can support.
In Process Extensions
The controller runs plugins into its own process, each plugin extends a controller behaviour.
USE WHEN…
- You want to limit plug-in side effects.
- Binary compatibility is an issue.
PROS…
- WebAssembly is a cool technology with growing adoption.
CONS…
- We are still early WASM (specially WASI) stages.
- Support for WebAssembly isn’t at the same level of maturity in all languages.
- It doesn’t solve the plugin distribution problem.
Out Of Process Extensions
The controller watches for the webhook configurations pointing to plugin instances running across the network. The controller calls registered plugins.
USE WHEN…
- Plug-ins require a dedicated security context.
- It is requred to add/remove plugins at runtime.
PROS…
- Plug in side-effects cannot impact the main controller.
- same maintenance approach for plug in as the rest of the system.
CONS…
- The controller has a dependency on external services.
- It is required to manage authentication/authorization, network policies, failure policies, back pressure etc..
- Organizational implications.
In this post, we learned that Design Patterns For Extensible Controllers. Thanks for Fabrizio Pandini & Rafael Fernandez Lopez for this session.
That’s all I’m going to talk about for now. See you in my next post!
References:
KubeCon + CloudNativeCon Europe 2022: Schedule
KubeCon + CloudNativeCon Europe 2022: Kubernetes is Your Platform: Design Patt… (sched.com)
