The State of Custom Resource Definitions in Kubernetes

A comparison of Kubernetes native resources to CRDs.

Last updated July 2018.

We are heavy users of Kubernetes Custom Resource Definitions (CRDs) at Manifold. They are an excellent way to express configuration for Kubernetes native applications (like our Credentials controller) or to encapsulate related resources, like turbo-charged templating.

While each release of Kubernetes brings CRDs closer to feature parity with native Kubernetes resources, they aren’t there yet. This post highlights the differences between the two that we at Manifold are most interested in. We intend to keep the post up to date, moving entries out of our wish list as they’re resolved.

What’s a Custom Resource Definition?

Kubernetes is a container orchestrator. It places containers on servers, and wires up networking for them. It is also quite pluggable and extensible. At the container level a cluster operator can plug in different networking drivers. At the cluster level an operator can add a custom controller to respond to changes to resources. Beyond that, control of the underlying cloud provider infrastructure is pluggable, as well. Most of the Kubernetes extension points are handled through Remote Procedure Calls. This necessitates APIs for accessing objects within the Kubernetes system for the handlers of these remote calls.

In a way, the Kubernetes API server can be thought of as a database. It allows storage and retrieval of Kubernetes resources. Controllers are notified of changes to these resources, and manipulate them and other resources in response, or reflect these changes into external systems (like a cloud provider’s load balancer). If the API server is a database, then CRDs are user-provided schema definitions. Once you have a method to define your own schema, you can apply the same controller mechanisms to it as used for native resources.

As stated above, CRDs are not complete replacement for native Kubernetes resources. They likely never will be; it’s difficult or impossible to design a way to declaratively express resources in a way that will be as flexible as writing arbitrary code. CRDs are good enough for most cases, and being able to use Kubernetes as a storage engine is a worthy tradeoff.

If CRDs don’t meet your needs, you might want to consider API server aggregation instead. API server aggregation gives you complete control over how your resources are defined, stored, and accessed. This complete control is also their burden.

Our Custom Resource Definition Wish List

From Manifold’s point of view, these items are the remaining places where CRDs fall short of native Kubernetes resources to make them true replacements. This list isn’t long, and it gets shorter with every Kubernetes release!

CRD Version Conversion

While we can now define multiple versions of a CRD, conversion between them is isn’t supported yet. The tracking issue is here.

Multi-version support is not as important for our internal-use only CRDs. For those, we can create new versions under a completely different name.

Strategic Merge Patch

Strategic merge patch is an enhancement to the standard JSON merge patch format that makes the patch format more supportive of concurrent modifications, or performing updates to a single field, without caring about any other fields on the object. The format adds directives that describe how to handle non-scalar types. These directives may exist in the patch itself, or as part of the schema of the object being patched. Effectively, strategic merge patch can turn Kubernetes resources into conflict-free replicated data types (CRDT — an unfortunate initialism in the context of this post).

It is possible to perform a strategic merge patch on a custom resource definition. It is not yet possible to do so on a resource that implements that definition’s schema.

Simplified Development

Building a complete custom resource requires many components:

• The CRD itself
• Validation rules
• Validating admission webhooks
• Mutating admission webhooks
• A controller that provides the backing logic for your CRD
• A way to package and deploy everything to a Kubernetes cluster

This isn’t something Kubernetes itself should solve. It makes more sense as a library or framework on top of Kubernetes.

A few projects exist to make authoring CRDs easier, such as metacontroller and kubekit (from Manifold’s own Jelmer Snoeck!)

Wishes Already Granted

With four releases a year, Kubernetes moves fast. This section holds features we wished for at one point, but now exist.

Validation

Added in Kubernetes 1.9 (December 2017)

Having a method to ensure that the values of CRD fields are valid, that required fields exist, and so on, is important, particularly when designing CRDs for wider use than your immediate team.

Custom Resource Definitions can declare validation rules with OpenAPI schema, which is a subset of JSON Schema. Note that OpenAPI schema does not support all features of JSON Schema, and CRD validation does not support all features of OpenAPI schema. Still, it should be enough for most cases.

If you need more advanced validation features, like validating one field based on the value of another field, you may need to use a validating admission webhook as well.

Default Values

Added in Kubernetes 1.9 (December 2017)

JSON Schema does include a default keyword, but the specification does not enforce that tools support it; primarily it’s for documentation. As such, it’s not too surprising that the JSON Schema via OpenAPI validation supported for CRDs does not support the default keyword.

Fortunately, we can use mutating admission webhooks to populate default values on resources as they enter the system.

If you’re certain no other components will use your CRD, you can make the default values implicit in your controller implementation.

Multi-Version Support

Added in Kubernetes 1.11 (June 2018)

A compelling feature of native Kubernetes resources is their ability to automatically and transparently migrate between api versions. Consumers of the resources can use mixed API versions, and all get the version of the resource they expect. As of Kubernetes 1.11, CRDs support multiple versions, but conversion between them must be done manually. The design doc for multiple version support is viewable here.

Wrap-up

CRDs have come a long way in their short life span! This is a testament to the solid release process of Kubernetes, and the Open Source mantra of:

“Release early, release often.”

This post only captures the issues and enhancements that are most interesting to us at Manifold. Many more issues have been resolved (and a few more are still outstanding) than are listed here, without even touching on the previous iteration of this extension mechanism, Third Party Resources.