Building higher-level abstractions in Kubernetes

Over the past few years, Kubernetes has established itself as the de-facto standard for container orchestration. There are plenty of reasons for its success, including a rich community and a strong backing by large companies. However, setting the organizational reasons aside, where Kubernetes especially stands out is in its well-designed, rich, and extensible API that can account for a wide range of use cases.

The downside of having such an extensive API is the complexity and the non-trivial barrier of entry that it imposes on the end-user. This problem is especially pronounced in fast-growing microservice-based ecosystems, such as the one at Personio, where things are changing all the time. In this blog post, we explain how simpler, domain-specific abstractions can be built on top of the Kubernetes API when tools like helm or kustomize stop being effective.

Limitations of Helm Charts

At Personio we have close to 100 services running in production. Of those, around 40 serve product workloads and that number is continuously growing. The deployment manifests for each microservice are bundled as a helm chart and are part of the same repository as the service itself. With the growth of the engineering organization, our knowledge and understanding of Kubernetes evolved, and so did the charts used for deploying these microservices. Typical improvements which we introduced over time include adding affinity rules and PodDisruptionBudgets for Deployment objects. Each iteration required updates to each microservice chart and as the number of services grew, the cost of making such changes from an organizational perspective grew as well.

Our initial solution to this problem was maintaining a reference chart as part of a microservice template, with every team being responsible for keeping up with updates. This solution served us well at a smaller scale, but started to create unnecessary work for every team.

Some of the updates that we did, such as adding a PodDistruptionBudget, could have been rolled out safely without needing involvement from the teams owning the services. In contrast, we ended up having to create a pull request in each repository, identify the owning team, and follow up to make sure the PR was reviewed, merged, and rolled out to production. This particular example illustrated the need for a more centrally-managed solution where we could make improvements and adjustments without disturbing teams. In other words, we started to see the need for a domain-specific platform on top of Kubernetes.

Enter Custom Resource Definitions

We wanted to create a minimal configuration file for our engineers that allows them to run a docker image on Kubernetes without them needing to worry about all the plumbing and wiring. In addition to this, we needed the possibility to roll out changes centrally across all microservices without creating toil for other engineering teams.

After some research into what is available in the Kubernetes ecosystem, we decided to introduce a custom Microservice resource as a Kubernetes CRD:

apiVersion: personio.de/v1
kind: Microservice
metadata:
  name: ground-breaking-microservice
spec:
  image: ground-breaking-microservice:v1.1
  cpu:
    request: 500m
    limit: 1
  memory: 256Mi
  configMapName: config-name
  secretName: secret-name

The `spec` of the custom resource exposes all of the parameters that need to be specified by the team owning the service. The custom resource in itself does not contain any behavior and merely serves as an interface. The behavior associated with the resource needs to be fulfilled by a Kubernetes controller.

In our particular case, we had to develop a controller with the responsibility of taking the `Microservice` `spec` and creating well-configured Kubernetes resources from it, such as a Deployment, Service, Ingress, etc.

Implementing a Kubernetes Controller

There are a few open source frameworks which greatly simplify the development of Kubernetes controllers, including Kubebuilder and OperatorSDK. These are good general-purpose frameworks which can address a wide range of use cases. They can help with managing resources inside as well as outside of Kubernetes.

Our particular case however, was a perfect match for Metacontroller, a framework that makes it easy to write and deploy custom controllers in the form of HTTP webhooks and is only scoped to managing Kubernetes specific resources.

Metacontroller is installed as an operator in Kubernetes. It takes care of watching for updates on our custom `Microservice` resources and making sure they are processed in a FIFO manner to prevent race conditions. Our responsibility then becomes implementing an idempotent webhook which takes in the parameters of the `spec` and returns a list of Kubernetes objects for those parameters. Based on the webhook response, Metacontroller will interact with the Kubernetes API to reconcile the desired state by creating, updating, or deleting Kubernetes objects.

Metacontroller will also run the reconciliation loop for all resources periodically, and in our specific case every minute. This allows us to propagate changes to all microservices by making a change in our webhook. This periodic reconciliation completely eliminates configuration drift between different microservices and gives us the confidence that all of them are deployed and configured in Kubernetes in exactly the same way.

Ensuring a High Quality Bar

A centralized solution to manage deployments has its own drawbacks which should not be taken lightly. Problems in the reconciliation workflow can affect all microservices or cause global outages.

For example, exposing the wrong port in a Service object when deploying a Microservice could cause network calls between microservices to suddenly start failing.

To mitigate this problem, we rely heavily on both unit and integration tests. Our webhook is thoroughly tested using conventional unit tests to ensure Kubernetes resources for different configurations are created and wired up appropriately.

In addition to unit tests, we also leverage integration tests against an actual Kubernetes cluster. The integration tests run for every branch as part of a deployment pipeline and use kind to dynamically spin up and tear down a Kubernetes cluster inside a docker container. Every pipeline run creates a new cluster with Metacontroller and the supporting webhook. It then deploys a microservice resource and runs assertions verifying that the appropriate objects were created in Kubernetes. At the end of the test run, the cluster is torn down.

Conclusion

In the Developer Experience (DX) team, our mission is to actively work on reducing complexity within our stack. To make progress on that mission, we often need to do more so that everyone else can do less.

What this particular project has taught us about Kubernetes is what the community has been trying to say all along: Kubernetes is a great platform for building platforms. It provides well designed primitives that can be used to run containers across many nodes.

But, those primitives are not always the simplest to use, understand, or manage on a larger scale. This is why we at DX constantly ask ourselves which parameters our engineers want to control, and which ones are toil that they have to deal with.