Writing Kubernetes custom resource definitions (CRDs) with custom controllers
Using Kubebuilder in production
Four common mistakes to watch out for when using Kubebuilder in production
At Data Minded, we use the Kubebuilder project in our Datafy product offering. Datafy aims to help customers build and run data projects at scale without worrying about the required infrastructure.
We use Kubebuilder to manage Spark batch and streaming jobs, run containerized batch jobs, Airflow, and notebooks for our customers. It is at the core of everything we develop on the Datafy platform. Over the last years, we got a lot of experience with it, and we want to share some best practices we are currently using.
What is Kubebuilder?
For those who don’t know Kubebuilder, a short introduction: Kubebuilder is a project that scaffolds your project and that uses the underlying controller-runtime SDK to allow you to manage your own custom resource definitions (CRDs) in Kubernetes. CRDs allow you to add your own objects to Kubernetes and build your own logic around it. This means that users can use kubectl to create, for example, a PostgreSQL object in Kubernetes, similar to how they use it to create a deployment, service, or service account…. Kubernetes will then use your controller logic to manage that object. Inside, you can automatically create the necessary disk, statefulset, and config in Kubernetes to create the PostgreSQL database for the user.
Kubebuilder has made a whole tutorial on creating your own controller; you can read about it in the kubebuilder book. Our tips are an addition to the concepts explained there.
Saving errors as events on CRD’s
One of the things we always do when developing a new CRD is to ensure that we store events on the CRD every time something goes wrong in our control loop. These events can be seen when using kubectl describe. For example, these are the events of a pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 38m default-scheduler Successfully assige...
Normal Pulling 38m kubelet Pulling image "6749...
Normal Pulled 38m kubelet Successfully pullin...
Normal Created 38m kubelet Created container a...
Normal Started 38m kubelet Started container a...To store events on your own CRD, you can use the following snippet:
Exporting events like this has been a lifesaver in production. Doing this allows us to easily debug why there are issues with a certain object without having to scrub through the controller's logs. Remember your controller might be managing thousand of resources in production, and when multiple of them are having issues, it is easy to get swamped in logs.
Managing the error logs
When we started creating our own controllers, we had lots of error logs on our controller when something went wrong. A couple of things made error logging more manageable for us.
Don’t log errors yourselves. The Kubebuilder framework logs every error you return in the reconciler loop. Not logging the error yourself cuts logging lines by half.
Automatically retry certain actions. When updating resources or saving the state of your object, you sometimes get conflicts.
For example, our Airflow controller manages a deployment for the airflow web application. Our controller will thus manage this deployment, but the Kubernetes deployment controller is also making changes to the object. This means we saw lots of conflict errors in our logs, but these errors are not that interesting as in the next loop iteration, things are likely to succeed.
But there is a better way: the apimachinery package contains a function called: wait.ExponentialBackoff. With this function, you can construct a function that behaves like controllerutil.CreateOrUpdate but with retry functionality:
The documentation of controller-runtime mentions the following onretry.DefaultBackoff:
DefaultBackoff is the recommended backoff for a conflict where a client may be attempting to make an unrelated modification to a resource under active management by one or more controllers.We can see that these are the recommended settings for when a resource is managed by one or more controllers, which is exactly what we are doing.
Another way to handle these conflicts is using Server Side Apply; however, it is at the moment poorly supported by the framework. Most of our code was also written before this functionality even existed.
Calling the API of your cloud vendor
Your cloud vendor often has an API and an SDK to call that API. However, you need to remember that most of the cloud vendor APIs are rate limited or have a price attached to them. So you have to watch out.
Something that worked for us is to keep a hash of the configuration of the AWS resources we are managing in the state of our CRDs. If we need to change the AWS object, the hash changes, and we can apply our changes to the AWS cloud. However, this has a downside that when something changes in AWS, we will not correct it since our hash isn’t changing. We are managing a relatively small amount of AWS resources, so it is not really an issue for us. Later we want to build in a last checked timestamp so we can check every x minutes if the resource needs an update. That way, we can correct manual mistakes made by users.
Using maps in your CRD
You can use maps in your CRD spec or state. There is, however, one gotcha you have to be aware of. The JSON serialization/deserialization does not guarantee any order. This is usually not a problem as you will never loop by index over a map, but it can result in issues.
For example, we send updates to our API when a spark job has started or finished. We don’t want to send updates multiple times, so we calculate a hash of the message we will be sending. We save this hash in the state so we can detect if we have already sent this message. To calculate a hash of an object, you can convert it to JSON and calculate a string hash in Golang. Something like this:
Using a map means that our JSON representation can change, and thus the hash will change continuously, resulting in us spamming our own API. It’s better to use lists for these use cases and make sure you sort them before calculating a hash. That way, your hash representation will remain the same on every iteration.
Conclusion
At Datafy, we love the Kubebuilder framework and use it to manage almost everything for our customers. We hope you will also have an even better time using the kubebuilder framework by using these extra tips.
