This is a story about why we wrote an Apache HTTPd based ingress operator for GCP’s GKE and why you should do the same (incl. some tips).
Since a couple of years, the operator- and controller pattern are one of the most discussed and implemented Kubernetes patterns out there, at least for me. Both patterns are overlapping at some point. The controller pattern is mostly used to watch Kubernetes included resources and the operator pattern is mostly understood as automation of operating tasks to keep the desired state in place. A well-explained summary about the similarities and differences between Kubernetes operators and controllers can be found inside this blog-post of Josh Rosso.
I fully agree with his definition of what qualifies a Kubernetes operator, which is:
- Contains workload-specific knowledge
- Manages workload lifecycle
- Offers one or more Kubernetes Custom Resource Definitions (CRD)
Basically, a Kubernetes operator takes care of the CRD’s which are provided by the operator. This is the first and very important difference between an operator and a controller. However, the operator at its core is a controller because it controls and manages the workload lifecycle it is responsible for. The workload itself is defined by the CRD’s. In addition, the operator has to know about what additional components, besides the core workloads are needed. This means, that the CRD’s can be rather simple, but the operator must create a lot of resources to satisfy the defined state. This resources can be GKE resources like Kubernetes services, GCP resources like load balancers or other resources located in other clouds like Route53 DNS entries for example as shown in the picture below.
Why we wrote an Apache HTTPd based ingress operator
As we now know what a Kubernetes operator is used for, we can put some light on the details on why we wrote a Kubernetes ingress operator and why it is Apache HTTPd based.
The first reason is, as you can read in the title of the picture above, that we have to handle a pile of legacy services, which are often based on Apache Tomcat. Apache HTTPd and Apache Tomcat are able to work together closely by using the Apache Tomcat mod_jk connector module for the Apache HTTPd. The Apache Tomcat mod_jk connector offers various load balancing mechanisms and it does also care about the backend health checking automatically. That’s why our Kubernetes ingress operator is Apache HTTPd based. The Bosnd-pod which is shown in the picture above contains a self-written Golang binary which discovers the Docker container backends in the on-premises setup and which reads out the backend pods in the Kubernetes environment.
The second reason for writing a Kubernetes ingress operator is, that the creation of an ingress load balancer based on Apache HTTPd is a semi-automatic process in our on-premises environment. There are multiple steps included, which are mostly scripted today, but some components have to be allocated manually. Here’s a list about the current semi-automated steps we have to do and which are now fully automated by using a Kubernetes ingress operator a showed in the picture above:
- allocate a new IP address on the Docker host which runs the Docker container which contains the Apache HTTPd for the Docker container backends
- create the Apache HTTPd Bosnd configuration which is used for the automatic configuration and Docker backend container discovery for this load balancer instance
- create the Puppet (config management) configuration to get the ingress Docker container started on the desired host automatically
- create a DNS record (AWS Route53) for CoreDNS which points to the IP address of the ingress Docker container
- if needed, create the same for the internet setup and allocate an external internet-facing IP address
- request a firewall forwarding from the external IP address
- create an additional DNS record (AWS Route53)for the internet name resolution
The third reason for us to write a Kubernetes ingress operator is, that we would like to delegate the backend service configuration which comes from the application developers to the application developers because they know what they need for their applications and services best.
The fourth and final reason for us to write a Kubernetes ingress operator is a summary of what’s written above. In some situations, we cannot use an NGINX based ingress controller, because simple HTTP forwarding is not enough and also deep insight for Apache Tomcat based backends is not given in detail. Furthermore, a GCP based load balancer does no fit because applications and services are subject to be changed by the developers in a flexible way and, however, not every developer is also an operator who is able to configure and operate Kubernetes.
As you can see, there are a lot of steps, which could cause a lot of pitfalls. So, that’s why we decided to write a Kubernetes ingress operator at the same moment we decided to use GKE in a hybrid/public cloud environment.
Why you should do the same
I think it's obvious that if you have to do a lot of manual steps to get your infrastructure up and running, everything can happen in an unpredictable way. That’s a situation you have to avoid. Therefore, if you extend your infrastructure by using cloud services, try to achieve some advantages besides pure technology change. If you can, try to also ease your work. We have done it by using better technologies (Kubernetes ingress operator) and by changing the process of how we work by making it possible to delegate tasks to other teams.
How can you do it too?
Currently, there are two main projects which are providing a framework for the implementation of a Kubernetes operator. The first one is the kubebuilder framework from the k8s-sig-api-machinery SIG, both the kubebuilder itself and the SIG are owned by Google. The second one ist RedHat’s Operator framework. At the time that we started implementing the Kubernetes operator, RedHat’s operator framework had slightly better documentation and was a little bit more recognized in online resources. However, both projects are a good starting point to learn about the implementation of CRD’s, the reconcile loop and many other aspects.
If you would like to learn more about it, leave a comment! I can share more information about the implementation and about the way we did it! 🤗
Thanks for reading and hopefully you’ve got an idea upon Kubernetes ingress operators and why they are useful even if it demands some effort to implement one.
Last edited on 13th December 2020