Running Kafka on Azure Kubernetes Service
Yes, Kubernetes was not designed for Stateful workloads. Yes StatefulSets and PersistentVolumes can be hard to reason about. Also when Kesley HighTower cautions against running stateful applications on kubernetes, it makes us wonder if this is worth trying.
On the other hand, Confluent — the awesome company behind Kafka — is running Confluent Cloud by deploying Kafka on Kubernetes using Operators.
There have been several tech talks in past few KubeCons where people from different companies share their experiences with running Kafka on Kubernetes. If you have time, do check them out. I also highly recommend reading this whitepaper by Gwen Shapira on recommendations about running Kafka on Kubernetes.
With this background I decided to deploy Kafka on Azure Kubernetes Service (AKS) and without a doubt, I wanted to leave Helm charts and try out Operators to help me accomplish this task, and in my quest of choosing the right operator for my use-case I ended up with the awesome Strimzi Operator for Kafka.
