Kafka Workers Autoscaling With Horizontal Pod Autoscaler

This is a step-by-step article that guides you from scratch. I assume you only have access to a vanilla Kubernetes cluster. In this article, I’m using k8s external metrics to autoscales my kafka consumers. There are other articles online that shows you how to use custom metrics instead. I chose to use external metrics since it’s more applicable to real production environment where you might want to scale based on other metrics that are available on Prometheus. For a great tutorial for custom metrics you can go to this medium article.

Outline

There are a few steps we need to do to use HPA with custom and/or external metrics. You can skip these steps if you happen to already have kafka or prometheus running.

1. Deploy Kafka
2. Deploy Prometheus
3. Expose Prometheus and Grafana services
4. Deploy a Kafka consumer application
5. Deploy Prometheus Adapter
6. Deploy HPA

Deploy Kafka

We will use Helm for most of our installation process out of convenience. Run this command on your terminal to install Kafka.

$ helm install \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
kp prometheus-community/kube-prometheus-stack

Deploy Prometheus

Run this command on your terminal to install Prometheus.

$ helm install \
--set prometheus.kafka.enabled=true \
--set prometheus.operator.enabled=true \
--set prometheus.operator.serviceMonitor.releaseNamespace=true \
kafka incubator/kafka

Expose Prometheus and Grafana Services

For ease of use, we expose Prometheus and Grafana services. As a side bonus, Prometheus Adapter also needs to connect to prometheus service using a url, which defaults to http://prometheus.svc.default:9090.

$ kubectl create -f prometheus-service.yaml
$ kubectl expose deployment kp-grafana --type=NodePort --name=grafana

At this point, you can open your Prometheus or Grafana dashboard to verify that Kafka and Prometheus are connected. You can run minikube service prometheus or minikube service grafana to open the dashboards respectively.

Deploy a Kafka Consumer Application

The Kafka consumer application will simulate your Kafka workers. Here the consumer app use a bootstrap-server of kafka:9092 and listens to the my-topic topic.

$ kubectl create -f kafka-consumer-application.yaml

Deploy Prometheus Adapter

We deploy Prometheus Adapter using Helm. We also needs to change some values since Prometheus Adapter doens’t recognize custom and/or external metrics by default. In values.yaml, we define the metrics that we want to use for HPA. In this example, we use kafka_consumergroup_lag as our chosen metrics. The query that we use alongside this metric is avg_over_time(kafka_consumergroup_lag{topic="my-topic",consumergroup="we-consume"}[1m]).

$ helm install prometheus-adapter prometheus-community/prometheus-adapter -f values.yaml

To verify that everything is working properly, you can run this command to check whether or not k8s metrics API has acknowledged the metrics needed for HPA. In the command below we verify the existence of external metrics, but you can also run the same command for custom.metrics.k8s.io.

$ kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq

Deploy HPA

As we already verified that HPA should be able to receive the external metrics, we can go ahead and deploy the HPA.

$ kubectl create -f consumer-hpa.yaml

Test

You can run a simple test to verify that HPA is working properly by loading the `my-topic` topic using a kafka producer. You can do so by running the command below.

$ kubectl run kafka-producer -i --image=confluentinc/cp-kafka:5.0.1 --rm=true --restart=Never -- kafka-console-producer --broker-list kafka:9092 --topic my-topic < file.txt