Container Talks
Published in

Container Talks

Autoscaling Using KEDA Based On Kafka Lag

In this blog, we will discuss about scaling kubernetes deployments based on Apache Kafka topic lag with KEDA.

Kafka Lag Based Autoscaling with Keda

What is Apache Kafka ?

Kafka is a distributed message streaming platform that uses a publish and subscribe mechanism to stream the records or messages.

What is Keda ?

KEDA is a Kubernetes-based Event Driven Autoscaler. KEDA can be installed into any Kubernetes cluster and can work alongside standard Kubernetes components like the Horizontal Pod Autoscaler (HPA). When we install KEDA in any kubernetes cluster, two containers run. First one is keda-operator and second one is keda-operator-metrics-apiserver. Here, Role of keda-operator is to scale Deployments, Statefulsets or Rollouts from zero or to zero on no events. KEDA acts as a Kubernetes metrics server that exposes rich event data like queue length or stream lag to the HPA to drive scale out.The metric serving is the primary role of the second container i.e. keda-operator-metrics-apiserver.

Installing Keda (Using Helm)

  1. Add helm repo & update it
helm repo add kedacore https://kedacore.github.io/charts
helm repo update

2. Install keda Helm chart

helm install keda kedacore/keda --version 1.4.2 --create-namespace --namespace keda

Now as have installed keda, let’s take an example and see how we can trigger autoscaling.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: kafka-scaledobject
namespace: demo3
spec:
scaleTargetRef:
apiVersion: argoproj.io/v1alpha1
name: keda-test-demo3
kind: Rollout
triggers:
- type: kafka
metadata:
bootstrapServers: kafka-demo3.demo3:9092
topic: test
consumerGroup: console-consumer-76109
lagThreshold: "300"
offsetResetPolicy: latest
maxReplicaCount: 10
minReplicaCount: 1

In .spec.triggers section, we provide the informations that KEDA uses to trigger the autoscaling. Here are some of the parameters which can be used for triggering autoscaling.

keda autoscaling parameters & descriptions

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Devtron Labs

Devtron Labs

Devtron is an open source no-code solution for Kubernetes deployments. https://github.com/devtron-labs/devtron