Autoscaling Kafka Consumers using Kubernetes Event Driven Autoscaling (KEDA) — Part 1

Leveraging autoscaling in kubernetes has become way more important in the current cloud native technology to optimise on the utilisation of resources and save costs.

There are different autoscaling technologies (HPA, VPA, KEDA, Cluster Autoscaler, etc) available today which can be used for autoscaling compute resources. These technologies helps us in scaling based on different metrics in our environment which includes either CPU/Memory exported by metrics server running in the cluster or external metric sources like Cortex, Prometheus, Queues, Etc.

KEDA along with Cluster Autoscaler helps in scaling compute resources exactly based on need thus ensuring we do pay only for compute when required and scale as per the input which we are getting from the source.

In this series of articles we are going to see how KEDA can be utilised to scale ELT workloads (source-destination sink) in data ingestion pipeline to reduce cost while handling latency and lag. While this is specific to Kafka workers, it can be applied to any deployment running in kubernetes clusters.

Background

As part of Data Environment we often run several stateless workers which consume messages from different source Kafka Clusters and pushes messages to various sink destinations based on user configurations. These workers can run as kubernetes deployment in the cluster and orchestrated via templates rendered through helm charts.

In this PART 1 of the series of articles, we are going to see about KEDA.

What is KEDA?

KEDA is kubernetes-based event driven autoscaling. On a high level it comprises of 3 components,

1. KEDA Operator: It’s a singleton resource with the active-standby and can be run as a deployment in kubernetes cluster. It runs an agent which helps in managing custom resources created through KEDA and also helps in scaling target resource from 1–0 (de-activation) or 0–1 (activation) and from 1-n and n-1 using HPA (Horizontal Pod Autoscaler) in the background.
2. KEDA Metrics Server: Metrics server is an agent running in kubernetes cluster as deployment and acts as a bridge between API Server and custom objects requesting for metrics. Custom Metrics API server endpoint registered in kubernetes is directed to this backend service.
3. KEDA Admission Webhooks: Validates custom objects created through KEDA and ensures best practices are followed while creating the custom objects.

KEDA Custom Objects

1. scaledobjects: Mapping between monitor source and kubernetes objects defining /scale sub resource like Deployment, Statefulsets, etc.
2. scaledjobs: Mapping between event source and kubernetes job.
3. Trigger Authentication: Authentication information for monitoring sources passed as reference in scaler object.
4. Cluster Trigger Authentication: Same as trigger authentication but works at cluster level.
5. Scalers: Keda supports multiple sources as scalers which can be combined as triggers to scale target resources.

Deployment

KEDA operator, metrics server and admission controller can be deployed via helm chart in kubernetes.

You will require to whitelist port 6443 in google cloud and port 443 in other clouds to Pod IP range for communication of API server with the APIService created for external metrics.

Sample values file for helm chart:

resources:
  operator:
    limits:
      memory: "1000Mi"
    requests:
      cpu: "100m"
      memory: "100Mi"
  metricServer:
    limits:
      memory: "1000Mi"
    requests:
      cpu: "100m"
      memory: "100Mi"
  webhooks:
    limits:
      memory: "1000Mi"
    requests:
      cpu: "100m"
      memory: "100Mi"

prometheus:
  metricServer:
    enabled: true
    path: /metrics
    portName: metrics
    serviceMonitor:
      enabled: true
  operator:
    enabled: true
    serviceMonitor:
      enabled: true
  webhooks:
    enabled: true
    serviceMonitor:
      enabled: true

upgradeStrategy:
  operator:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  metricsApiServer:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  webhooks:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

To install in kubernetes cluster

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda -f values.yaml

values same as in above snippet.

Monitoring of keda objects can be done via sample dashboard using prometheus and grafana and alerts set up failures of keda scalers can be configured to alert when scaler is unable to compute metrics or fails due to some reason.

Conclusion

In this article, we have seen basics of KEDA and understood the deployment and monitoring aspects of KEDA in kubernetes.

In next articles,

1. We will understand about the scalers and implementation details for the Kafka Consumers Autoscaling.
2. Challenges faced during the implementation and discuss potential solutions for the same.

Please keep looking for more in the series.

You can follow me on LinkedIn and Medium for more such articles. Happy Learning!!!