Κubernetes Horizontal Pod Autoscaler using external metrics

Agile Actors
Jun 22 · 4 min read

By Dimitris Kordas -Sr. DevOps engineer

As DevOps go more and more into the spotlight, there are some needs and ideas that make a huge difference in how we see and perceive the DevOps role and structure. “How can you use external metrics to scale out your application” is definitely one of them. Do you want to know how? Great! Read below.

Scaling out in a k8s cluster is the job of the Horizontal Pod Autoscaler, or HPA for short. The HPA allows users to scale their application based on a plethora of metrics such as CPU or memory utilization. This is all well and good but what happens if you want to scale out your application based on an application-specific business metric?

This is where using an HPA with the External metrics API comes in. It allows users to scale their applications on what they already know to be the key metric of their applications. The metric that will make-or-break their application when it is under load.

This metric might not be CPU or memory. Luckily K8S allows users to “import” these metrics into the External Metric API and use them with an HPA.

In this example we will create a HPA that will scale our application based on Kafka topic lag.

It is based on the following software:

Kafka: The broker of our choice.

Prometheus: For gathering metrics.

Kafka-lag-exporter: For computing kafka topic lag and importing metrics into Prometheus.

Pometheus-adapter: For querying prometheus for stats and providing them to the external metrics API.

Installation and configuration will be performed by helm v3.

First of all we are going to install Prometheus in the monitoring namespace:

Prometheus is now deployed in the monitoring namespace:

Next we are going to install Kafka in the kafka namespace using the Bitnami chart:

We are going to enable metrics.kafka.enabled so as to create a standalone kafka exporter as per the docs: https://artifacthub.io/packages/helm/bitnami/kafka

Kafka is now deployed in the kafka namespace:

Now that both Kafka and Prometheus are installed we need to find a way to get topic (+lag) information into Prometheus.

We do this by using kafka-lag-exporter:

Kafka-lag-exporter needs to be told where to find out kafka cluster. This is achieved by specifying the clusters[0].name and clusters[0].bootstrapBrokers properties. Feel free to adjust these settings to your liking.

The last piece of the puzzle is prometheus-adapter. It will query Prometheus using a predefined query and register these metrics with the external API server.

We use the following prometheus_adapter_values.yaml configuration:

This configuration instructs prometheus-adapter to perform the query defined in seriesQuery on Prometheus and expose the metrics as computed by metricsQuery as a new metric in the External API named kafka_lag_metric.

We can now query the external metrics API directly to determine if our configuration indeed works.

For the sake of this tutorial I have created a sample application using Quarkus that uses kafka queues. Here’s the deployment configuration. It includes the deployment as well as a service so that we can view the application.

Last but not least, comes the HPA

This HPA targets the kafkademo deployment and defines a minimum of 3 replicas and a maximum of 12 with a target value of 4.

Now when we try to get some more information about our HPA we will see the following:

Here you can see that the HPA is currently in the process of scaling down the kafkademo deployment since the current metric is below the defined target.

Well, as you can by now understand, we do love great and innovative ideas and, also, we love technology. That is also one of the reasons we’ve created a whole DevOps chapter in Agile Actors. Do you want to be part of this chapter? Do you seek improvement and professional development every day? Check out our current openings here!

You can check out this article and more on https://dkordas.com/ and dive into “The inner workings of a programmer’s mind”.

PlayBook