NI Tech Blog
Published in

NI Tech Blog

Autoscaling your Airflow using DataDog External Metrics

Follow this step-by-step guide and learn how to autoscale your Airflow K8s pods based on ExternalMetrics

One of the strengths of Kubernetes lies in its ability to achieve effective resources autoscaling. To make scaling decisions, Kubernetes requires rational decision-based on service metrics.

In this post we will learn about the scaling decision, and how we can extract, monitor and autoscale Airflow workers based on Datadog external metrics.

Following are the steps one takes in order to accomplish Airflow HPA (Horizontal Pod Autoscaling) within the Kubernetes environment:

Setup Datadog cluster agent

The External Metrics Provider resource made it possible for monitoring systems like Datadog to expose application metrics to the HPA controller. Our scaling policy will be based on external metrics which we collect from our applications.

To learn how to activate and use DatadogMetric, refer to Datadog’s documentation.

Enable Airflow StatsD

Now, let’s configure Airflow to send metrics to StatsD. Our scaling decision metric is the executor.running_tasks. We decided to observe the number of running tasks on the executor and to get a rational decision whether to scale the Airflow workers’ pod up or down.

Step 1: Install StatsD

pip install 'apache-airflow[statsd]'

Step 2: Configure StatsD environment variables:

Decide what metrics to expose by configuring an allowed list of prefixes. Here we choose to send the executor metrics as we are interested in the metric of the running tasks.

statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
datadog_enabled = true
# Allow List of Metrics
allow_list: executor
# Datadog Tags of the environment
datadog_tags: "statsd_group:datainfra,statsd_env:production"

Create a Custom Resource Definition (CRD) like DatadogMetric

DatadogMetric allows the HPA to autoscale applications based on metrics that are provided by third-party monitoring systems. External metrics support target types of Value and AverageValue.

The Horizontal Pod Autoscaler is a great tool for scaling stateless applications. But you can also use it to support scaling statefulsets.

Following is an example of how to create it using Terraform:

resource "kubectl_manifest" "DatadogMetric" {
yaml_body = <<YAML
kind: DatadogMetric
name: running-tasks
namespace: production
query: avg:airflow.executor.running_tasks{*}.rollup(max, 60)

Validate the value of a metric by sending a raw GET request to the Kubernetes API server:

kubectl get --raw "/apis/" | jq .


"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "",
"resources": [
"name": "datadogmetric@production:running-tasks",
"singularName": "",
"namespaced": true,
"kind": "ExternalMetricValueList",
"verbs": [

Create HPA using the DatadogMetric

Next, we will expose executor.running_tasks which will be queried by the HPA.

The HPA fetches executor.running_tasks value from the metrics API and will scale up the airflow-worker if the current running tasks go over 1 task.

Using HPA in combination with cluster autoscaling can help you achieve cost savings for workloads that see regular changes in demand by reducing the number of active nodes as the number of pods decreases.

Here is the set of the configuration in the Helm chart value:

enabled: true
replicas: 1
cpu: 800m
memory: 3.5Gi
enabled: true
maxReplicas: 2
- type: External
name: "datadogmetric@production:running-tasks"
type: AverageValue
averageValue: 1


DatadogMetric allows the HPA to autoscale Airflow’s workers based on statsD metric provided by third-party monitoring systems.

It’s possible to take advantage of these insights and implement them in other areas within your production environment.

Airflow HPA workers with Datadog External Metrics

In the following, you can see a full working example of how to deploy the Airflow with a helm chart including the HPA and DatadogMetric.

## add this helm repository & pull updates from it
helm repo add airflow-stable
helm repo update
## install using helm 3
helm install \
airflow \
airflow-stable/airflow \
--namespace production \
--version "8.5.2" \
--values ./custom-values.yaml

custom-values.yaml can be found: here




Natural Intelligence is a global leader in the online comparison market. This publication allows members of Natural Intelligence technology teams to share their knowledge and experience.

Recommended from Medium

Planning a beginner open source sprint day for data scientists

Use SocketCAN as data link type of CAN frame

How we used Continuous Delivery to deliver a Continuous Delivery tool


Windows 11: The Optimization and Performance Improvements

Performance: Improving memory needs and dict lookup time with string interning

The rise of MongoDB

Joe Rezendes — The rise of MongoDB

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Lidor Ettinger

Lidor Ettinger

Productivity Can Drive Improvements in Work/Life

More from Medium

Running Spark Pipelines on EMR Using Spots Instances

Orchestration of AWS EMR Clusters Using Airflow —The Insider Way

Apache Airflow migration journey from self-hosted to AWS Managed Airflow

Deploying Airflow 2.0 on EKS using Terraform, Helm and ArgoCD — Part 1/2