Integrating Airflow + Datadog using docker-compose
3 min readJan 24, 2020
Integrating Airflow running on Docker + Datadog took way longer than I expected, so I decided to simplify it.
Couple of requirements before you get into this guide:
- Your Airflow instance is Dockerized
- You are familiar with docker-compose
Configuring Airflow to write to Datadog’s StatsD agent
You can do this a few ways: editing airflow.cfg or by setting configuration options through ENV. I’m going to outline how to configure statsd on airflow through ENV vars:
- Install airflow with statsd packaged in it. This is as easy as updating your
requirements.txt
fromapache-airflow==1.10.7
toapache-airflow[statsd]==1.10.7
- In your
docker-compose.yml
environment section add below ENV vars to enable statsd:
environment:
- AIRFLOW__SCHEDULER__STATSD_ON=True
- AIRFLOW__SCHEDULER__STATSD_HOST=datadog
- AIRFLOW__SCHEDULER__STATSD_PORT=8125
- AIRFLOW__SCHEDULER__STATSD_PREFIX=airflow
3. Congratulations you are done.
Adding Datadog to your docker-compose file
- If you follow datadog’s example for configuring
docker-compose.yml
integrating datadog to your network is pretty simple. But there is a few settings that you (probably) need. - You’ll need to expose
8125/udp
port - I think you’ll need to enable a couple of other ENV vars:
DD_APM_ENABLED
,DD_DOGSTATSD_NON_LOCAL_TRAFFIC
,DD_DOGSTATSD_ORIGIN_DETECTION
,DD_AC_INCLUDE
- I’d recommend also setting your
DD_HOSTNAME
ENV var as well. I choseAIRFLOW-PROD
(don’t fuck up and use an underscore, that was my rookie mistake) - The datadog section should look like:
datadog:
image: datadog/agent:latest
container_name: datadog
ports:
- "8125:8125/udp"
links:
- airflow
environment:
- DD_API_KEY=
- DD_HOSTNAME=AIRFLOW-PROD
- DD_APM_ENABLED=true
- DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
- DD_DOGSTATSD_ORIGIN_DETECTION=true
- DD_AC_INCLUDE="image:*"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /proc/:/host/proc/:ro
- /sys/fs/cgroup:/host/sys/fs/cgroup:ro
networks:
- main
The finished docker-compose.yml file
version: "3"
services:
airflow:
image: apache/airflow
container_name: airflow
restart: unless-stopped
ports:
- "8080:8080"
- "8888:8888"
environment:
- AIRFLOW__SCHEDULER__STATSD_ON=True
- AIRFLOW__SCHEDULER__STATSD_HOST=datadog
- AIRFLOW__SCHEDULER__STATSD_PORT=8125
- AIRFLOW__SCHEDULER__STATSD_PREFIX=airflow
networks:
- main
# agent section
datadog:
image: datadog/agent:latest
container_name: datadog
ports:
- "8125:8125/udp"
links:
- airflow
environment:
- DD_API_KEY=
- DD_HOSTNAME=AIRFLOW-PROD
- DD_APM_ENABLED=true
- DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
- DD_DOGSTATSD_ORIGIN_DETECTION=true
- DD_AC_INCLUDE="image:*"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /proc/:/host/proc/:ro
- /sys/fs/cgroup:/host/sys/fs/cgroup:ro
networks:
- mainnetworks:
main:
Running it
Run docker-compose
and wait to see:
Creating airflow ... done
Creating datadog ... done
Hopefully your dashboard starts populating — this was my (shitty) finished product:
Mistakes to avoid
- Following this guide for using the datadog airflow-integration — the huge issue here is that it’s aimed at users that are running airflow on the local machine + the datadog agent on that machine. If you just have your airflow up on an EC2 box — feel free. But it looks like most airflow metrics come through to data-dog without it!
- Following the datadog docker-compose guide and trying to use autodiscovery for the airflow integration above. Turns out the airflow integration isn’t in the public catalog and cannot be auto discovered!
But I really want to use the airflow integration…
Neither me nor Datadog support recommend this approach — but here’s their response:
Hi Lukas,
I just wanted to follow up with you based on our chat from earlier as it seems like we got disconnected.
The airflow integration is viable, but just not available to use with autodiscovery.
As a workaround, you could run this Airflow integration as a custom check: https://docs.datadoghq.com/developers/write_agent_check/?tab=agentv6v7#overview. This would involve copying the Airflow python code into a file in the agent's checks.d directory.
Once this integration has been set up as a custom check, you can mount your custom yaml file using a ConfigMap: https://docs.datadoghq.com/agent/kubernetes/integrations/#configmap.
Would this suit your use case?
Best,Redacted | Solutions Engineer | Datadog
Good luck in your Airflow adventures.