Create Event Driven Airflow Pipeline with Amazon SQS

I have been thinking about making Airflow pipelines more event driven and wondering how Amazon’s SQS could facilitate that. Partly I would like to compare performance to AWS Lambda as an Airflow trigger.

So the thought process here is this: if you had a bunch of Airflow DAGs with some of them working in the downstream of earlier DAGs, how could you make it such that when DAGs A, B and C complete individual runs, they add a message to a queue what could be processed by a DAG D.

I put together a simplistic approach below that allows one DAG to insert messages to an SQS queue while another DAG listens for messages in the queue and processes any messages received.

Architecture for this demo

Set Up

AWS account access: grab AWS access key and secret key for your account so that your airflow job can connect.

SQS queue: create an Amazon SQS queue. I used all the default settings and called my queue ‘test_queue’.

SQS queue ‘test_queue’

Airflow Connection to AWS: Set up the AWS connection in the Airflow UI.