What is Airflow

Apache Airflow is an open source platform for programmatically authoring, scheduling, and monitoring workflows.

With Airflow, engineers build directed acyclic graphs (DAGs) to define workflows. One DAG contains a set of tasks, and each task is one instance of an operator. Airflow provides a lot of built-in operators: BashOperator to run bash scripts, PythonOperator to run any Python code, S3FileTransformOperator to move data in S3, etc. Once a DAG is defined, the Airflow scheduler executes those tasks on an array of workers while following the specified dependencies. …