What is Airflow

Apache Airflow is an open source platform for programmatically authoring, scheduling, and monitoring workflows.

With Airflow, engineers build directed acyclic graphs (DAGs) to define workflows. One DAG contains a set of tasks, and each task is one instance of an operator. Airflow provides a lot of built-in operators: BashOperator to run bash scripts, PythonOperator to run any Python code, S3FileTransformOperator to move data in S3, etc. Once a DAG is defined, the Airflow scheduler executes those tasks on an array of workers while following the specified dependencies. …

Tony Qiu

Data Platform Architect at Kabbage

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store