10 Benefits to using Airflow

In continuation from a series of posts, where I have explained the basics of airflow and how to setup airflow on azure, and what considerations to have when using airflow, I wanted to cover in details what makes airflow a great tool to use for data processing.

1. DAGs:

Dags are a way to setup workflows, they can setup a sequence of operations that can be individually retried on failure and restarted where the operation failed. Dags provide a nice abstraction to a series of operations.

2. Programmatic Workflow Management:

Airflow provide a way to setup programmatic workflows, Tasks for instances can be generated on fly within a dag. While Sub-DAGs and XCom, allows to create complex dynamic workflows.

Dynamics Dags can for instance be setup based on variables or connections defineed within the Airflow UI.

3. Automate your Queries, Python Code or Jupyter Notebook

Airflow has a lot of operators setup to run code. Airflow has operator for most databases and being setup in python it has a PythonOperator that allow for quickly porting python code to production.

Papermilll is an extension to jupyter notebook, allowing the parametrization and execution of…