Apache Airflow Dynamic Task Mapping: Tricks That Distinguish Seniors from Juniors
Remember this article that explains Apache Airflow DAG Factory? In this article We will learn the same tool, I’m a fan of Apache Airflow this tool makes my life easier in terms of orchestrating complex workflow since we are working on a lot of data sources that have their own complexities, volumes, and used in the different downstream cases.
Problem Statements
For you who working on big volumes of data that should be moved through the same pipeline, I know that this could be a daunting process. The idea is we can separate the process into a number of predefined groups of tasks, hence we can process it like a multiprocessing. But, what if we don’t know how much group we should define? so the number of tasks dynamically changing. Then, we should call Apache Airflow Dynamic Task Mapping to resolve this issue.
Dynamic Task Mapping
From Apache Airflow's official website, Dynamic Task Mapping allows us to author workflow by creating a number of tasks at runtime based on current data, even if the DAG author doesn’t know in advance how many tasks would be needed. Interesting right? Here are the reasons why we need Dynamic task mapping :
- Dynamic Task Mapping allows you to design workflows…
