Airflow: Tips, Tricks, and Pitfalls
Marcin Tustin
62315

Thanks for the great writeup! I’ve just gotten started with Airflow and have a quick question I thought you could help with:

I’m creating dynamic DAGs with the PythonOperator similar to what you have using LogIngesterOperator. Since the task_id is dynamically assigned, which task_id do we pass through to the .set_upstream function to create the actual pipeline? The DAG doesn’t seem to run unless .set_upstream is set. Also, how would you recommend using **kwargs with the PythonOperator when dynamically creating these? Thanks for your time!