How to trick Airflow to reduce wasteful processing?
At Nextdoor, the data team uses Airflow to orchestrate data transfer between various data sources and data lake, our central data store. In this post, I will talk about one particular challenge we faced when we started migrating to Airflow…