Airflow 2.5 brings valuable new features — including improvements to Airflow’s dynamic task mapping and data-dependent scheduling features — and delivers noteworthy performance improvements, along with fixing a lot of bugs. This is the third significant release of Airflow this year, shipping just three months after Airflow 2.4. A faster…

When Airflow was first open-sourced by Airbnb in 2015, it took the data engineering community by storm, quickly becoming one of the most viral, widely-adopted open-source projects in the world. At over 10 million pip installs a month, 28,000 Github stars, and approaching a Slack community of 30,000 users, it…

As data becomes more democratized and increasingly essential to business decisions, failures or delays in a data pipeline become more disruptive. But when a pipeline fails in Airflow, it’s often hard for data engineers to know where and when the failure has occurred. As a result, they spend critical time…

Lineage is becoming an essential tool for data teams operating within distributed data ecosystems. …

As data teams work to discover and deliver insights, they use a large variety of data sources, such as applications, databases, and files. In the process, they usually have to transform some or all of the data so that it can be aggregated in a consistent form. After this, the…

Data-driven scheduling, a new feature in Airflow 2.4, equips Airflow with the built-in logic it needs to run and manage different kinds of data-driven dependencies, through a new object type called Dataset. Basically, a Dataset is any output from one or more upstream (“producer”) tasks that is used as a…

How do you know when you can trust your data? Your dashboards? The predictions from your machine learning (ML) models? Data confidence starts with data quality checks running continuously. As you might expect, at Astronomer we use Airflow — powered by Astro — as the heart of our data platform…

When Airflow 2.3 dropped just over four months ago, we called it one of the most important-ever releases of Apache Airflow, thanks mainly to its introduction of dynamic task mapping. The same can be said of today’s Airflow 2.4 release, which introduces a “datasets” feature that augments Airflow with powerful…

Whether it’s Cloud or Core, using Airflow and dbt together makes life better for everyone. Learn about all the ways they can be combined, as well as a new dbt Cloud provider. The best way to describe the relationship between Airflow and dbt might be spiritual alignment. Both tools exist…