dbt + Airflow = ❤

Building dbt-airflow: A Python package that integrates dbt and Airflow

Photo by Jr Korpa on Unsplash

In today’s dynamic and competitive landscape, businesses rely heavily on data-informed decisions. To achieve this, organisations require a robust data platform and quality data they can trust. This means having systems in place that not only collect and store data effectively but also ensure its accuracy and reliability.

Managing hundreds, or even thousands of data models is not always a straightforward process. Data teams often grapple with the challenge of efficiently building, testing, and maintaining their data assets. Here at Plum, we rely on data build tool (dbt), a CLI tool that enables teams to manage their data models effectively.

However, dbt presents a challenge due to its command-line nature. The question then arises: how can teams take advantage of dbt when deploying it in production-grade environments?

While dbt Cloud might seem like the obvious answer, not all teams or companies are prepared to invest money in this product. In such scenarios, an alternative approach is needed to orchestrate the execution of dbt models while preserving the dependencies between them.

Why did we invest on building our own integration