In my previous posts:
I covered different topics that you might want to discuss, with your data engineering team, at the beginning of your journey with dbt.
In this article, I will discuss at a high level:
As you probably know dbt works with models DAGs (Directed acyclic graphs).
The question is, do you think this is enough to orchestrate your entire platform or some form of external orchestration is required? …
In my previous post:
Practical tips to get the best out of Data Build Tool (dbt) — Part 1
I explained different approaches to split functionalities in dbt projects while building a data platform and how to best organize dbt models.
In this article, I will discuss the following topics:
Dbt_project.yml contains your dbt project configuration. It can easily become an unreadable monster or be almost empty depending on how you are using dbt.
For each dbt model, it is possible to define the model configuration at the model level but when the number of models starts to grow it might become complicated to jump from one model to another to read these configurations.
For this reason, in Photobox, we adopted the dbt_project.yml file as the main configuration file. This can be integrated with the metadata defined on the top of each dbt model when needed. …
This series of articles is for those who already have a basic experience with dbt (Data Build Tool) and want to get the best out of it when building a data platform. If you want to read an introduction regarding how to build data pipelines using DBT, I suggest having a look at this article that I wrote a few months ago or the official DBT documentation.
dbt is undoubtedly great for performing ELT but, sometimes, is presented as a tool mostly designed for analysts. …