Image for post
Image for post

In my previous posts:

Practical tips to get the best out of Data Build Tool (dbt) — Part 1

Practical tips to get the best out of Data Build Tool (dbt) — Part 2

I covered different topics that you might want to discuss, with your data engineering team, at the beginning of your journey with dbt.

In this article, I will discuss at a high level:

  • How to orchestrate data pipelines and dbt models.
  • Use dbt documentation and serve it to an entire organisation.

How to orchestrate data pipelines and dbt models

As you probably know dbt works with models DAGs (Directed acyclic graphs).
The question is, do you think this is enough to orchestrate your entire platform or some form of external orchestration is required? …


Image for post
Image for post

In my previous post:

Practical tips to get the best out of Data Build Tool (dbt) — Part 1
https://medium.com/photobox-technology-product-and-design/practical-tips-to-get-the-best-out-of-data-building-tool-dbt-part-1-8cfa21ef97c5

I explained different approaches to split functionalities in dbt projects while building a data platform and how to best organize dbt models.

In this article, I will discuss the following topics:

  • get the best out of your dbt_project.yml file.
  • dbt makes you rethink some aspects of traditional data warehousing.
  • dbt Macros usage.
  • wrap dbt.

Get the best out of your dbt_project.yml file

Dbt_project.yml contains your dbt project configuration. It can easily become an unreadable monster or be almost empty depending on how you are using dbt.

For each dbt model, it is possible to define the model configuration at the model level but when the number of models starts to grow it might become complicated to jump from one model to another to read these configurations.
For this reason, in Photobox, we adopted the dbt_project.yml file as the main configuration file. This can be integrated with the metadata defined on the top of each dbt model when needed. …


Image for post
Image for post

This series of articles is for those who already have a basic experience with dbt (Data Build Tool) and want to get the best out of it when building a data platform. If you want to read an introduction regarding how to build data pipelines using DBT, I suggest having a look at this article that I wrote a few months ago or the official DBT documentation.

dbt is undoubtedly great for performing ELT but, sometimes, is presented as a tool mostly designed for analysts. …

About

Stefano Solimito

Principal data engineer @ Photobox

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store