Understanding the Value of dbt™…In My Heart

A glimpse into the past to put the present into perspective.

Disclaimer: The company that I work for, EON Collective, are partners with dbt, but this is my personal journal that I will take with me throughout my career.

I always love talking with “old heads” about the struggles and challenges that I am facing, as they tend to give the best advice. However, there are those handful that no matter what you are dealing with, without fail, you would hear them say, “you don’t know how easy you got it” or the timeless classic, “back in my day…”. The moment I would hear the first syllable of the sequence I would think “Hear we go…I get it, your times were rough”. However, looking back, did I…do I really get it?

I’ve been working in tech for two years now, primarily with dbt and mostly on the engineering side rather than analytics. Since I’m relatively new to the industry, dbt has always been a part of my career and I don’t know a universe without it. It was actually my dbt skills that helped me score my first full-time gig in the tech world.

Basically, dbt is a programming suite for your database. You write your code, dbt compiles it into SQL, and then executes it on your database. You can connect models, create reusable functions, and run a range of data tests. Plus, you can even generate documentation and host a slick website. And as if that wasn’t enough, you can now also run Python models with dbt.

Awesome, dbt is capable of some seriously cool stuff. Over the past two years, I’ve had to get creative with macros and packages, migrate stored procedures and Talend jobs, and even whip up a couple of custom materializations. Despite all that, something still didn’t quite click for me. I suspect that’s because I don’t have any experience with what life was like before dbt came along.

However, I recently got a blast into the past, tasting what life is like without dbt. I was messing around with an idea and didn’t want to go through the hassle of setting up a formal process, so I had to do some dbt-esce tasks without actually using dbt… and it hit me like a ton of bricks — I finally get it now.

dbt is saving you from the nightmare of trying to manage data transformations.

If you try to do transformations without dbt, you’re setting yourself up for a disaster. It’s not that you’ll mess up the transformations as you’re doing them — usually, joining those five tables together makes perfect sense, as if handed down from the heavens.

The disaster happens later, when you need to go back and figure out what was done so you can do something else. You’ll find yourself asking questions like, why did I even combine those tables? Or, how did I make these tables in the first place? If you’re lucky, you will only have to comb through four of five different consoles of ad-hoc SQL code trying to understand the transformation that you are looking at 🥲.

For perspective — the project that inspired me to write this article was my own personal project and I had to create only 7 tables. But now, think about the situation where a team of developers is working globally, handling different systems with hundreds of tables.

At its core, when you strip away all the bells and whistles, the real value of dbt is similar to that of a cryptocurrency. It provides you with a ledger that tracks every single change made to your data at the transformation level, giving you full visibility. This visibility helps you answer some critical questions, such as:

• What exactly does this transformation do, and why does it even exist?
• What data is being used as input for this transformation?
• Who is the intended audience for this transformation, and how is it being utilized?
• Where can I locate the transformation code if I need to take a closer look or make some changes?
• Who was responsible for creating or modifying this transformation, when was it done, why was it necessary, and who authorized it?

If you can answer those questions easily then you can continue to build out your data platform with little problems. Making those answers easy to find WITH dbt still takes effort. Good luck getting them without. I knew why dbt was valuable in my head, but now its value feels fully integrated into my body.

You were right old head, you were right. You had it real bad.

Side Rant: In my opinion, I don’t think one generation has it worse than the other in the grand scheme of things. The “game” remains equally challenging, but the difficulty shifts from one area of life to another. Challenges present themselves in different ways. For example, in the past, you had to physically mail your resume, while nowadays it’s just a matter of one-click application. However, in today’s job market, your application faces global competition.

A3I Written (Accelerated & Augmented w/ Artificial Intelligence)