VALIDIO
Published in

VALIDIO

dbt and the Analytics Engineer — what’s the hype about?

If you work in the world of data, you have at this point heard a lot of talk about the Modern Data Stack. It has gained a lot of buzz and attention as companies have begun a fundamental shift in how they think about analytics and machine learning. The Modern Data Stack is built on the new cloud-native technologies that have emerged in the last decade that are fast, reliable, scalable, and, most importantly, accessible everywhere. Some of the technologies that have made this possible are massively parallel processing (MPP) cloud data warehouses like Redshift, Snowflake, and BigQuery; ingestion tools like Stitch, Airbyte and Fivetran that have improved reliability and connector coverage; and analytics platforms like ThoughtSpot that enhance users’ experience when finding and sharing data insights, and make that data accessible to everyone, everywhere.

The rise of Fishtown Analytics and the dbt community

Fishtown Analytics’ data build tool (known as dbt) has made pretty sizable waves in the analytics and data space during the past few years. Fishtown Analytics (today dbt Labs) pioneered the practice of analytics engineering, built the primary tool in any analytics engineering toolbox (dbt) and has seen a fantastic community emerge around the analytics engineering workflow. Today there are over 5,000 companies using dbt; there are 13k+ data professionals in the dbt Slack community; and 800+ companies are paying for dbt Cloud.

Image source dbt Labs

The rise of dbt and the analytics engineer

In short, dbt or data build tool is a full-stack data tool that allows data analysts and analytics engineers to own the “Transform” step in the Extract, Load, Transform (ELT) pipeline. Described as the “T in ELT”, dbt lets you write SQL models that can be executed in order (specifically a DAG). It’s also open source and has developed a thriving community around it, proven by the numbers mentioned earlier. The main drivers behind this technology are the rise of cloud-based data warehouses and the SQL knowledge available on the market.

Image source GitHub/Fishtown Analytics
  • Data Analysts: partner with business stakeholders to answer questions with data, build dashboards and reporting, and carry out exploratory analysis.
  • Data Scientists: use e.g. statistics and machine learning to explore and extract value from data: solving optimization problems, building prediction models, running A/B experiments, etc.
Image source Striim
  • Tools like dbt enable analysts with SQL-based workflows and give them the picks and shovels to work like software engineers do.
  • Data consumers have come to simply expect good and usable data at their fingertips at all times.
  • Organizations (large and small) are recognizing the need to invest more resources in data modeling.

What’s next for dbt and the Analytics Engineer

We at Validio see increasing adoption levels of dbt and a thriving future for analytics engineering, just like with data engineering, which was the fastest-growing job in tech in 2020.

Image source KDNuggets
Image source KDNuggets

Final thoughts

The rise of the Modern Data Stack has transformed the data industry in a fast pace. A lot of job titles like analytics engineer, operations analyst were virtually non-existent up until a few years back. The fact is that the role of the Analytics Engineer is still very much in its early innings (like almost all parts in the Modern Data Stack). As Spotify's Peter Gilks & Paul Glenn discuss in their Medium post “Analytics Engineering at Spotify— while it may sound like they have analytics engineering all figured out at Spotify, one of the most exciting parts of the job is that they are in fact constantly evolving the definition of the role — something that seems to be happening all over the industry. With every new blog post that comes out, whether from Michael Kaminsky, Fishtown Analytics, Netflix or the many others who have written on this topic, they see both echoes of the journey Spotify has already taken and ideas for new opportunities they can take on.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Oliver Molander

Co-founder at Validio and early-stage tech investor at J12 Ventures. Preaching about the realities & possibilities of Data & ML.