VALIDIO
Published in

VALIDIO

dbt and the Analytics Engineer — what’s the hype about?

If you work in the world of data, you have at this point heard a lot of talk about the Modern Data Stack. It has gained a lot of buzz and attention as companies have begun a fundamental shift in how they think about analytics and machine learning. The Modern Data Stack is built on the new cloud-native technologies that have emerged in the last decade that are fast, reliable, scalable, and, most importantly, accessible everywhere. Some of the technologies that have made this possible are massively parallel processing (MPP) cloud data warehouses like Redshift, Snowflake, and BigQuery; ingestion tools like Stitch, Airbyte and Fivetran that have improved reliability and connector coverage; and analytics platforms like ThoughtSpot that enhance users’ experience when finding and sharing data insights, and make that data accessible to everyone, everywhere.

The rise of Fishtown Analytics and the dbt community

Image source dbt Labs

The rise of dbt and the analytics engineer

Image source GitHub/Fishtown Analytics
  • Data Engineers: manage core data infrastructure, ensuring data is available and accessible across the organization at the expected time at the expected quality.
  • Data Analysts: partner with business stakeholders to answer questions with data, build dashboards and reporting, and carry out exploratory analysis.
  • Data Scientists: use e.g. statistics and machine learning to explore and extract value from data: solving optimization problems, building prediction models, running A/B experiments, etc.
Image source Striim
  • The shift from ETL to ELT means people who know SQL have the power to transform and clean data once it arrives in the warehouse. Cloud data warehouses are powerful enough to handle the transformation workload.
  • Tools like dbt enable analysts with SQL-based workflows and give them the picks and shovels to work like software engineers do.
  • Data consumers have come to simply expect good and usable data at their fingertips at all times.
  • Organizations (large and small) are recognizing the need to invest more resources in data modeling.

What’s next for dbt and the Analytics Engineer

Image source KDNuggets
Image source KDNuggets

Final thoughts

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Oliver Molander

Co-founder at Validio and early-stage tech investor at J12 Ventures. Preaching about the realities & possibilities of Data & ML.