Member-only story

How to Build a Simple Data Stack on BigQuery

Building data infrastructure on BigQuery is a highly capable, escalable, extensible, accessible and future-proof choice for any team or organisation, at any scale

Jim Barlow
Decode Data
17 min readJun 5, 2024

--

Nice Stack. Photo by Mae Mu on Unsplash

🔎 I am the Lead Engineer at Transformation Flow, a boutique Google Cloud Data and Analytics Engineering consultancy specialising in BigQuery and adjacent technologies. Get in touch at hello@transformationflow.io if you need our advice and help to architect, build, extend, manage and monitor anything discussed in this article, or in your wider BigQuery-centred world. 🔎

Introduction

In the data field, we are always striving for simplicity. Simple solutions to complex problems; toolsets which are simple to install, integrate and operate; simple transformations which are easy to understand, update and debug.

However, in the context of the ‘Modern Data Stack’, selecting the tools to use and the ways to use them is anything but simple. The promise that we could have best-of-breed, ideally open source tooling for each data activity held some allure, especially against the backdrop of tired, expensive, monolithic all-in-one data platforms of old. However, the reality of selecting, integrating and managing all of these…

--

--

Decode Data
Decode Data

Published in Decode Data

Automated data pre-modelling and augmentation in Google BigQuery

Jim Barlow
Jim Barlow

Written by Jim Barlow

Senior Data Engineer @ Data to Value

No responses yet