How to Build a Simple Data Stack on BigQuery

Building data infrastructure on BigQuery is a highly capable, escalable, extensible, accessible and future-proof choice for any team or organisation, at any scale

Nice Stack. Photo by Mae Mu on Unsplash

🔎 I am the Lead Engineer at Transformation Flow, a boutique Google Cloud Data and Analytics Engineering consultancy specialising in BigQuery and adjacent technologies. Get in touch at hello@transformationflow.io if you need our advice and help to architect, build, extend, manage and monitor anything discussed in this article, or in your wider BigQuery-centred world. 🔎

Introduction

In the data field, we are always striving for simplicity. Simple solutions to complex problems; toolsets which are simple to install, integrate and operate; simple transformations which are easy to understand, update and debug.

However, in the context of the ‘Modern Data Stack’, selecting the tools to use and the ways to use them is anything but simple. The promise that we could have best-of-breed, ideally open source tooling for each data activity held some allure, especially against the backdrop of tired, expensive, monolithic all-in-one data platforms of old. However, the reality of selecting, integrating and managing all of these…