A Guide to Functional Data Engineering in BigQuery

An exploration of the Functional Data Engineering paradigm and how to implement this in BigQuery

Data goes in, data comes out. Just don’t forget your date-partitioning grommet. Photo by rivage on Unsplash

Introduction

What is functional data engineering?

Functional Data Engineering is an approach to working with data which mirrors functional concepts in Mathematics and Software Engineering.

Maxime Beauchemin, the founder of Apache Airflow and Apache Superset outlines the objectives and the practical application of functional engineering principles to data in his 2018 publication Functional Data Engineering — a modern paradigm for batch data processing, also presented here.

Functional Data Engineering — a modern paradigm for batch data processing

By defining a clear set of objectives and constraints, this approach aims to achieve clarity, reproducibility, stability, reliability and traceability in data. Practically, this can simplify tooling, development, testing and automation, and help build a scalable and efficient foundation for data-related operations for any organisation…