A Guide to Functional Data Engineering in BigQuery
An exploration of the Functional Data Engineering paradigm and how to implement this in BigQuery
Introduction
What is functional data engineering?
Functional Data Engineering is an approach to working with data which mirrors functional concepts in Mathematics and Software Engineering.
Maxime Beauchemin, the founder of Apache Airflow and Apache Superset outlines the objectives and the practical application of functional engineering principles to data in his 2018 publication Functional Data Engineering — a modern paradigm for batch data processing, also presented here.
By defining a clear set of objectives and constraints, this approach aims to achieve clarity, reproducibility, stability, reliability and traceability in data. Practically, this can simplify tooling, development, testing and automation, and help build a scalable and efficient foundation for data-related operations for any organisation…
