Automate the execution of BigQuery queries with Cloud Workflows

Márton Kodok
Google Cloud - Community
4 min readMay 18, 2021

--

Resolve repetitive queries with scheduled invocations, combine simple queries to complex DML statements by using Cloud Workflows instead of BigQuery Scripting.

Automate the execution of BigQuery queries with Cloud Workflows

SQL pipelines! In this article, we will explore how to use Workflows Connectors to run BigQuery query pipelines to execute a query, retrieve query results, loop through the resultset, and repeatedly launch subsequent queries.

Also, you could reuse these techniques to various business logics to execute a SQL pipeline process that combines several consecutive steps.

Introduction

We are going to define a problem that currently is not supported out of the box by BigQuery and is frequently asked on Stackoverflow.

Is there a way to run scheduled DML dynamic queries on wildcard tables such as Firebase analytics dataset?

Often people want to clean up the dataset of their unused events to have smaller tables, lower storage costs.

By trying to execute something like:

--

--

Márton Kodok
Google Cloud - Community

Speaker at conferences, a Google Developer Expert top user on Stackoverflow, software architect at REEA.net, co-founder IT Mures, life-long learner, mentor