Member-only story
Automate the execution of BigQuery queries with Cloud Workflows
Resolve repetitive queries with scheduled invocations, combine simple queries to complex DML statements by using Cloud Workflows instead of BigQuery Scripting.
SQL pipelines! In this article, we will explore how to use Workflows Connectors to run BigQuery query pipelines to execute a query, retrieve query results, loop through the resultset, and repeatedly launch subsequent queries.
Also, you could reuse these techniques to various business logics to execute a SQL pipeline process that combines several consecutive steps.
Introduction
We are going to define a problem that currently is not supported out of the box by BigQuery and is frequently asked on Stackoverflow.
Is there a way to run scheduled DML dynamic queries on wildcard tables such as Firebase analytics dataset?
Often people want to clean up the dataset of their unused events to have smaller tables, lower storage costs.
By trying to execute something like:
