Using Cloud Workflows to load Cloud Storage files into BigQuery

Loading Data Into BigQuery From Cloud Storage by using Cloud Workflows.
Loading Data Into BigQuery From Cloud Storage by using Cloud Workflows.

Cloud Workflows

Cloud Workflows — orchestrate & integrate

What is Cloud Workflows?

  • Workflow orchestration (step engine as a service)
  • Integrate any Google Cloud API, SaaS API, or private APIs
  • Out of the box authentication support for Google Cloud products
  • Fully managed service — requires no infrastructure or capacity planning
  • Serverless with Pay-per-use pricing model
  • Declarative workflow language using YAML syntax
  • Developer friendly built-in error handling with retries

Problem definition

Workflow steps to load Cloud Storage files into BigQuery.
Workflow steps to load Cloud Storage files into BigQuery.
gs://bucket/folder/file_12_2020.csv
gs://bucket/folder/file_01_2021.csv

Subworkflow to read Cloud Storage objects

Looping an array with Cloud Workflows

Doing an intermediate step — Parsing and extracting a segment from the URI

Workflow to execute synchronous BQ query

Workflow to load Cloud Storage files into BigQuery

Deploying and executing Cloud Workflows

Wrap Up

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Márton Kodok

Speaker at conferences, a Google Developer Expert top user on Stackoverflow, software architect at REEA.net, co-founder IT Mures, life-long learner, mentor