BigQuery partitioning with Beam streams

Alex Van Boxel
Jun 1, 2017 · 4 min read

This article assumes that your are knowledgable around BigQuery and Apache Beam (Cloud Dataflow)

Partitioned table

https://unsplash.com/photos/24tsXm7qGQE
WHERE
_PARTITIONTIME
BETWEEN
TIMESTAMP("2017-06-01")
AND
TIMESTAMP("2017-06-14")

Inserting Data

Auto Partitioned Streaming

Get a fixed window, and supply the TableReference function to the BigQuery writer.
https://gist.github.com/alexvanboxel/0f38ceb5ccebbccebcd759576be428c2

max = 2017–01–01T23:59:59.99999 >>> tablename$20170101


Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Thanks to Wout Scheepers.

Alex Van Boxel

Written by

Google Developer Expert for Cloud

Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.