Google Pub/Sub to BigQuery the Simple Way

A hands-on guide to implementing BigQuery Subscriptions in Pub/Sub for simple message and streaming ingestion

Google’s latest planet-scale data warehouse subscription-based streaming ingestion water-borne military capability: BigSub. In this case, the Pub never made it to General Availability, so you will have to get your pints elsewhere. Photo by Thomas Haas on Unsplash

Motivation

I have encountered many situations in the past where I wanted to get Pub/Sub messages into a BigQuery table, but I never managed to find a particularly simple way of doing this.

You could set up a dataflow pipeline, but this requires additional infrastructure to understand, configure, manage and debug. Plus Dataflow (which is a managed Apache Beam service) is designed for high-throughput streaming, so always seemed like overkill for a simple message logging or monitoring system.

And it’s Java. But Python 😀! And Java… 😫!

public static string args void main... public static string args void main... public static string args void main... public static string args void main... public static string args void main... arrrrrrrrrrrrgh

Sorry, I still get flashbacks from my first attempts to learn to code (last century) in Java. Please do not attempt to use that code snippet … step away from the code snippet.

I then stumbled upon this, which — although promising simplicity — seems to be even more complicated than the previous method…