How to run Python code on your BigQuery table

Use a Python 3 Apache Beam pipeline

Lak Lakshmanan
Jun 6, 2019 · 2 min read

You can do lots of things in SQL, and SQL is undeniably convenient, but every once in a while, you will find yourself needing to run Python code on your BigQuery tables. If your data is small, you can use Pandas (and the BigQuery client library), but if your data is large, the best approach is to use Apache Beam and execute it in a serverless, autoscaled way with Cloud Dataflow.

Here’s the full code for the example in GitHub. It comes from our forthcoming book on BigQuery.

Python 3 Apache Beam + BigQuery

Here’s the key Beam code to read from BigQuery and write to BigQuery:

Essentially, we are running a query on a BigQuery table, running the Python method compute_fit, and writing the output to a BigQuery table.

This is my compute_fit method. As you can see, it’s just plain Python code:

Make sure to specify the Python packages that you need installed on the Dataflow workers in a requirements.txt:

Enjoy!

Google Cloud - Community

Google Cloud community articles and blogs

Lak Lakshmanan

Written by

Data Analytics & AI @ Google Cloud

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Lak Lakshmanan

Written by

Data Analytics & AI @ Google Cloud

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store