How Data Engineers Can Use Python to Schedule BigQuery Queries

BigQuery provides guidance for using Python to schedule queries from a service account but does not emphasize why this is an important, if not overlooked step of automating and sustaining a data pipeline.

Service accounts are preferable to personal accounts because service accounts can be accessed by anyone on the team with the corresponding IAM role, meaning that even if someone in the organization leaves, their work can still be accessed, edited and scheduled with ease.

--

--

--

Offering original and aggregated data engineering content for working and aspiring data professionals. Content posted here generally falls into one of three categories: Technical tutorials, industry news and visualization projects fueled by data engineering.

Recommended from Medium

Command Line Modules to Ansible Playbooks

How to Study for Data-Structures and Algorithms Interviews at FAANG

Kind of a Newsletter: New Horizons

Real-Time Private Chat Application using PHP Ratchet Websockets

Tunneling Through the Clouds

Tableau: LOD play with Dates — Part 1

Creating a Free Universal Visual SSML Editor🤖

Which types of state should be placed in my NGRX store?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Zach Quinn

Zach Quinn

DE @ Forbes. Pipeline: A Data Engineering Resource. Editor: Learning SQL. Opinions are my own.

More from Medium

How to Use Python & SQL to Append New Rows to a BigQuery Table Without Overwriting Your Data

A python, coiled.

Upload Data to BigQuery via Python

How to handle iframe in Selenium (Python)

Data Lake vs. Data Warehouse: What are the differences?