Go From Staging Table To Production Data With 1 Stupid Simple SQL Line

A risk-averse approach to “flipping the switch” from test tables to production tables featuring a subtle BigQuery SQL function.

Currently job searching? Give yourself an edge by developing a personal project using my free 5-page project ideation guide.

Data engineering tutorials don’t talk enough about the s-word. Not the four-letter curse you reflexively blurt out when your pull request breaks production. Or what you mutter when your schema doesn’t match the target table.

I’m talking about staging.

Because for all the questions I get from stakeholders that want to know: “When will (data source) be available in production?” I stop them and say, “Well, first it will need to load to our staging tables.”

And, at the risk of over-clarifying, a staging environment is not just a sandbox where you throw your spaghetti code to see what sticks. It’s a data payload’s last stop before we “flip the switch” and point our pipelines at production end points, tables and dashboards.

Ideally, your staging environment will mimic your production table and still maintain high data engineering standards like:

• Clustering
• Partitioning
• De-duplicated