Managing BigQuery field descriptions — bq-utils 0.1.0
A small Python library that helps you with copying and uploading your BigQuery field descriptions.
We’ve been using BigQuery for a while now and its field description support is rather useful. You can view and edit the descriptions on the schema tab and once you hover over a field on the query pane the description pops up, which can improve your productivity by not having to search for it all the time. I mean who knows by heart all the fields and what they represent in the Google Analytics or Firebase export schemas? :)
The problem is that once you start using authorized views to manage table access or if you have several tables with almost identical fields, copying descriptions from one table to another “by hand” can easily become tedious. To speed up this cumbersome task, I wrote a small Python library (GitHub, PyPI) built on top of official BigQuery Python client library, which helps you with copying field descriptions and uploading them from a csv file. It works seamlessly with tables and views and with nested and repeated fields also so you can easily use it with complex schemas.
We use this simple library extensively with views and also to add field descriptions to the GA 360, and Firebase export tables. From the linked websites we copy pasted the field names and descriptions to Google Sheet that we then exported to csv format. Once we got rid of the header all we needed to do was running
pip install bq-utils and executing the following script:
After this you can easily set up an Airflow task that copies the descriptions every day from yesterday’s.
If you want to take advantage of authorized views and also have all your table descriptions just copy them like this:
I do hope that you find this library useful and if you have any suggestions on how to improve the current utilities or what to add next let me know.