Improved Storage Read API Quotas in Google BigQuery

How Google empowers it’s Data Warehouse even more

Photo by Jeroen den Otter on Unsplash

Good news regarding the BigQuery Read API, Google has now significantly improved it, so you can query more data here. This is of course great if you want to analyze data or e.g. also transfer to other systems, especially in times of Big Data is the processing of vast data amounts in an efficient and parallel way very important.

Normally, you have two options in BigQuery for accessing BigQuery data [1]:

• Using record based paginated access by using the tabledata.list or jobs.getQueryResults REST API methods. The BigQuery API provides structured row responses in a paginated fashion appropriate for small result sets.
• Alternatively, using the bulk data export with BigQuery extract jobs that let you export table data to Cloud Storage in a variety of file formats such as CSV or JSON.

With the BigQuery Storage Read API you have an alternative option along the advantage that when you use the Storage Read API, structured data is sent over the wire in a binary serialization format. This allows for additional parallelism among multiple consumers for a set of results [1]. Keep in mind that the Storage Read API don’t let you use managing BigQuery resources such as datasets, jobs, or…