Google Cloud — Export Billing Data to BigQuery
FinOps is an evolving cloud financial management discipline and cultural practice that enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions. — FinOps Foundation Technical Advisory Council
In other words, FinOps is a practice where users take ownerships of their cloud consumption, manage their costs accordingly, and think about how their cloud spend affects business value. This involves a combination of education, visibility, and shared best practices. For Google Cloud customers, one piece of a FinOps strategy is to export billing data into BigQuery to make costs visible across the organization.
The Google Cloud billing reports helps you monitor, optimize, and analyze your Google cloud costs. You can use it to review your cost history, cost trends, and forecasted costs from the Google console. However, for more granular analysis, use “Exporting your billing data into BigQuery”. Exporting your billing data into BigQuery allows you to slice and dice your billing data using the querying capabilities of BigQuery and offers the most customizable form of analysis. You can fine tune your analysis using advanced queries and then visualize your results using Data Studio or Looker. The recommendation from Google Cloud is to setup the billing export when you create the billing account.
Benefits:
- Cost analysis beyond the default billing reports dashboard.
- Create custom queries and dashboards.
- Export data to visualization tools or csv files.
- Part of a FinOps strategy.
Setup Billing Export
- Enable the BigQuery Data Transfer Service API
2. Create a BigQuery dataset to manage all your billing data. The Google recommendation is to use a multi-region option since some features such as detailed usage cost data is only supported in multi-region location. Click the 3 dots next to the project ID and select “Create Dataset”.
To create the dataset:
- Enter a dataset ID. This is user defined.
- Select a data location. The Google recommendation is multi-region.
- Ensure table expiration option is cleared.
- Ensure Use a customer-managed encryption key (CMEK) option is cleared.
You should see the create dataset under the project ID in BigQuery.
3. Enable cloud billing export to the BigQuery dataset from the billing menu. Select one of Standard usage cost or Detailed usage cost. The detailed usage cost provides the same information as the standard usage cost along with additional fields that provide resource-level data.
In the billing export configuration, select the project where the BigQuery lives and select the dataset you created.
The Biling export page will show which items are enabled and exported. These settings can be changed by enabling/disabling the options. You can also export pricing, which is pricing data applicable to your cloud billing account.
After a few minutes, billing data tables are automatically created in the BigQuery dataset. Do NOT modify any of these data tables. Depending on the type of exported data, it could take up to 48 hours to start seeing data.
Google Cloud provides several sample queries you can use to analyze your billing data.
Visualization
Since the billing data is in BigQuery, you can use any visualization tool that is compatible with BigQuery. Two recommended tools from Google Cloud are Looker and Data Studio. Looker is feature rich and the more flexible of the two options. If you need quick and free, then try Data Studio. Data Studio also has a sample dashboard you can copy for quick visualization.
Visualize Billing Data with Data Studio
On the left hand side, select “(READ ME)”. This shows instructions on how to copy the dashboard and change the data source to the BigQuery table you created.
TIP: To easily find the fully qualified table name, create a query in BigQuery
When you reconnect and review the confirmation, you should see the following pop-up screen. No fields should change.
When complete, data studio will visualize your data using the pre-built queries.
Further Watching
Further Reading
Google Cloud: Export Cloud Billing daa to BigQuery
Google Cloud: Understand BigQuery Cloud Billing Data Tables
Google Cloud: Example Queries for Cloud Billing Data Export
Google Cloud: Labelling and grouping your Google Cloud Platform Resources
Google Cloud: Visualize spend over time with Google Data Studio
Google Cloud: Optimizing your Google Cloud spend with BigQuery and Looker
Medium: Visualize GCP Billing using BigQuery and Data Studio (2017) FinOps Foundation: What is FinOps?
