Visualize Google Kubernetes Engine usage and cost metrics using Google Data Studio.

Anil Saravade
Aug 21 · 4 min read
Credit: Lukas/www.pexels.com

Google Data Studio is a visualization and reporting tool, and an integral part of Google Marketing Platform. It enables an enhanced approach to decision-making, allowing your entire organization to readily access and use marketing insights.

Data Studio lets you create beautiful, informative reports that are customizable and easy to understand and share. Empowered with the information they need, your teams can make better decisions at the exact moment they need to. With Data Studio, you can build reports and dashboards with just a few clicks.

GKE usage metering is helpful for scenarios like the following:

  • Tracking per-tenant resource requests and actual resource consumption in a multi tenant cluster where each tenant operates within a given Namespace.
  • Determining the resource consumption of a workload running in a given cluster, by assigning a unique label to the Kubernetes objects associated with the workload.
  • Identifying workloads whose resource requests differ significantly from their actual resource consumption so that you can more efficiently allocate resources for each workload.

This article will go through the process of setting up Google Data Studio to visualize Google Kubernetes Engine usage and cost metrics.

  • Prerequisites

Before you use usage metering, you must meet the following prerequisites:

1] Enable the BigQuery API for your project.

2] Create a Google Kubernetes Engine cluster v1.11.4-gke.8 or higher, v1.12.6-gke.8 or higher, or v1.13.0 or higher.

  • Setup Billing export for BigQuery in Google Kubernetes Engine.

1] In the BigQuery UI under your project, create a new dataset for the GKE usage metric.

2] Enable the GKE usage metering for your GKE cluster and provide the dataset which you’ve created earlier for GKE usage metric.

To verify that GKE metric is enable for your cluster run the below command in the Google cloud shell.

gcloud beta container clusters describe <cluster-name> \ --format="value(resourceUsageExportConfig)"
  • Setup Billing export for BigQuery in Billing account data.

1] In the BigQuery UI under your project, create a new dataset for the billing account data.

2] In the Billing section, go to the linked billing account and select your project.

2] Click on the Billing export section, select export data to BigQuery.

3] From the Project list, select the project where your BigQuery dataset which you have created earlier for billing account data.

Note: The table within the billing dataset will take up to 5 hours to appear after exporting to BigQuery and start to be populated. When the table appears, note its name.

It will be named something similar to

gcp_billing_export_v1_<var>BILLING_ACCOUNT_ID</var>

  • Setup the Data Studio Dashboard.

1] Enable Data Studio if it is not already enabled.

2] Create Data Source.

We need to create two separate data sources for creating a GKE usage metric dashboard.

— Replace PROJECT_ID with your project's ID (not the project name).

— Replace USAGE_METERING_DATASET with the dataset where your kubernetes cluster writes its usage metering.

— Replace BILLING_DATASET and BILLING_TABLE with the dataset and table where Cloud Billing data is exported.

1} Create a data source for BreakdownQuery.

In the Data Studio UI, create a new data source and select custom query and copy the below snippet.

SELECT
resource_usage.cluster_name,
resource_usage.cluster_location,
resource_usage.namespace,
resource_usage.labels,
resource_usage.resource_name,
resource_usage.sku_id,
resource_usage.start_time AS usage_start_time,
resource_usage.end_time AS usage_end_time,
resource_usage.usage.amount * gcp_billing_export.rate AS cost
FROM
`PROJECT_ID.USAGE_METERING_DATASET.gke_cluster_resource_usage` AS resource_usage
LEFT JOIN (
SELECT
sku.id AS sku_id,
SUM(cost) / SUM(usage.amount) AS rate,
MIN(usage_start_time) AS min_usage_start_time,
MAX(usage_end_time) AS max_usage_end_time
FROM
`PROJECT_ID.BILLING_DATASET.BILLING_TABLE`
WHERE
project.id = "PROJECT_ID"
GROUP BY
sku_id) AS gcp_billing_export
ON
resource_usage.sku_id = gcp_billing_export.sku_id
WHERE
resource_usage.start_time >= gcp_billing_export.min_usage_start_time
AND resource_usage.end_time <= gcp_billing_export.max_usage_end_time

Note: This above code joins the joins the billing data with the GKE usage metrics.

2} Create a data source for GKEProjectCostFromGCP.

In the Data Studio UI, create a new data source and select custom query and copy the below snippet.

SELECT
*
FROM
`PROJECT_ID.BILLING_DATASET.BILLING_TABLE`
WHERE
project.id = "PROJECT_ID"

3] Create the Data Studio dashboard.

We created a dashboard you can copy into your project. When you copy the dashboard, you are prompted to select three data sources you just created.

  1. Click this dashboard link.
  2. Click the Copy button, in the top right of the screen.
  3. For the data sources, select the data sources that you created before. Use BreakdownQuery in the first field and GKEProjectCostFromGCP in the second.
  4. Click Create Report.

After the Google Data Studio report is created you can visualize the GKE usage based on different dimensions and analyze it.


Related links:

https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-usage-metering#dashboard-animation

Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on medium or LinkedIn.

Searce Engineering

We identify better ways of doing things!

Anil Saravade

Written by

Cloud Engineer @Searce | AWS & GCP Certified | DevOps solutions | Kubernetes & Docker | ML | Former AWS Employee

Searce Engineering

We identify better ways of doing things!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade