Effortlessly Visualize GCP Resources Hierarchy and Billing Data with Sunburst Diagrams
Introduction
The goal of this article is to explore the use of a Sunburst diagram as a way to combine GCP Billing data and Resources Hierarchy into a single, user-friendly visual representation.
By using this approach/representation, GCP customers will have the ability to effortlessly navigate their GCP resources hierarchy, while also gaining a clear understanding of the GCP spend (fees) associated with each resource.
This approach is particularly valuable for:
- GCP Org Admins who would like to easily keep an eye on their GCP landscape.
- GCP Billing Admin who would like to continually track GCP spend incurred by their GCP resources.
Below is an example of a sunburst showing a GCP Org with all its resources and their spend (billing data).
Concept Overview
Google Cloud Platform (GCP) resources are organized in a hierarchical structure, starting with the organization node. The hierarchy includes the following levels:
- Organization: The organization node is the top-level entity in GCP and represents an organization that has multiple projects and folders.
- Folders: Folders are optional and can be used to organize projects within an organization. Folders are used to manage access controls and sharing policies for a set of projects. Projects:
- A project is a container for all other resources in GCP, such as virtual machine instances, databases, storage buckets, and network resources. A project is unique within GCP and can contain multiple services, applications, and virtual machine instances. A GCP project have a 1:1 relationship/assignment to a GCP Billing Account.
If you have a large GCP footprint it can be challenging to navigate through such a hierarchy while keeping track of costs incured by resources on each project.
The idea is to use a sunburst diagram to better represent the GCP resources hierarchy combined with billing data. This will make it easy to visualize and navigate through it.
By using a sunburst diagram, the GCP resources hierarchy is represented by nested circles, where:
- the outermost circle represents GCP projects (the lowest level in the hierarchy)
- the innermost circle represents the GCP Organization node the highest level in the hierarchy
- each inner circle represents GCP folders (intermediate levels)
Below is an example of a GCP resources hierarchy represented using a sunburst diagram:
The innermost circle represents my GCP Org node (bougdal.de). The first level of nested circles represents the first level of folders (e.g Central IT, Dept A, etc…). The outermost circles represent GCP projects.
The sunburst drill-down feature allows you to explore the hierarchy of GCP resources by drilling down into nested levels of the hierarchy. For example, you can easily click on a specific part of a nested circle representing a GCP folder and you will be able to see all resources (sub-folders or projects) that are under that specific folder.
To return to a higher level in the hierarchy, you click on the center of the sunburst diagram. This will zoom out to reveal the previous level of the hierarchy.
By using GCP Billing data, we can aggregate GCP spend by projects and folders and display them as labels on top of the sunburst elements. This will allow GCP customers to easily see their GCP spend broken down and/or aggregated by projects or folders. The inner circle of the sunburst will show the total GCP spend of the customer organization (in GCP) since the start of the current month.
Technical Architecture
The diagram below shows how you could implement the concept described in the previous section.
The architecture shown in the diagram above makes good use of GCP serverless solutions. This makes it easy to implement and very cost-effective. It is composed of four main building blocks:
Billing Data
a) First and foremost, we configure a Billing export to BigQuery.
b) Once the billing data is exported to BigQuery, we configure a scheduled query that aggregates billing data (spend) per project for the current month. This query runs regularly (interval is flexible) and writes to a BigQuery table (e.g. billing_data_per_project). It also publishes an event to a pre-defined Pub/Sub topic.
c) A GCP Cloud Function is triggered by the event in b)
d) The Cloud Function reads the billing data from BigQuery (e.g. table billing_data_per_project) and builds a JSON file that contains GCP spend for all active GCP projects for the current month.
e) The Cloud Function writes the JSON file (e.g. billing.json) to a private GCP bucket.
Resources Hierarchy
This module is responsible for transforming the GCP resources hierarchy into a “flat one” and writing the result to a JSON file that will be used later on by the Sunburst diagram. It works as follows:
- A Cloud Scheduler cron job is scheduled to trigger an event on a regular basis. This event is pushed to a Pub/Sub topic.
- A Cloud Function is triggered each time an event is published to the Pub/Sub topic above.
- The Cloud Function uses the GCP Resource Manager’s APIs to read the GCP resources hierarchy starting from the GCP Org node and going down until the project(s) level. For each project, the function will read the corresponding billing data (spend) from the billing.json mentioned earlier. For folders, it aggregates the billing data of all child projects and sub-folders.
- The Cloud Function builds a JSON Object (e.g. resources.json) and saves it to a private Cloud Storage bucket.
Frontend
For simplicity, the frontend is a simple HTML page that is hosted in a private GCS bucket. All we need to configure it to use the resources.json file mentioned earlier.
A Cloud Run service is used to expose the webpage (front end). The GCP Identity Aware Proxy is used to make sure that only allowed individuals are able to access the frontend page (you don’t need everyone to access your resources hierarchy and billing data :) ).
The Cloud Run service is exposed through a HTTP (s)Global Load Balancer.
Service Accounts
All the interactions between the main components in the diagram above rely on Service Accounts following the least-privilege approach.
Note: the service account used to read the GCP Resources hierarchy is highly sensitive because it requires “read permissions” at the GCP Org level. For this purpose, I have created a custom GCP role called “Sunburst Admin” that has the following permissions:
The service account above is also used to read and update objects in Cloud Storage.
Demo
A demo of a sunburst diagram showing both the GCP resources hierarchy and billing data is available here.
Conclusion
A sunburst diagram can be used as a powerful representation of GCP resources and billing data. It makes it easy to visualize and navigate through GCP resources while keeping a close eye on your GCP spend. This is very useful for both Billing Admins or GCP Org Admins. It can also be made available to folders admin or team members to create awareness about GCP cost control.
The above concept is provided as “as-is”. It has been tested on my own GCP Org with a few folders and projects. You may need a large screen to have a readable sunburst diagram in case you have many (nested) folders and projects.
Feel free to get in touch if you are interested in further implementation details like the source code of the cloud functions, etc.
