Visualising Google Cloud

By Alfred Wahlforss

Google Cloud Platform (GCP) provides an incredible foundation for building modern applications and services. In fact, it’s so incredible, King has decided to move its entire data warehouse to GCP. As with all new technologies, there are some missing parts — though Google is aware of these and is actively working with customers like us to improve their platform.

One of the most evident missing components in GCP currently is the lack of a clear overview of activity and consumption at the organisational level. For example, it is easy to see BigQuery slot usage for one project (on BigQuery flat-rate pricing), but there is no simple overview of total slot usage across all projects. The same goes for BigQuery storage, Cloud Storage, and many other GCP services. At King, we have over 500 GCP projects currently, and without a good organisational overview, it is difficult to know what is going on.

We decided to fix this problem by creating our own dashboards. These dashboards give us a clear overview of our GCP organisation. We wanted to know everything from how we use BigQuery slots, to the bytes we have stored in Cloud Storage. This data needed to be fresh, ideally not older than a couple of minutes.

Luckily, Google opens up most of its cloud via APIs — although sometimes a lot of APIs need to be called to obtain useful data. Connecting these multiple APIs together sometimes felt like putting the pieces of a jigsaw puzzle together.

To minimize mess and ease future operations, we decided to use GCP’s serverless offerings to build our dashboards. Serverless means that the cloud provider handles all of the configurations of the servers. We found this approach reduced maintenance, decreased complexity, and helped us launch faster.

Architecture diagram of the app

To aggregate data from Google’s APIs, we used the serverless Cloud Functions. Cloud Functions belongs to the new paradigm of computing called Functions as a Service, or FaaS. In essence, FaaS enables isolated functions that run in the cloud and only cost when they are called. Perfect for collecting data from APIs.

With App Engine, there is no need to upgrade or maintain any servers. Our standard approach, by contrast, requires spinning up virtual machines and configuring them. Virtual machines often require software upgrades on a regular basis. Configuration and upgrading software takes precious time away from our IT department. The fact that App Engine requires neither of these things is a major time saver for a small app like this.

Our Dashboards

Total storage in BigQuery. Numbers displayed do not reflect actual numbers.

Turning to the end product — the dashboards — the information in the dashboard shown above was extracted using three separate Cloud Functions. The first function fetches all projects that use BigQuery, which is a list of hundreds of projects. A second function gets all the datasets per individual project and finds all the tables contained in each dataset. The last Cloud Function fetches the bytes stored in each table and sums it for every project. Finally, after over 30,000 Google API calls, we know how much we are storing in BigQuery.

A similar process was repeated to get the data for each of the dashboards shown below.

Before making this app King did not know how many projects it had in Google Cloud.
We can use a fixed amount of BigQuery slots at any given moment. This shows our total usage.

When we launched the dashboards we noticed one immediate benefit — they acted as conversation starters. We discovered that there are some discussions that could only be properly had when everyone shares the same information.

This dashboard lets us monitor which projects are using BigQuery slots.

The table in the above dashboard was an example of a great conversation starter. It spurred discussions such as whether or not we should limit slot usage per project, what time to schedule jobs, and so on. Before we knew which projects were using slots it was impossible to have productive conversations on these topics.

When we visualised BigQuery storage as above, it became clear which projects were responsible for the storage costs. That started a conversation around exactly what we should store in BigQuery vs Cloud Storage vs not at all.

May or may not be a staged photo!

A further fun effect has been that people stop in new places in the office. They might walk past a screen and see something interesting. Suddenly a discussion is sparked between people who have never spoken before. Maybe that conversation leads to other topics and all of a sudden a dashboard starts a friendship.

Join the Kingdom — Jobs.King.com

Share this article!


Originally published at techblog.king.com on October 15, 2018.