Big Data in Google Cloud — Cost Monitoring (part III)
This is part three of three
In our last two posts, part I and part II, we’ve seen how we approached the cost issue in the migration of our BigData services to Google Cloud Platform (GCP). This was about how we instructed our internal users in terms of good practices in cost terms when dealing with BigQuery (BQ), how we used Budgets to have a rudimentary alerting system, and how we imported data from Billing and from Cloud Audit Logs (CAL) to BQ.
In this post, we are going to discuss our current state of affairs:
A fully functional cost monitoring system which can indicate things such as a single query that crossed some threshold or an increase in costs from one day to the other greater than a configurable percentage.
Cloud Monitoring
With the steps we showed in the mentioned previous posts, we’ve laid out the groundwork that allowed us to now have a full-fledged alerting system. Although it might sound complicated, the main idea is very very simple. It’s all about sending data points to Cloud Monitoring (CM).
So the questions that arise are:
- What are “data points”?
- How to send them to CM?
- How to set alerts?
What are “data points”?
According to Digital Ocean,
A data point is a single measurement of a single metric.
Okay, what do these “data points” really mean?
Let’s take the example from our “the costlier query of the day” policy:
Once a day, we automatically run a query in BQ which returns the value of the costlier query, something like:
SELECT ROUND(MAX(TotalCost), 2) FROM billing.bigquery_usage WHERE DATE(date) = DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY);
The single result of this query will be the value of the “data point” that we are about to send to CM. Besides the value, we still need a timestamp to attach to the data point (which in our example is the time we run the query).
How to send them to CM?
Just as an example, I implemented a simple Java utility class and its client to show how to send these data points to Cloud Monitoring. These can be found in my GitHub account:
First, check the MetricSender.java:
we need to create a TimeSeries object, lines 50–60. This object will wrap two other important other objects that we’ve already created, the Metric and the data points (in the example, just a single data point for each run).
Again according to Digital Ocean,
Time series data is a series of data points that represent changes over time. Most metrics are best represented by a time series because single data points often represent a value at a specific time and the resulting series of points is used to show changes over time.
With the TimeSeries created, we can create a request (CreateTimeSeriesRequest) and finally use the client (MetricServiceClient) to send it to CM.
One last thing before your code can send the “data points”, your service account should have the “Monitoring Metric Writer” role.
How to set alerts?
1. Go to Monitoring > Overview
If it is the first time you are using CM, you might have to wait a few moments for the Workspace to be created, as you can see in the picture below.
2. Check your metrics
Once the workspace is ready, the best first step is to take a look at your data.
Go to Metrics Explorer.
For Metric, type the name of the metric you chose, in our example, it was “custom.googleapis.com/bigquery/my-metric-name” and for the Resource Type, type “Generic Node”.
This is what I got testing with different values for the “data points” and also different labels.
Once the data points are in CM and you already got a grip of them, it is just about using the tool itself to create alerts, which, in our case, writes to a slack channel.
3. Creating notification channels
If you haven’t yet configured any “Notification channel”, then that should be your first task:
Go to “Alerting”
On the very top, click on “EDIT NOTIFICATION CHANNELS”
You’ll see a page with several distinct types of channels. Since we are using email and Slack, we are focusing on them.
For configuring emails as notification channel:
To add an email, click on the “ADD NEW”, a popup with be displayed:
Fill in the desired “Email Address” and “Display Name” and click “SAVE”.
For configuring Slack as notification channel:
On the Slack part, click on the “ADD NEW”
it will redirect to your own Slack account and ask for permission
Click on “Allow”, fill the few fields displayed and there you go.
4. Creating a policy
Now it is about creating a policy, Go to “Alerting”, click on the “Create Policy” button.
Click on “Add Condition”
This is exactly like the metrics explorer
Then configure the condition itself
Look at the possible configurations, their meaning is pretty straightforward to grasp. In the example, if the value is above 200 for at least 30 minutes, a notification will be sent to the previously configured notification channels.
This is an example of our costlier query policy
So, in this post, we tried to show you the main idea we used to create a cost monitoring system. Of course, this is a very flexible solution and you can think of alerting on whatever data we’ve been able to collect in our last post.
We also skipped the part on how to retrieve the data from BigQuery, as this is not the core of this series of posts, but if you have problems to overcome this part, please leave a comment and I will be happy to help.
This was the third and last post on this series about Cost Monitoring in Google Cloud. The first two can be found:
Big Data in Google Cloud — Cost Monitoring
Big Data in Google Cloud — Cost Monitoring (Part II)
If you have any questions, comments or constructive feedback, please leave them in the comments.
