Slack notifications for Google Cloud Billing

Photo by Fancycrave on Unsplash

Google Cloud Platform gives you easy access to computing resources and services that you can use to build and scale your business. These come at a cost and it is important to actively monitor how much you are spending to keep your costs under control. You can do this is by using Google Cloud Billing alerts. We are going to show how to use these alerts to send your current budget’s spending to a Slack channel via Slack notifications. This is a great way to help you stick to your budget and allows you to take action when your budget is out of control.

Budget alerts can be applied to either a billing account or a project. Alerts can be set at a specific amount or matched to the previous month’s spend. The alerts will be sent when spending exceeds a percentage of your budget. Alerts can be configured to send an email or publish an event to a Cloud Pub/Sub. Emails are sent to billing administrators and billing account users, once the threshold is reached. The email is a standard template that cannot be customised. While for the Cloud Pub/Sub, an event is published to a topic at regular intervals of around 20 mins. This data in the event contains billing account id, current threshold limit reached, budget limit and cost accrued so far for the month.

The Cloud Pub/Sub option gives greater control over how the budget alert is consumed. For example, you can subscribe to the billing alert’s topic to create custom email templates, log the data or send a slack notification. The one drawback with publishing alerts to Cloud Pub/Sub is that events are regularly triggered and you may receive a message multiple times as Cloud Pub/Sub only guarantees at-least-once delivery. If you naively send every event to users, around one every 20 minutes, you will quickly inundate them with alerts. For our Slack notifications, we will look at a simple solution using BigQuery to only trigger alerts once per budget threshold.

Architecture

The architecture for the solution is shown below in Figure 1. Billing Alerts raise events that are pushed to a Cloud Pub/Sub topic. A Cloud Function subscribes to the topic and processes the billing alert event. This function saves the data to a BigQuery table and then sends a Slack notification if needed.

Figure 1: Architecture

We have also included the deployment pipeline, using Cloud Build to automatically deploy the code changes from GitHub.

Setup

Before we write the code for our notifications, we first need to setup the billing alerts, create a BigQuery table and setup an access token for Slack.

  1. Create a new Cloud Pub/Sub topic called billing-alerts. This will topic will be used for the billing alert publisher.
  2. Navigate to Billing in the Google Cloud Console and create your budget. When creating your budget in the section “Manage notifications” configure the Cloud Pub/Sub topic billing-alerts that was created in the previous step¹.
  3. In BigQuery, add a new dataset called billing and create a new table called budget. The schema for the table is shown below:
[
{
“name”: “createdAt”,
“type”: “TIMESTAMP”,
“mode”: “REQUIRED”
},
{
“name”: “costAmount”,
“type”: “NUMERIC”,
“mode”: “REQUIRED”
},
{
“name”: “budgetAmount”,
“type”: “NUMERIC”,
“mode”: “REQUIRED”
},
{
“name”: “budgetName”,
“type”: “STRING”,
“mode”: “REQUIRED”
},
{
“name”: “threshold”,
“type”: “NUMERIC”,
“mode”: “REQUIRED”
}

4. Create a new Slack application and set up an access token: refer to https://api.slack.com/slack-apps. Keep the Bot User OAuth Access Token handy as you will need it below to configure access from our Cloud Function to Slack.

Figure 2: Slack Application OAuth & Permissions

Sending Notifications

We use a cloud function to subscribe to the Billing Alerts Cloud Pub/Sub topic, write the event data to BigQuery and send a notification.

Navigate to Cloud Function in the Google Cloud Console and create a new cloud function with the following values:

  • Name: slack-billing-notification
  • Trigger: Cloud Pub/Sub
  • Topic: billing-alerts
  • Source code: inline editor
  • Runtime: Node.js 8
  • Function to execute: notifySlack

Before adding code for our function, we will setup the BOTTOKEN environment variable that contains Bot User OAuth Access Token from configuring the Slack Application from Step 4 in the Setup section. In the Advanced Options, add an environment variable for the BOTTOKEN. Copy the value of the Bot User OAuth Access Token.

In the package.json file, add the following dependencies:

{
"name": "slack-billing-notifications",
"version": "0.0.1",
"dependencies": {
"slack": "^11.0.1",
"@google-cloud/bigquery": "^2.0.6",
"@google-cloud/pubsub": "^0.18.0"
}
}

For index.js, enter the following code to create the of our notification function:

const slack = require('slack');
const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();
const CHANNEL = 'gcp-notifications';
const DATASET = 'billing'
const TABLE = 'budget';
const PROJECT = process.env.GCP_PROJECT;
const DATASET_LOCATION = 'australia-southeast1';
const BOT_ACCESS_TOKEN = process.env.BOTTOKEN;
exports.notifySlack = async (data, context) => { 
// code to send notification to slack
};

After we import the libraries for Slack and BigQuery, we define the constants for the function. Notably we define PROJECT to be the inherited GCP_PROJECT environment variable and BOTTOKEN to be the eponymous environment variable defined for our Cloud Function above.

Let’s flesh out the implementation details of our notifySlack function. The notifySlack function is configured to be called when our Cloud Pub/Sub topic receives an event. The event data is passed in the first argument as a Base64 encoded JSON string. We parse this data and perform some basic formatting:

 const pubsubMessage = data;
const pubsubData = JSON.parse(Buffer.from(pubsubMessage.data, 'base64').toString());
const formatter = new Intl.NumberFormat('en-US', {style: 'currency', currency: 'USD', minimumFractionDigits: 2})
const budgetId = pubsubMessage.attributes.budgetId;
const costAmount = formatter.format(pubsubData.costAmount);
const budgetAmount = formatter.format(pubsubData.budgetAmount);
const budgetName = pubsubData.budgetDisplayName;
const createdAt = new Date().toISOString();
let threshold = (pubsubData.alertThresholdExceeded*100).toFixed(0);

if (!isFinite(threshold)){
threshold = 0;
}

Save this parsed data to our BigQuery table:

 const rows = [{createdAt: createdAt, 
costAmount: pubsubData.costAmount,
budgetAmount:pubsubData.budgetAmount,
budgetId: budgetId,
budgetName: budgetName,
threshold: threshold}]
await bigquery.dataset(DATASET).table(TABLE).insert(rows);

Once we have saved our data, we then check if we have any other data for this budget threshold. We count the number of events that have been saved for the current month, matching the budget and threshold. If this count is only 1, then this is the first event for the threshold for the month and we need to send a notification, otherwise we can stop processing and exit the function.

const query = `SELECT count(*) cnt
FROM \`${PROJECT}.${DATASET}.${TABLE}\`
WHERE createdAt > TIMESTAMP( DATE(EXTRACT(YEAR FROM CURRENT_DATE()) , EXTRACT(MONTH FROM CURRENT_DATE()), 1))
AND Threshold = ${threshold}
AND BudgetId = '${budgetId}'
`;
const options = {
query: query,
location: DATASET_LOCATION,
};
const [results]  = await bigquery.query(options);

if (results.length > 0 && results[0].cnt > 1 ){
return;
}

The last block of code posts a simple message to Slack. We add a 🔥 emoticon if 90% of the budget has already been used. Even though the composition of our message is relatively simple, Slack notifications supports a rich set of features².

const emoticon = threshold >= 90 ? ':fire:' : '';
notification = `${emoticon} ${budgetName} \nThis is an automated notification to inform you that your billing account has exceeded ${threshold}% of the monthly budget of ${budgetAmount}.\nThe billing account has accrued ${costAmount} in costs so far for the month.`
const res = await slack.chat.postMessage({
token: BOT_ACCESS_TOKEN,
channel: CHANNEL,
text: notification
});

The completed code listing is available on GitHub.

Testing

Once we have saved index.js, we can test our function. In our Cloud Function, select the Testing tab. This displays an input where we can enter a JSON string to test our function.

We will test our function by triggering a 50% budget threshold alert. First we need to Base64³ encode our test data:

{
“costAmount”: 501,
“budgetAmount”: 1000,
“budgetDisplayName”: “test”,
“alertThresholdExceeded”: 0.50
}

Once the above data is encoded we can create the JSON to send to our function:

{
“data”: “ew0KCQkiY29zdEFtb3VudCI6IDUwMSwNCgkJImJ1ZGdldEFtb3VudCI6IDEwMDAsDQoJCSJidWRnZXREaXNwbGF5TmFtZSI6ICJ0ZXN0IiwNCgkJImFsZXJ0VGhyZXNob2xkRXhjZWVkZWQiOiAwLjUwDQoJfQ==”,
“attributes”: {
“budgetId”: “test”
}
}

Note the value for the data is our Base64 encoded test data. Click the Test the Function button. This will execute the Cloud Function with the test data. If successful, you will see a Slack notification appear in your Slack channel:

Figure 3: Slack notification for test data

If the notification does not appear, review the Stackdriver logs to debug any issues.

Deployment

Up to this point, our code has been manually deployed. This is less than ideal as we need to remember to deploy any changes that we make in our code repository. It would be better to automatically deploy our code when it changes in GitHub. We will use Cloud Build to achieve this.

For Cloud Build, we need to create a configuration file that will define the steps that need to be executed when the code in the repository is updated. For our Cloud Function, we just need the following steps:

  1. Run npm install,
  2. Run npm test, and
  3. Deploy our cloud function.

The configuration file, CloudBuild.yaml, for these steps is:

# CloudBuild.yaml
steps:
- name: 'gcr.io/cloud-builders/npm'
args: ['install']
- name: 'gcr.io/cloud-builders/npm'
args: ['test']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['beta',
'functions',
'deploy',
'slack-billing-notification',
'--entry-point=notifySlack',
'--source=.',
'--runtime=nodejs8',
'--trigger-topic=billing-alert']

With our Cloud Builder configuration file, we can now create a trigger to deploy our code when it changes in GitHub. In the Google Cloud Console, select Cloud Build > Triggers. Click the Add Trigger button to display the first step for creating a trigger.

Figure 4: Selecting a source for Cloud Build trigger

Since we are storing our code in the GitHub repository momenton/gcp-billing-notifications, we select GitHub as the source and select the checkbox for the consent. Google with authenticate with the repository service and display a list of repositories for your user.

Figure 5: Selecting a repository for Cloud Build trigger

Select the code repository and click the Continue button to display the trigger settings. In Build Configuration, select the Cloud Build configuration file option.

Figure 6: Selecting build configuration for Cloud Build trigger

Click the Create Trigger button to finish the process.

We can test that the trigger is working by pushing a commit to GitHub and then navigating to Cloud Build -> Build History. If successful you will see the build details displayed, see Figure 7 below.

Figure 7: Cloud Build history

Summary

We have shown how to use Billing Alerts to send your current budget’s spending to a Slack channel via Slack notifications, using BigQuery to only trigger alerts once per budget threshold.

We can extend this example to take actions to enforce cap costs and stops usage for a Google Cloud project by disabling billing⁴. This will cause all Google Cloud services to terminate non-free tier services for the project.

Notes:

¹: To be able to create billing alerts, you must be a billing administrator. For information about billing administrators and billing permissions, see Overview of Access Control.

²: Check out https://api.slack.com/messaging/composing for an overview of what is possible for composing Slack messages.

³: You can easily Base64 encode this data using one of the online encoding sites, such as https://www.base64encode.net/.

⁴: It is recommend that if you have a hard funds limit, you set your budget below your available funds to account for delays. Due to usage latency from the time that a resource is used, to the time that the activity is billed, some additional costs might be incurred for usage that hasn’t arrived at the time the all services are stopped.