Kill Unexpected Cloud Costs: Integrate GCP Budget Alerts with Slack, Microsoft Teams, Hangouts, and Webex Teams

tl;dr

Cost control in the cloud is hard — GCP Budget Alerts can help, but budget notifications are limited to email out of the box. Supercharge the effectiveness of your Budget Alerts by notifying your teams where they work, whether it’s Slack, Microsoft Teams, Hangouts, or Webex Teams. In this article I’ll show you how to do this at scale, and I’ll give you the instructions and code to get it up and running!

Scroll to the bottom of the article for step by step instructions on implementing this with Slack, Microsoft Teams, Hangouts Chat, and Webex Teams. All code provided can be accessed at https://github.com/wapfel20/GCPBudgetAlertIntegrator.

Introduction

As we all have heard, ignorance of the law is not a defense that will get you far in a courtroom. So why do we enable ignorance of cloud spend? It happens all the time — machines get spun up, scripts go rogue, engineers mistakenly over-provision, or demand for a service goes through the roof. Whether positive or negative, beneficial or harmful, your teams should have a finger on the pulse of their spend at all times and they should be made well aware of any significant deviations, so they can course correct accordingly.

Using GCP Budget Alerts to proactively notify teams when they exceed, or are expected to exceed, their budget is a great way to promote awareness and accountability for cloud spend at the individual team level.

Why Use GCP Budget Alerts?

GCP Budget Alerts can help foster a culture of financial accountability across distributed project teams. By forcing teams to identify budgets for their projects, Budget Alerts promote mindfulness about the cost of projects, innovation efforts, scope changes, and other investments. By providing alerts when those budget thresholds are exceeded, GCP Budget Alerts not only raise awareness of unexpected cost increases, they prompt immediate action to address them.

But out-of-the-box, GCP Budget Alerts only send notifications to individuals with “Billing Account Admin” and “Billing Account User” IAM roles, and those notifications are delivered via email. I can’t say for certain, but I’d be willing to bet that:

1. Emails are not the preferred medium for alerting in your organization.
2. Not everyone who should be notified about billing fluctuations currently has a “Billing” role.
3. You don’t intend to give those individuals access to your GCP Billing Console.

The good news is that you don’t have to — by using the following approach, you can ensure all appropriate team members are receiving budget alerts, in a place where they will be seen, without opening up unnecessary access to your Billing Console.

We do this by configuring Budget Alerts to leverage a Pub/Sub topic for publishing, allowing us to integrate those alerts into each teams’ preferred collaboration channel, such as Slack, Microsoft Teams, Hangouts Chat, or Webex Teams.

The General Approach

Regardless of which collaboration platform your organization or team is using, the process for integrating Budget Alerts is essentially the same:

Step 1: Analyze your current and projected Project spend

Step 2: Create a Project Budget and Define your Alerting thresholds in the Billing Console

Step 3: Configure your Budget Alerts to push to a Pub/Sub topic

Step 4: Create a webhook in your collaboration space (Slack, Microsoft Teams, Hangouts, Webex Teams, etc)

Step 5: Create a Cloud Function to push alerts from Pub/Sub to the collaboration space, via the webhook

Below you will find the exact steps and code for doing this for 4 of the most common collaboration platforms: Slack, MS Teams, Hangouts Chat, and Webex Teams.

Managing Alerts at Scale with Distributed Teams

As you can imagine, setting this up for all projects in your organization could quickly become a very cumbersome and complex thing to manage, especially if every team has a Pub/Sub topic and Cloud Function for every project in your organization… that could result in thousands of extra cloud resources!

To simplify the overhead of this solution at scale, I recommend using the following approach for deploying at scale:

1. Build your strategy around “teams”
2. Define clear Naming Standards for Budgets, Pub/Sub Topics, and Functions to enable reusability

Building your strategy around “teams”: Each team in your organization will likely have a set of GCP projects they own and manage. For instance, the “eComm Team” may have separate Dev, Stage, and Prod GCP Projects they manage that are associated with the eCommerce Platform. They will also likely have a common platform for collaborating.

For each team, establish the following components to be reused across the team’s projects for all budget alerts:

• a dedicated Pub/Sub topic
• a dedicated Cloud Function
• a dedicated collaboration space (and webhook)

Define naming standards for Budgets, Pub/Sub Topics, and Functions: Defining and enforcing naming standards ensures traceability, reduces confusion, and makes troubleshooting much easier at scale. Below are some key considerations when defining your naming standards for this solution:

• Budget naming standards should clearly convey which project the budget is associated with to ensure budget alerts are actionable.
• Pub/Sub topic and Cloud Function naming standards should not be project-specific since they will serve many projects for a given team. However, they should clearly convey which team they belong to for traceability.

The Framework: Below is a visual guide to use when implementing this for each team. Remember, by using well-defined naming standards and a single team channel, you are able to reuse the same Pub/Sub topic, Cloud Function, and webhook across all the team’s projects, making this approach much more scalable and sustainable.

Creating a Budget and Configuring Alerts

Creating a project budget in GCP is easy, so long as you have the right level of access. To do so, you will need the “Billing Admin” IAM role. I am also going to assume that you’ve assessed your current spending trends on the project and have a good idea of what “normal” looks like — that’s an important part of the budget setting process. To create your budget in GCP:

1. Go to the GCP Billing Console and select the Billing Account associated with the project you want to set a budget on.
2. Select Budgets & Alerts from the left-hand Billing menu and click the Create Budget button.
3. Name your Budget and select the GCP Projects and Products in scope for the budget. Click Next. Note: I recommend selecting only one Project per budget and always selecting all Products. I also highly recommend putting the project name or ID in the Budget name so that you can trace the budget alert back to the project.
4. Select the type of budget you want to set (“Specified Amount” or “Last Month’s Spend”), and provide an amount if choosing the first option. Click Next. Note: For traditional IaaS projects with consistent levels of spend, I recommend using the “Specified Amount” option. For projects with expected month-over-month growth and variability, I suggest choosing “Last Month’s Spend” — this will provide a dynamic budget target that is updated for you monthly.
5. Configure your alert thresholds and trigger types (“Actual” vs “Forecasted”). Note: I like to use 3 alerts: one for when my actual budget is exceeded (actual at 100%), one for when the system suspects I’ll exceed my budget (forecasted at 100%), and one for when I flagrantly exceed my budget (actual at 120%).
6. Check the box to “Connect a pub/sub topic to this budget” and click the dropdown to “Select a pub/sub topic.” If this is your first time setting up a budget for this team, select Create a Topic option and name the topic in a way that clearly articulates the team it’s related to. If you have already created a budget in the past, select the appropriate pub/sub topic from the list.

Configuring alerting thresholds and configuring a pub/sub topic

7. Click Finish. Your budget, alerts, and pub/sub topic have all be created for a project. You can now rinse and repeat for all your team’s projects.

Integrating Budget Alerts with Slack

Prerequisites: This section assumes that you’ve already configured your budgets and alerts, and you have established a pub/sub topic as described in Creating a Budget and Configuring Alerts section.

Step 1: Follow these instructions to create a webhook in the Slack Channel of your choosing. Remember to save this webhook url for use in step 2.4 below.

Step 2: Create a Cloud Function that processes and posts alerts to your Slack Channel via the webhook. To do this:

1. Navigate to the Cloud Function page in the GCP Console and click the Create Function button. Note: Be sure that you have selected the appropriate project where you want this Cloud Function deployed.
2. Name the Cloud Function (remember to specify what team it maps to in the name), select the minimum memory allocation (128 MiB), set the Trigger to “Cloud Pub/Sub,” and select the pub/sub topic you created in step #6 in the section above.
3. Using the inline editor, select Node.js 10 as your Runtime and paste in the following code in the editor: index.js code and package.json code.
4. In the index.js code, where it says “<insert webhook url>”, update the uri in the request body with the webhook url you created in “Step 1” above.
5. In the Function to execute input, specify “notifySlack” as the function (do not use quotes when pasting). This is the name of the function written in the code — you can see it after the “exports.” notifier, which signifies a function in Node.js.
6. Click Finish to provision the function.

Step 3: Test your Cloud Function with a test alert.

1. Once the function has been deployed, navigate to the Cloud Pub/Sub console and select the topic that you created for the Budget Alerts.
2. Click the Publish Message button at the top of the page. In the Message Body section, paste in the following json string that represents a billing alert:

{
    "budgetDisplayName": "name-of-budget",
    "alertThresholdExceeded": 1.0,
    "costAmount": 100.01,
    "costIntervalStart": "2019-01-01T00:00:00Z",
    "budgetAmount": 100.00,
    "budgetAmountType": "SPECIFIED_AMOUNT",
    "currencyCode": "USD"
}

3. Click Publish.

In a matter of seconds, you should see a billing alert message posted to your Slack Channel like this! You have successfully configured your alerting pipeline and will now receive updates like this when a billing alert fires. Remember to rinse and repeat using the same pub/sub topic and cloud function for all projects managed by this particular team.

Example of a GCP Billing Alert in Slack

Integrating Budget Alerts with Microsoft Teams

Prerequisites: This section assumes that you’ve already configured your budgets and alerts, and you have established a pub/sub topic as described in Creating a Budget and Configuring Alerts section.

Step 1: Follow these instructions to create a webhook in the Microsoft Teams channel of your choosing. Remember to save this webhook url for use in step 2.4 below.

Step 2: Create a Cloud Function that processes and posts alerts to your Microsoft Teams channel via the webhook. To do this:

1. Navigate to the Cloud Function page in the GCP Console and click the Create Function button. Note: Be sure that you have selected the appropriate project where you want this Cloud Function deployed.
2. Name the Cloud Function (remember to specify what team it maps to in the name), select the minimum memory allocation (128 MiB), set the Trigger to “Cloud Pub/Sub,” and select the pub/sub topic you created in step #6 in the section above.
3. Using the inline editor, select Node.js 10 as your Runtime and paste in the following code in the editor: index.js code and package.json code.
4. In the index.js code, where it says “<insert webhook url>””, update the uri in the request body with the webhook url you created in “Step 1” above.
5. In the Function to execute input, specify “notifyMicrosoftTeams” as the function (do not use quotes when pasting). This is the name of the function written in the code — you can see it after the “exports.” notifier, which signifies a function in Node.js.
6. Click Finish to provision the function.

Step 3: Test your Cloud Function with a test alert.

1. Once the function has been deployed, navigate to the Cloud Pub/Sub console and select the topic that you created for the Budget Alerts.
2. Click the Publish Message button at the top of the page. In the Message Body section, paste in the following json string that represents a billing alert:

{
    "budgetDisplayName": "name-of-budget",
    "alertThresholdExceeded": 1.0,
    "costAmount": 100.01,
    "costIntervalStart": "2019-01-01T00:00:00Z",
    "budgetAmount": 100.00,
    "budgetAmountType": "SPECIFIED_AMOUNT",
    "currencyCode": "USD"
}

3. Click Publish.

In a matter of seconds, you should see a billing alert message posted to your Microsoft Teams channel like this! You have successfully configured your alerting pipeline and will now receive updates like this when a billing alert fires. Remember to rinse and repeat using the same pub/sub topic and cloud function for all projects managed by this particular team.

Example of a GCP Billing Alert in SlackExample of a GCP Billing Alert in Microsoft Teams

Integrating with Hangouts Chat

Prerequisites: This section assumes that you’ve already configured your budgets and alerts, and you have established a pub/sub topic as described in Creating a Budget and Configuring Alerts section.

Step 1: Follow these instructions to create a webhook in the Hangouts Chat room of your choosing. Remember to save this webhook url for use in step 2.4 below.

Step 2: Create a Cloud Function that processes and posts alerts to your Hangouts Room via the webhook. To do this:

1. Navigate to the Cloud Function page in the GCP Console and click the Create Function button. Note: Be sure that you have selected the appropriate project where you want this Cloud Function deployed.
2. Name the Cloud Function (remember to specify what team it maps to in the name), select the minimum memory allocation (128 MiB), set the Trigger to “Cloud Pub/Sub,” and select the pub/sub topic you created in step #6 in the section above.
3. Using the inline editor, select Node.js 10 as your Runtime and paste in the following code in the editor: index.js code and package.json code.
4. In the index.js code, where it says “<insert webhook url>””, update the uri in the request body with the webhook url you created in “Step 1” above.
5. In the Function to execute input, specify “notifyHangouts” as the function (do not use quotes when pasting). This is the name of the function written in the code — you can see it after the “exports.” notifier, which signifies a function in Node.js.
6. Click Finish to provision the function.

Step 3: Test your Cloud Function with a test alert.

1. Once the function has been deployed, navigate to the Cloud Pub/Sub console and select the topic that you created for the Budget Alerts.
2. Click the Publish Message button at the top of the page. In the Message Body section, paste in the following json string that represents a billing alert:

{
    "budgetDisplayName": "name-of-budget",
    "alertThresholdExceeded": 1.0,
    "costAmount": 100.01,
    "costIntervalStart": "2019-01-01T00:00:00Z",
    "budgetAmount": 100.00,
    "budgetAmountType": "SPECIFIED_AMOUNT",
    "currencyCode": "USD"
}

3. Click Publish.

In a matter of seconds, you should see a billing alert message posted to your Hangouts Chat room like this! You have successfully configured your alerting pipeline and will now receive updates like this when a billing alert fires. Remember to rinse and repeat using the same pub/sub topic and cloud function for all projects managed by this particular team.

Example of a GCP Billing Alert in Hangouts Chat

Integrating with Webex Teams

Prerequisites: This section assumes that you’ve already configured your budgets and alerts, and you have established a pub/sub topic as described in Creating a Budget and Configuring Alerts section.

Step 1: Follow these instructions to create a webhook in the Webex Teams room of your choosing. Remember to save this webhook url for use in step 2.4 below.

Step 2: Create a Cloud Function that processes and posts alerts to your Webex Teams room via the webhook. To do this:

1. Navigate to the Cloud Function page in the GCP Console and click the Create Function button. Note: Be sure that you have selected the appropriate project where you want this Cloud Function deployed.
2. Name the Cloud Function (remember to specify what team it maps to in the name), select the minimum memory allocation (128 MiB), set the Trigger to “Cloud Pub/Sub,” and select the pub/sub topic you created in step #6 in the section above.
3. Using the inline editor, select Node.js 10 as your Runtime and paste in the the following code in the editor: index.js code and package.json code.
4. In the index.js code, where it says “<insert webhook url>”, update the uri in the request body with the webhook url you created in “Step 1” above.
5. In the Function to execute input, specify “notifyWebexTeams” as the function (do not use quotes when pasting). This is the name of the function written in the code — you can see it after the “exports.” notifier, which signifies a function in Node.js.
6. Click Finish to provision the function.

Step 3: Test your Cloud Function with a test alert.

1. Once the function has been deployed, navigate to the Cloud Pub/Sub console and select the topic that you created for the Budget Alerts.
2. Click the Publish Message button at the top of the page. In the Message Body section, paste in the following json string that represents a billing alert:

{
    "budgetDisplayName": "name-of-budget",
    "alertThresholdExceeded": 1.0,
    "costAmount": 100.01,
    "costIntervalStart": "2019-01-01T00:00:00Z",
    "budgetAmount": 100.00,
    "budgetAmountType": "SPECIFIED_AMOUNT",
    "currencyCode": "USD"
}

3. Click Publish.

In a matter of seconds, you should see a billing alert message posted to your Webex Teams room like this! You have successfully configured your alerting pipeline and will now receive updates like this when a billing alert fires. Remember to rinse and repeat using the same pub/sub topic and cloud function for all projects managed by this particular team.

Example of a GCP Billing Alert in Webex Teams

Conclusion

GCP Budget Alerts are a great tool for any organization trying to manage cloud spend, and they can be even more effective when they are surfaced to project teams in the tools they work in daily. In this article, I showed you how to accomplish this using Pub/Sub, Cloud Functions, and webhooks, and I provided a framework for doing this at scale with highly distributed teams. Now it’s your time to go be a hero at your organization!