GCP Operations Suite Alerts into Google Chat

UPDATE: As of the 4th of October 2023 this is now available natively as per: https://cloud.google.com/monitoring/support/notification-options#google-chat

Introduction

As I have said previously I love the tooling on offer in the cloud, especially around monitoring. I am also a big fan of the Google Workspace (formerly Gsuite) group of applications and have used Gmail, Drive, Docs, Calc, and Slides personally for a long while (including when I was at university some 10 years ago!). So I was delighted when I joined CTS to get to use these tools in a workplace setting (and I say that as someone who has done two Exchange to 365 Migrations!).

The Problem

Anyway, to come back round onto topic within Workspace the instant messaging tool is ‘Chat’ which we use extensively at CTS to communicate with one another all over the UK, NL, and the USA. We also work exclusively in GCP and try and use as much of the native tooling as possible including Operations Suite (formerly Stackdriver).

Operations Suite contains several tools but I am going to focus on ‘Monitoring’ which allows you to send notifications when alert thresholds on metrics are met. To use Google SRE terminology (on which they wrote the book!) I want to know when my SLIs cause me to believe I won’t meet my SLOs so I don’t get anywhere close to my SLAs.

Now, Google monitoring currently supports a few different ‘notification channels’ for alerts which are (at the time of writing in April 2022):

• Email
• SMS
• Pager Duty
• Cloud Console Mobile App
• Slack
• Webhooks
• Pub/Sub

Now in my opinion there is a little bit of a glaring omission here in that Google’s very own Chat isn’t a listed option, especially when Slack, which I would consider to be a competitor, is! You might also think that you might be able to use webhooks to link into chat but I found, unfortunately, this doesn’t work for reasons unknown (to me anyway), possibly to do with the formatting of the message.

The Solution

I have recently been doing some work with a customer deep-diving into both Cloud Logging and Cloud Monitoring to assist them in highlighting actions of concern to their SOC team. During this work, we tried various options including Log Based Alerts (which I wrote about in a separate blog post) but eventually, we settled on a solution using a combination of Pub/Sub and Cloud Functions written in Python.

When you set Pub/Sub as a notification channel Monitoring will send a message to a specified topic which can then be used as the triggering event for a cloud function.

Don’t forget to grant the Monitoring service account the pub/sub publisher role on your pub/sub topic!

The great news is that this message is in everyone's favourite format for sending information, JSON! Below is the actual message sent when you use the test the notification channel:

As you can see there is quite a healthy amount of information. Now all that really needs to be done is to write a cloud function that extracts out the JSON values we want, format it into a message and send it to chat using a webhook. Here is some example code I have written which does this:

To step through the code the first thing I decide to do is check to see if it is an incident opening or closing (which can also optionally be notified for) as I don’t want to notify on incident close. Assuming it is a new incident I then decided to take the scoping project ID, the name of the alert policy, and the incident URL and create a message which is then sent to chat:

Example of Webhook Bot Message

Creating the webhook within chat is really straightforward and described well by these Google-provided instructions.

Concluding Thoughts

Notification Flow from: Monitoring -> Pub/Sub -> Functions -> Chat

So whilst I wish there was a native Monitoring to Chat notification channel option building something custom using pub/sub and cloud functions does not take long at all. This method also allows you to do a few other useful things:

• Have a single pub/sub topic set as the notification channel for several projects (not forgetting to set the correct IAM as mentioned above!).
• Being able to pick and choose from the JSON object how much information you want to be parsed into the chat message.
• Being able to expand the complexity of the function to handle for example multiple targets (eg to send to different teams based on project ID).

As always if these are the sort of challenges you find interesting at CTS we are constantly looking for people who share our passion but otherwise, until next time — keep it Googley ;-)

— — —

About CTS

CTS is the largest dedicated Google Cloud practice in Europe and one of the world’s leading Google Cloud experts, winning 2020 Google Partner of the Year Awards for both Workspace and GCP.

We offer a unique full stack Google Cloud solution for businesses, encompassing cloud migration and infrastructure modernisation. Our data practice focuses on analysis and visualisation, providing industry specific solutions for; Retail, Financial Services, Media and Entertainment.

We’re building talented teams ready to change the world using Google technologies. So if you’re passionate, curious and keen to get stuck in — take a look at our Careers Page and join us for the ride!