[Tutorial] Setting up Budget Alert in GCP — English ver.
This article is written in both Thai and English
Thai version: https://mesodiar.com/setting-up-budget-alert-in-gcp-22e0b6b7bd92
Have you ever opened a service on the cloud and accidentally forgot to close it, ending up with charges ?
Have you ever used the cloud extensively, and the billing updates were slow, leaving you unaware of how much the cost has accumulated at the moment?
This article will teach you how to set up a budget alert flow when the usage cost exceeds the specified budget. This flow will include certain actions such as sending notifications, logically alerting once per month, or performing automated conditional actions as desired.
This flow is designed specifically for Google Cloud Platform (GCP), starting from Cloud Billing, Cloud Pub/Sub, Cloud Functions, and utilizing Firestore.
This case can be employed when you wish to monitor costs in a project, from a single project upwards. In the case of numerous projects, this approach can alleviate the burden of manual cost checking.
In these example scenarios:
Scenario 1:
Given: There is frequent querying of data this month, leading to high usage of BigQuery.
When: The cost exceeds 75% of the monthly budget we have
Then: The system will send a notification informing us of the current cost and that 75% has been reached.
Scenario 2:
Given: There is frequent querying of data this month, leading to high usage of BigQuery.
When: The cost exceeds 100% of the budget we have
Then: The system will send a notification disclosing the current cost, reaching 100%, and recommend adjusting the pricing plan from on-demand to BigQuery Editions.
Prerequisites
- For projects involving the use of BigQuery (or any other service), if you want to experiment without the fear of excessive costs, I recommend signing up for a new GCP account to claim the $300 free credit.
- We need to have a role called
Billing Account Costs managerorBilling Account Administratorwhich holds significant privileges at the organization level. For larger companies, this role is considered quite substantial. If you want an easy testing approach, consider signing up for a new account. - Keeping a budget in mind.
- Choose the channels through which you want to receive notifications. For instance, if you want alerts on Slack or Microsoft Teams.
Next, we will proceed with the setup as follows: Cloud Billing > Cloud Pub/Sub > Cloud Firestore > Cloud Functions.
Cloud Billing Budget & Alerts
- When we navigate to console.cloud.google.com and enter the term ‘billing alert’ in the search bar, the words ‘Budgets & Alerts’ will appear.
2. If we have the Billing Account Costs manager or Billing Account Administrator privileges, we will be able to see this page, and this page will display a list of budget alerts that we can access and configure.
Click on “Create Budget” to initiate the creation process.
3. Let’s name this alert as mils-project-bigquery-notification
This name is derived from the GCP project name “mils-project” (which is my name in it LOL) and the term “BigQuery,” aligning with the fact that this alert pertains to the BigQuery service used in this project.
Now set the time range to Monthly and select only the BigQuery service.
However, if you want to monitor costs for the entire project, you can simply choose all services.
Don’t forget to uncheck Discounts and Promotions and Others because we want to monitor the actual incurred costs initially. If we include discounts, notifications might not be triggered.
You’ll notice that if the project is in use, there’s a Cost Trend graph displayed on the right.
4. Enter the desired budget.
5. Choose the desired thresholds.
We will select thresholds at 50%, 75%, and 100%. For each threshold, there will be different actions. The logic will be processed by Cloud Functions later.
You will observe that there are 3 choices under Manage Notifications. You need to choose either 1 or 2. If you’re as admin of GCP project, the choice doesn’t make much difference.
However, if you’re using a newly created account for testing, you might need to create additional Notification Channels. Option 2 allows custom notification via email. Create an email alias for the current email you’re using.
At this point, if you think your flow only requires sending emails or using a direct webhook, that’s achievable.
But if you want to add variations in each Threshold condition, then you should go for choice 3.
So, we’ll choose option 1 and 3. This means sending an email and also message to pub/sub, which acts as a bridge to reach Cloud Functions. The further customization will be within the Cloud Functions themselves.
Cloud Pub/Sub
In the option 3, we will create a Pub/Sub topic from this page.
After clicking on ‘create topics,’ a popup window will appear. We will name it as mils-project-bigquery-budget-alert-topic.
We will also enable message retention and set it to 31 days, as we want to retain messages for an adequate period as necessary.
Now click on ‘Save,’ and it’s done.
Next, let’s go and take a look at Cloud Pub/Sub.
You will notice that there’s a Pub/Sub topic that we created from the Billing Alert page.
When we click into mils-project-bigquery-budget-alert-topic, you will see that there are no subscriptions yet. Therefore, we need to create a subscription.
Click on ‘Create Subscription.’
We’ll name it mils-project-bigquery-budget-alert-sub to align with the topic name. Configure it as follows and then click on 'Create.'
Once we’ve successfully created the subscription, we can go and view the generated messages.
Navigate to the subscription > Messages tab > Click on ‘Pull.’
However, due to the fact that we’ve just set up this subscription freshly, there won’t be any messages flowing in yet.
If we leave it for a while, messages will start coming in continuously. This is a sample scenario illustrated in the image below.
Message Example in Pub/sub
Messages pulled from the subscribed topic come in two formats:
- Messages when the cost has not yet reached the threshold.
{
"budgetDisplayName": "mils-project-bigquery-notification",
"costAmount": 0.0,
"costIntervalStart": "2023-07-01T07:00:00Z",
"budgetAmount": 1.0,
"budgetAmountType": "SPECIFIED_AMOUNT",
"currencyCode": "USD"
}- Messages when the cost has reached the threshold.
{
"budgetDisplayName": "mils-project-bigquery-notification",
"alertThresholdExceeded": 1.0,
"costAmount": 120.0,
"costIntervalStart": "2023-07-01T07:00:00Z",
"budgetAmount": 100.0,
"budgetAmountType": "SPECIFIED_AMOUNT",
"currencyCode": "THB"
}A slightly headache-inducing aspect of Pub/Sub is that it sends messages continuously regardless of the cost value. After this, if we integrate Cloud Functions, it will be triggered every time a message is sent to the Pub/Sub topic, regardless of the cost.
Hence, we need to configure Cloud Functions to handle the possibility of receiving repetitive messages. This is crucial as Cloud Functions will be responsible for sending notifications to us, and we wouldn’t want redundant notifications flooding our end.
Cloud Functions
Now let’s configure Cloud Functions. We’ll name it budget-alert-notifications.
For this, we need to click on ‘Add Trigger’ and select the Pub/Sub trigger. Choose the name from the Pub/Sub topic we’ve set up.
Once you click on ‘Next,’ you’ll arrive at the page where you can set what actions Cloud Functions should take.
We want Cloud Functions to receive values from the message and check if alertThresholdExceeded is present.
If it is, it means the cost has reached the threshold. We will then differentiate actions for the 75% and 100% thresholds.
The code might look something like this :D
# Triggered from a message on a Cloud Pub/Sub topic.
@functions_framework.cloud_event
def subscribe(cloud_event: CloudEvent) -> None:
# Print out the data from Pub/Sub, to prove that it worked
print(
"Hello, " + base64.b64decode(cloud_event.data["message"]["data"]).decode() + "!"
)
pubsub_data = base64.b64decode(cloud_event.data["message"]["data"]).decode("utf-8")
pubsub_json = json.loads(pubsub_data)
cost_amount = pubsub_json["costAmount"]
budget_amount = pubsub_json["budgetAmount"]
budget_display_name = pubsub_json["budgetDisplayName"]
alertThresholdExceeded = pubsub_json.get("alertThresholdExceeded", None)
if alertThresholdExceeded is None:
if cost_amount <= budget_amount:
print(f"No action necessary. (Current cost: {cost_amount})")
return
else:
content = """
Budget name '{}'<br>
Current cost is <b>${}</b>.<br>
Budget cost is <b>${}</b>.<br>
💸 💸 💸 💸
""".format(
budget_display_name, cost_amount, budget_amount
)
title = "💸 Cost reaches {:.0f}% from {}".format(
alertThresholdExceeded * 100, budget_display_name
)
print(f"############################################")
print(f"budget threshold is {alertThresholdExceeded}")
print(f"{title}")
print(f"{content}")
print(f"############################################")
if alertThresholdExceeded == 0.75:
send_noti(content, title)
elif alertThresholdExceeded == 1.0:
### do something differently from 75% ###
send_noti(content, title)
def send_noti(content, title):
## send noti to somewhereHowever, the main issue here is that the Pub/Sub tends to send duplicate messages. Refering to document:
Budget notifications are sent to the Pub/Sub topic multiple times per day with the current status of your budget. This is a different cadence than the budget alert emails, which are only sent when a budget threshold is me
Now, if our requirement is to have the system send notifications only once per month, the challenge lies in Cloud Functions alone not being able to keep track of which projects have already received notifications. This is because the primary role of Cloud Functions is to check logic.
So, we decide to employ a separate database that can help us store information about whether notifications have been sent and when they were last sent. We have chosen Cloud Firestore for this purpose due to its simplicity.
Leaving Cloud Functions aside for now, let’s switch our focus to Firestore for a moment.
Cloud Firestore
Let’s switch over to Firestore for a moment.
We’ve chosen to create it in Native Mode, and we’ll set up a Collection named budget-alert-notification-stats.
The Firestore interface will have a layout somewhat like this. Within the budget-alert-notification-stats, there will be documents listing projects, such as project_a and project_b, to track whether notifications have been sent. These documents will also store the timestamp of the most recent notification.
At this point, this is just for example. You might not have project_a and project_b yet. Creating just the collection is sufficient. Now, let’s go back to Cloud Functions.
Back to Cloud Functions
Now, let’s edit the code within Cloud Functions to notice Cloud Firestore. It will check whether notifications have been sent for that particular month.
If no notifications have been sent yet, it will send one.
However, if notifications have already been sent, there’s no need to take any further action.
# Triggered from a message on a Cloud Pub/Sub topic.
@functions_framework.cloud_event
def subscribe(cloud_event: CloudEvent) -> None:
# Print out the data from Pub/Sub, to prove that it worked
print(
"Hello, " + base64.b64decode(cloud_event.data["message"]["data"]).decode() + "!"
)
pubsub_data = base64.b64decode(cloud_event.data["message"]["data"]).decode("utf-8")
pubsub_json = json.loads(pubsub_data)
cost_amount = pubsub_json["costAmount"]
budget_amount = pubsub_json["budgetAmount"]
budget_display_name = pubsub_json["budgetDisplayName"]
alertThresholdExceeded = pubsub_json.get("alertThresholdExceeded", None)
project_id = get_alert_name(pubsub_json["budgetDisplayName"])
if alertThresholdExceeded is None:
if cost_amount <= budget_amount:
print(f"No action necessary. (Current cost: {cost_amount})")
return
else:
content = (
"""Budget name '{}'.Current cost is ${}.Budget cost is ${}💸 💸 💸 💸""".format(
budget_display_name, cost_amount, budget_amount
)
)
title = "💸 Cost reaches {:.0f}% from {}".format(
alertThresholdExceeded * 100, budget_display_name
)
print(f"############################################")
print(f"budget threshold is {alertThresholdExceeded}")
print(f"{title}")
print(f"{content}")
print(f"############################################")
alert_percent = int(alertThresholdExceeded * 100)
should_notify = check_to_notify_per_month(project_id, alert_percent)
if alertThresholdExceeded == 0.75 and should_notify:
send_teams(
webhook_url,
content=content,
title=title,
color=green_code,
)
elif alertThresholdExceeded == 1.0 and should_notify:
### do something differently from 75% ###
send_teams(
webhook_url,
content=content,
title=title,
color=red_code,
)
def check_to_notify_per_month(project_id: str, threshold_percent) -> bool:
"""_summary_
check if already notify in Teams in that month or not
by checking from Cloud Firestore
Args:
project_id (str): project id get from alert name
Returns:
bool: True when already notify and update data in Firestore, False to skip notify
"""
db = firestore.Client()
last_notify_ref = db.collection("budget-alert-notification-stats").document(
project_id
)
current_timestamp = pendulum.now("Asia/Bangkok")
try:
last_notify_snapshot = last_notify_ref.get()
data = last_notify_snapshot.to_dict()
if not last_notify_snapshot.exists:
# Initialize with the current timestamp if it's the first time.
last_notify_ref.set({f"last_noti_{threshold_percent}": current_timestamp})
print("Never has notification before, should notify team")
return True
elif f"last_noti_{threshold_percent}" not in data:
# Update existing doc with new threshold
last_notify_ref.update(
{f"last_noti_{threshold_percent}": current_timestamp}
)
print(
f"Never has notification for {threshold_percent} before, should notify team"
)
return True
else:
# Both threshold is in doc, check if it should notify
last_notify_in_bkk = convert_to_timestamp_with_bkk(
last_notify_snapshot.get(f"last_noti_{threshold_percent}")
)
print("last_notify_in_bkk: ", last_notify_in_bkk)
print("current_timestamp: ", current_timestamp)
# Check if a month has passed since the last notify.
if has_one_month_passed(current_timestamp, last_notify_in_bkk):
# Proceed with sending the notify to Microsoft Teams.
# Update the last notify timestamp.
last_notify_ref.update(
{f"last_noti_{threshold_percent}": current_timestamp}
)
return True
else:
print("Skipping notify, All ready notify in this month.")
return False
except Exception as e:
print("Error processing notify:", e)
def convert_to_timestamp_with_bkk(timestamp_from_firestore: datetime) -> datetime:
last_notify_timestamp = timestamp_from_firestore.strftime("%Y-%m-%dT%H:%M:%S.%f%z")
last_notify_timestamp = pendulum.parse(last_notify_timestamp, tz="UTC")
last_notify_timestamp_in_bkk = last_notify_timestamp.in_tz("Asia/Bangkok")
return last_notify_timestamp_in_bkk
def has_one_month_passed(
current_timestamp: datetime, last_notify_timestamp: datetime
) -> bool:
current_month, current_year = get_month_and_year(current_timestamp)
last_notify_month, last_notify_year = get_month_and_year(last_notify_timestamp)
return (current_year > last_notify_year) or (
current_year == last_notify_year and current_month > last_notify_month
)
def get_month_and_year(timestamp: datetime):
month = timestamp.month
year = timestamp.year
return month, yearHow Cloud Functions looks like for now:
In brief, the implementation might look something like this. If you’re interested in reading the complete code, I recommend checking it out on GitHub.
Testing Cloud Functions
Since we need to wait for a while until messages start coming in, in the meantime, we can perform tests without having to wait. Go to the Pub/Sub topic > Messages > Publish Message.
Then, you can send a simulated message through the topics.
{"test": "yes","budgetDisplayName": "mils-project-bigquery-notification","alertThresholdExceeded": 0.75,"costAmount": 730.0,"costIntervalStart": "2023-07-02T07:00:00Z","budgetAmount": 1.0,"budgetAmountType": "SPECIFIED_AMOUNT","currencyCode": "THB"}
If you receive notifications, it means Cloud Functions is operational.
However, after testing, remember to delete the document in Cloud Firestore; otherwise, it won’t send new notifications. ;)
Results of the Setup
- Cloud Pub/Sub Topics
If you leave the system running for a while, within Cloud Pub/Sub topics, messages will start arriving, looking somewhat like this.
- Cloud Firestore
In Firestore, you will find documents named after projects, containing fields such as last_noti_100 and/or last_noti_75.
In this case, there is only last_noti_100 because the cost for this month has exceeded the budget.
- Notification Channels
There will be notifications coming through two channels: email and Teams, which we’ve configured.
Conclusion
In reality, the core flow necessary is Cloud Billing > Cloud Pub/Sub > Cloud Functions.
Firestore serves as an auxiliary component, as we want the system to send notifications only once per month.
It all depends on how the system architect designs the flow. For instance, for the 100% threshold, we could potentially adjust BigQuery to switch to the Editions pricing plan automatically (also known as assigning reservations). However, we didn’t include this in the article as we wanted to focus on illustrating the primary operations of these three components. Readers can customize as needed.
