Capping costs on GCP for many projects with a budget for many months — without paying a penny :)
During this hackathon, candidates would enjoy working and discovering Google Cloud Services to implement their solutions. As organizers (Umanis), we offered a budget of 500 EU for each project. A budget is usable for 2 months, the duration of the hackathon. Google Cloud, our partner, offered a limited credit that is attachable to one billing account. So you can see how we have a hard limit on how much money we can spend on Google Cloud Platform for the hackathon.
At the moment of writing, there is no straightforward solution. And here is in more details the wolf of our story :
- Budget spend resets the first day of each month, but we want to manage and reason about the project budget for the whole duration of the hackathon, that exceeds one month.
- Setting a budget just sends alerts, but does not cap resources or API consumption
- The billing account is linked to the hackathon projects and other projects, but we want to manage budgets only for the hackathon projects; and not one project but many hackathon projects
In Cloud Billing documentation, Cap (disable) billing to stop usage, you can find a considerable chunk of the solution. Unfortunately it does not solve our problems. In fact, it only caps costs for one project only, and does not span more than one month.
Google Cloud documentation is one of the best documentations I ever used. So sometimes I will not hesitate to copy paste, of course with modification for our solution, from Google Cloud Billing documentation, more specifically from these two sections,
The solution in few words
As with Cloud Billing solution, ours, caps costs by disabling billing. So you need to pay attention to the warnings in the link above (capping costs).
We will :
- Create a budget (a Google Cloud resource) per project we want to cap costs for
- Link the budget to the project we want to cap
- Enable programmatic notifications to receive Cloud Pub/Sub messages with the current status of the budget
- Configure each budget notifications to be published to the same Cloud Pub/Sub topic
- Use a unique cloud function to handle, programmatically and for all projects, the budget notifications received through Cloud Pub/Sub topic.
I recommend to create a side project for admin / ops purposes. In this post my admin project is named and has mehdi-labs-201811 as id; from now on I will refer to it as Admin project. This is where we will create our Cloud Pub/Sub topic to receive the budget notifications of all projects to monitor. We will call this topic : budgets-notifications. Also this is where we will deploy, our Cloud Function to handle the different budgets notifications and cap costs.
In this article I will use cap costs and monitor interchangeably. The same goes for budget and budget alert. Actually it is more about budget then budget alerts
You can find here in Google Cloud Billing documentation, a very good description of the budgets notifications that you would receive on Cloud Pub/Sub created topic. Below an example of a notification in JSON format that you can publish to the topic for testing.
So let’s begin :
Create a budget for a project
To enable programmatic budgets notifications:
Configure a Cloud Pub/Sub topic Billing to publish your budget notifications to. In our case we will create Cloud Pub/Sub topic to be used by all budgets
- Go to the Google Cloud Platform Console Cloud Pub/Sub page.
- Select the Admin project (mehdi-labs-201811).
- Click Create topic, name the topic budgets-control, and Save.
To create a budget:
- Go to the Google Cloud Platform Console.
- Open the console left side menu and click Billing.
- If you have more than one billing account, select Go to linked billing account to manage the current project’s billing. To locate a different billing account, select Manage billing accounts.
- On the left, click Budgets & alerts.
- Click Create Budget.
- Under Budget name, you need to enter the project id, that you want to monitor (in this post, one of the projects id to cap is project-id-to-cap). This is the only way I found to communicate the project id to my Cloud Function handling capping costs
- Under Project or billing account, select the project that you want to apply the budget alert to.
- Under Budget amount choose to set the budget alert to a Specified amount
- Enter that amount, in our case 500.
- Under Manage Notifications, select Connect a Cloud Pub/Sub topic to this budget.
- Select our Admin Project and budgets-control Topic for notifications and then click Save.
- Repeat the same actions, for creating a budget, for each project you want to cap the costs
The Cloud Function code from 30,000 feet
The Cloud Function implementation will overcome budget spends resetting the first day of each month, by storing and retrieving “budget state” using Cloud Firestore.
If the total spend of a project exceeds the budget fixed for the project, the billing will be disabled for the project using Cloud Billing API.
Of course we will not forget proper authentication and authorization when needed.
Fortunately, all, or let me say, most of Google products / services are accessible with code thanks to Google Cloud APIs which are a key part of Google Cloud Platform.
Google Cloud APIs Python client libraries
While you can use Google Cloud APIs by making direct HTTP requests (or RPC calls where available), Google provide client library for all its Cloud APIs. This makes it easier to access them from your favorite languages. But there is a subtlety : there are different types of client libraries provided and here are the 2 that we will use :
- Google Cloud Client Libraries: it is the recommended option for accessing Cloud APIs programmatically, when available. And this is what we will use for Cloud Firestore. You can find the Cloud Firestore API Python client library documentation here.
- Google API Client Libraries: A number of Google Cloud APIs do not yet have Google Cloud Client Libraries available. In this case you will need to use another version : Google API Client Libraries. This is what I did with Google Cloud Billing API. You can find how to use it with Python client library by reading here then here then here 😁
When I started writing this post, Cloud Billing documentation code examples for Cloud Function were only provided in Node.js 6 😅 At that time, I chose Python to write my Cloud Function. That’s why I linked to all that documentation in Python, because it was the long path I traveled.
Using Beta products (Python 3.7 Cloud Function Runtime) does not come without a price. One day I was testing my Cloud Function, I changed a small portion, I deployed it and it failed. I spent hours trying to identify the cause without success. Finally I deployed the builtin hello world Cloud Function in Node.js 8 (Beta) and it did not work and guess what, when I deployed the hello world in Node.js 6, IT DID WORK ! 2 days later, my Python Cloud Function worked again without any modification
Why Cloud Firestore
I chose Cloud Firestore for different reasons :
- Although Cloud Firestore is still in Beta at the moment of writing, it is the next major version of Cloud Datastore.
- It has a start free then pay as you go pricing model and its free tier is very generous :)
- No other Google Cloud storage solution for random access apart from Cloud Datastore has the pay as you go pricing model.
- It’s very easy to browse Cloud Firestore data through Google Cloud Console
The solution in more details
Let’s start by creating the Cloud Function :
Creating the Cloud Function
- Go to the Cloud Functions page of the Google Cloud Platform Console. Create a new function and give it a name that is meaningful to your budget. I chose budgets-notifications-handler.
- Under Memory allocated, select 128 MB, there is no need to use more resources than this
- Under Trigger, select Cloud Pub/Sub topic.
- Select the topic that you configured on your budgets.
- Provide source code and dependencies for the function to run. We will use the inline editor, so you will be able to copy / paste the source code I provided below. As for me, I used Cloud Source Repository. It’s up to you.
- Under Runtime, select Python 3.7(Beta). At the time of writing, Python Runtime is still in Beta as Node.js 8.
Proper authorization for the Cloud Function
At runtime, our Cloud Functions uses the service account email@example.com. It is listed at the bottom of our Cloud Function details page.
Our Cloud Function will need to access and disable billing for other projects. To be able to do so, it will need Billing Account Administrator role. Besides it will need Project Browser role, more precisely, resourcemanager.projects.get permission on each project it will monitor.
You can manage Billing Admin permissions on the Google Cloud Platform Console Billing page. As shown below, add the Cloud Function runtime service account as a member and give it Billing Account Administrator privileges.
To give the Cloud Function runtime service account the Browser role, you need to add it as a member to each project to monitor through Google Cloud Platform IAM page and give it the Browser role as shown below
The Browser Role has only 6 assigned permissions, so using it in our case, “is compatible” with the principle of least privilege
Do not forget to enable the Cloud Billing API
You can find below, our Cloud Function python code. I hope I made enough effort to document it to avoid explaining it further in this post. The requirements are also provided below. So now you can copy paste in the inline editor of your Cloud Function and start playing ✨
I created some test data (budget notifications) that you can modify and start publishing them through your own Cloud Pub/Sub topic. I also provided a bash script that you can modify and use to deploy your Cloud Function. All of this is available in my Github repository.
I deployed this solution 6 weeks ago. It is monitoring 14 projects. Each project sends a budget notification every hour. I know this because I added a timestamp to each budget notification before storing it and keeping track of it in Cloud Firestore (implemented in the code below). All these Cloud Billing API, Cloud Pub/Sub, Cloud Function and Cloud Firestore calls and data stored in Cloud Firestore have cost me nothing (I was also monitoring my Admin Project). In fact, Cloud Pub/Sub, Cloud Firestore and Cloud Function come with a free tier (the links point to the pricing documentation) as it is the case with many other services in Google Cloud Platform.
Finally if you are reading these lines, chances are that you are interested in understanding you cloud spends details. You can achieve this by exporting your billing data to BigQuery then playing and getting the needed insights from that data.