Understanding Stackdriver Logging and Monitoring Usage
Stackdriver announced new pricing in June meant to simplify the pricing structure. Moving to a usage-based pricing model brought Stackdriver in line with other consumption-based cloud products which makes it easier to reason about the cost of using the product.
I recently went through the GCP bill analysis exercise to understand current and anticipated spend for my projects and was pleased at how easily I was able to find the details that I needed. Though my bill was extremely low, I wanted to know how Stackdriver contributed to the overall bill.
If you want to dig into your GCP bill and understand your Stackdriver spend, there are a number of different tools that help surface the relevant information. Here are the steps in the analysis that I used to analyze the Stackdriver costs.
1. Understand Stackdriver pricing
The Stackdriver pricing site provides the specifics of the Logging and Monitoring costs and provides a baseline for understanding what drives costs in Stackdriver. There is a great diagram called “Life of a Log” that also helps to illustrate Stackdriver Logs ingestion and export.
2. GCP Bill Analysis
The first place I started was at the billing report for my GCP billing account. The Billing section in the GCP console has a Reports tab that allows you to analyze your bill by viewing your charges over time on a graph. The UI offers a useful range of filters to narrow the results by time, project, products, SKUs and labels.
In my case, I wanted to understand the Stackdriver charges over time. First, I looked through the previous month and then found a specific week to review. I selected the date range option for the specific week and then removed 1 project from the projects filter that I didn’t want to include in the total.
I wanted to see my Stackdriver charges linked to this billing account across all projects and so I left the rest of the projects selected. Finally, I selected the Stackdriver Logging and Stackdriver Monitoring under the Products filter. The resulting graph illustrates the charges by project. I noticed that all of my cost was coming from a specific project.
Next, I wanted to see the mix of costs between the Logging and Monitoring products. I changed the Group filter to Product and I saw that Stackdriver Logging was actually the product making up the cost.
Now that I understood the costs, I looked into which projects and logs were generating log volume.
3. Review Stackdriver Logs Ingestion
Stackdriver Logging provides detailed list of logs, current log volume and projected monthly volume. This makes it easy to see which logs have the highest volume.
Additionally, each log entry has a deep link into Stackdriver Monitoring which displays a graph of the volume of the specific log over time. I selected “View detailed usage in Metrics Explorer” for the Kubernetes Cluster metrics and opened the Metrics Explorer dashboard in Stackdriver Monitoring.
In the Metrics Explorer, I looked at the volume of logs being ingested over the specific timeframe.
This analysis helped to answer my question about which logs were generating the most volume and therefore contributing to the overall GCP bill.
4. Review Stackdriver Monitoring usage.
My project didn’t have significant Stackdriver Monitoring charges though I did find it useful to look at the usage information. In the Stackdriver Monitoring UI, I selected my project name from the drop-down and then the Account Settings option. I then selected the Usage details which displayed the overall usage details for my Stackdriver account.
The metrics ingested box includes both the current overall metric volume ingested and the end of month projection based on the current ingestion rate. By selecting the project name, the specific types of metrics and their associated volumes are ingested. Both custom metrics and agent metrics incur cost when usage exceeds the monthly free ingestion quota (currently 150MB).
To explore the metrics ingestion volumes over time, I selected the drop-down next to the project and selected the “View in Metrics Explorer” link which opened the Stackdriver Monitoring Metrics Explorer.
In the Metrics Explorer, I could see the volume of metrics being ingested over time.
This analysis helped to answer my question about which metrics were generating the most volume and therefore contributing to the overall GCP bill.
Conclusion
The recent changes to the Stackdriver pricing simplifies understanding the Stackdriver cost on the GCP bill. Stackdriver Logging and Monitoring both provide useful tools to analyze their respective usage. See the references below for additional information.
