Optimise Cloud logging in Google Cloud with logging query language
Is your cloud logging bill more than what you had estimated? Let’s learn a way to filter them out and reduce your monthly consumption.
Before jumping to the steps, let’s understand how Cloud Logging pricing works.
Cloud Logging incurs charges on the basis of following two factors:
- Ingestion : Writing data to the Cloud Logging API and routing it to log buckets
- Storage : Logs data that is retained in log buckets
Google Cloud creates two storage buckets automatically in Logs Storage which are _Default and _Required.
_Required log bucket contains certain logs like Audit logs and transparency logs which get automatically ingested into this bucket. We are not allowed to delete or edit this bucket. Most importantly, we are not charged for its ingestion and storage. However, we may get charged if you want to keep copies of these logs in other buckets.
Buckets other than _Required will get charged for its ingestion and storage. Currently, Cloud logging provides free storage for retaining logs for the first 30 days irrespective of any bucket. Post January 16, 2023, storage costs will apply to logs data retained longer than 30 days. Refer to the documentation here.
Following is the high level flow chart of how Cloud Logging routes and stores log entries. Refer to this link to read more in detail how logs are ingested by Logging and routed through the Log Router using sinks.
Now we understand that Log Router ingests and routes log into sinks. Therefore, to filter logs, we have to set one or more exclusion filters to exclude logs from the sink’s destination.
Exclusion filters are created by using the logging query language. We can use the logging query language in the Logs Explorer in the Google Cloud console, the Logging API, or the command-line interface. Read more here.
We can build queries based on following fields :
- resource.type
- resource.labels.*
- logName
- severity
- timestamp
- insertId
- operation.id
- trace
- httpRequest.status
- labels.*
- Split.uid
Apart from these fields, we can also build queries and configure them based on our custom fields (currently in Pre-GA). Read more here in detail.
Problem Statement :
There are scenarios where customers are overpaying for Cloud logs in Google Cloud, unknowingly and they are not aware how to filter the relevant logs.
Google Cloud recommends using the Logging query language to query data and to write filters to create sinks and log-based metrics. However, at times it is not that straightforward.
The solution :
To solve this problem and as a best practice we should add exclusion filters in the Log Router Sinks based on the fields which we want to exclude.
In order to enter the exclusion filter, go to Logging in your GCP console and click on Logs Router from the left panel (as mentioned below).
Click on the three vertical dots of _Default sink/custom sink and then click on Edit Sink.
Scroll down to the Choose logs to filter out of sink (optional) and click on Add Exclusion.
Now, let’s say you want to filter logs based on resource type “Google App Engine ‘’ application and severity “DEFAULT”. Give a name to your filter like sampleexclusionfilter, enter the following query in the exclusion and then click on Update Sink.
resource.type=”gae_app”
severity=DEFAULT
This will remove all the default logs from the GAE application. Similarly, you can enter more queries to filter out of the sink on the basis of log name, project id, module id and the selected zone.
Find out the highest number of irrelevant logs for you in the Logs explorer and copy the same query in the exclusion filter as outlined above.
Summary :
And this is how we learnt what are the pricing factors for Cloud logging in GCP , the process of logs ingestion and routing through the Log Router using sinks and ultimately, how we can bring a significant difference to our cloud consumption bill in Google Cloud Platform.
Read more about additional features of Cloud logging.
