Navigating Log Router Settings in Google Cloud for Optimal Efficiency

Cloud Logging

Introduction

Logging is a crucial tool in cloud computing, serving as a key resource for identifying bugs and tracing user activities. With the enforcement of strict data privacy regulations in various countries, it’s essential for DevOps engineers to adeptly navigate the complexities of log management. This includes understanding how to effectively filter and aggregate logs while adhering to legal standards for handling personally identifiable information (PII). In this article, I will explore the filtering methods for Log Router in Google Cloud, providing insights and techniques to help DevOps professionals manage their logging practices in a compliant and efficient manner.

About Log Router

The Log Router is a critical tool in Google Cloud’s logging system. At its core, the Log Router is responsible for managing the flow of logs generated by the various services within Google Cloud. Initially, these logs are collected via the Cloud Logging API. The Log Router then steps in to distribute these logs to different destinations, including services like Log Buckets, BigQuery, and Cloud Storage.

To illustrate, you might use the Log Router to direct logs to a Log Bucket for efficient exploration within Cloud Logging, or to a Cloud Storage Bucket for long-term storage and archival purposes. This process highlights the importance and versatility of the Log Router in handling log data across different services in the cloud.

An overview of the Logging Architecture in Google Cloud, as illustrated in the official documentation.

Additionally, I have discussed the Log Router for long-term storage in my previous post, which you can find here:

Budget-Friendly Techniques for Long-Term Log Storage in Google Cloud

Do Not Underestimate Cloud Logging Costs

Cloud Logging in Google Cloud can become surprisingly expensive, especially for large-scale services. It’s easy to overlook the costs associated with Cloud Logging, which include not just storage fees but also charges for ingesting logs. As of February 2024, the ingestion rate is $0.5 per GiB.

Pricing | Operations Suite | Google Cloud

Although log data, being predominantly text-based, might seem inexpensive to handle, the reality is different. Logs are generated continuously, and as your user base grows, the volume of logs — and consequently the cost — escalates proportionally. Moreover, a common pitfall is setting up multiple Log Routers with identical filtering conditions. This redundancy can inadvertently double your ingestion fees, as each Log Router processes the same logs independently.

To manage these costs effectively, it’s crucial to utilize log filtering strategically. Thoughtful configuration of Log Router can help in controlling the volume of logs ingested and stored, thereby keeping costs in check. This approach to cost management ensures that you maximize the efficiency of your Cloud Logging setup without incurring unnecessary expenses.

Utilize Log Router Effectively to Reduce the Costs

Configuring the Log Router efficiently is key to managing and sorting the volume of ingested logs. An example of setting up a Log Router using Terraform might look like this:

resource "google_logging_project_sink" "my_application" {
  project                = "my-project"
  name                   = "my-application-sink"
  destination            = "logging.googleapis.com/projects/my-project/locations/global/buckets/log-for-application"
  unique_writer_identity = true
  filter                 = <<EOF
    logName:"cloudaudit.googleapis.com"
    OR (resource.type="k8s_container" AND resource.labels.cluster_name="my-cluster" AND resource.labels.container_name="my-container")
    OR (labels.REGION="us-central1")
    EOF
}

In this filter, Monitoring Query Language (MQL) is utilized to specify the logs that should be ingested. This filtering helps in reducing costs by ensuring only relevant logs are processed. Additionally, the exclusions parameter can be used to further refine the filtering by eliminating specific logs that match defined MQL criteria. Here’s an example that includes an exclusion filter:

resource "google_logging_project_sink" "my_application" {
  project                = "my-project"
  name                   = "audit-log-bucket-sink"
  destination            = "logging.googleapis.com/projects/my-project/locations/global/buckets/log-for-application"
  unique_writer_identity = true
  filter                 = <<EOF
    logName:"cloudaudit.googleapis.com"
    OR (resource.type="k8s_container" AND resource.labels.cluster_name="my-cluster" AND resource.labels.container_name="my-container")
    OR (labels.REGION="us-central1")
    EOF

  exclusion {
    name = "Excluded-GCE"
    description = "This filter exclude GCE's log"
    filter = "resource.type = gce_instance"
  }
}

Implementing such detailed filters and exclusions in the Log Router configuration can significantly reduce the amount of log data ingested and stored, thus effectively lowering costs associated with Cloud Logging.

Understanding the Relationship Between Filter and Exclusion Parameters

In the previous sections, we discussed the use of two parameters, filter and exclusion, for log sorting. To better understand their roles, let's take a closer look. By default, a Log Router named _Default is created in Google Cloud, and its filter is typically configured as shown below:

Filter Setting of _Default Log Router

In this configuration, NOT is used within the filter parameter. Initially, I considered redefining this filter using exclusion for clarity, to distinguish between filter and exclusion. However, after moving all the filter conditions with NOT to exclusion, a caution message was displayed:

Caution message displayed after modifying the filter

This message indicates that the filter condition may take precedence over exclusion, leading to a situation where all types of logs could be ingested into the Log Router. Therefore, it’s crucial to carefully balance and understand how filter and exclusion work together to avoid unintentionally ingesting more logs than intended.

Setting the Filter for Log Router of _Default

As mentioned earlier, every Google Cloud project automatically includes two default Log Routers: _Default and _Required. The _Required Log Router, which cannot be modified or deleted, is essential for collecting critical logs such as Admin Activity and System Event audit logs. For more detailed information on the _Required Log Router, please visit:

Routing and storage overview | Cloud Logging | Google Cloud

The _Default Log Router, however, presents a potential pitfall. By default, it ingests all logs not captured by _Required. This means that without careful configuration, you might inadvertently incur higher costs due to duplicated log entries. For instance, if you want to segregate logs by region, you'll need to set specific exclusion conditions in _Default to avoid doubling your ingestion fees.

Here’s an example of an effective configuration (OK) versus an ineffective one (NG):

Effective Configuration (OK):

resource "google_logging_project_sink" "default" {
  project                = "my-project"
  name                   = "_Default"
  destination            = "logging.googleapis.com/projects/my-project/locations/global/buckets/log-for-application"
  filter                 = <<EOF
    NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT LOG_ID("externalaudit.googleapis.com/activity") AND NOT LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT LOG_ID("externalaudit.googleapis.com/system_event") AND NOT LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT LOG_ID("externalaudit.googleapis.com/access_transparency")
    EOF
  unique_writer_identity = true
  exclusion {
    name = "Excluded-Tokyo"
    description = "This filter exclude asia-northeast1's log"
    filter = <<EOF
      labels.REGION="asia-northeast1"
      EOF
  }
}

resource "google_logging_project_sink" "tokyo" {
  project                = "my-project"
  name                   = "tokyo-log-bucket"
  destination            = "logging.googleapis.com/projects/my-project/locations/global/buckets/log-for-application"
  unique_writer_identity = true
  filter                 = <<EOF
    labels.REGION="asia-northeast1"
    EOF
}

Ineffective Configuration (NG):

resource "google_logging_project_sink" "default" {
  project                = "my-project"
  name                   = "_Default"
  destination            = "logging.googleapis.com/projects/my-project/locations/global/buckets/log-for-application"
  unique_writer_identity = true
  filter                 = <<EOF
    NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT LOG_ID("externalaudit.googleapis.com/activity") AND NOT LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT LOG_ID("externalaudit.googleapis.com/system_event") AND NOT LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT LOG_ID("externalaudit.googleapis.com/access_transparency")
    EOF
}

resource "google_logging_project_sink" "tokyo" {
  project                = "my-project"
  name                   = "tokyo-log-bucket"
  destination            = "logging.googleapis.com/projects/my-project/locations/global/buckets/log-for-application"
  unique_writer_identity = true
  filter                 = <<EOF
    labels.REGION="asia-northeast1"
    EOF
  
}

Additionally, it’s important to assign the role of Logs View Accessor (roles/logging.viewAccessor) to personnel who need access to view logs ingested into specific Log Buckets.

Summary

In this article, I have outlined filtering methods for the Log Router in Google Cloud, focusing on the use of filter and exclusion parameters. With the growing need for efficient log management and cost optimization, these skills are becoming increasingly important for DevOps engineers. I hope this guide will be valuable in your cloud projects!