Centralised audit logs in Google Cloud, the new way: Log Analytics

Logging is a critical component of any cloud environment, but it can be challenging to manage all the data generated by your applications and infrastructure. Fortunately, Google Cloud offers an built-in logging platform that allows you to aggregate your logs, and stream them to BigQuery, Pub/Sub or storage.

About two years ago I configured centralised audit logging for the company I worked for, and wrote a blog about it. The goal was to ensure traceability and, at the same time, enable all the analytics one could brew out of such detailed log information, stored in one place. Now I work for a different company, and thought it would be a wise thing to set up here, too.

The usual way to enable analytics on your Google Cloud logs is to sink the logs from different GCP sources to BigQuery. It’s a feature natively available in Google Cloud logging, and all you need to do is to create a dataset, point a logging sink to it and give the logging sink service account roles/bigquery.dataEditor on the dataset.

I was about to open a PR with my terraform (a simplified version of this module since I don’t need VPC SC compatibility) when I saw this:

Log Analytics? What is Log Analytics?

According to the article:

[Log Analytics is] a capability that allows you to search, aggregate and transform all log data types including application, network and audit log data

And it’s powered by BigQuery! Does this mean.. what I think it means? Yes!

You can set up centralised logging in Google Cloud, with the analytical power of BigQuery, without copying the data into BigQuery!

Now, for me, the aggregate part of that statement above is of particular interest. Aggregation for logs means streaming them from all different sources in your systems into a single location, with the ability to apply filters, exclusions, redactions etc. I want to aggregate logs from the organization, as well as from specific folders, into one location.

Diagram of aggregated logging in Google Cloud, with logs from the organization ‘my-organization’ and 2 folders, ‘My Audited Folder’ and ‘Prod Folder’ sinked into logging buckets in the ‘my-audit-logs’ project

So how do I set this up?

1. Create the log bucket where you will stream your logs
2. Create the logging sink(s)
3. Enjoy!

Below you will find instructions on how to do this in the GCP Console, via gcloud, and using Terraform.

Creating a log bucket and enabling Log Analytics

UPDATE February 2024: In early 2023, when this article was first written, you had to create the log buckets manually. Now, it can be done in Terraform! I am leaving the UI and gcloud instructions here, but Terraform is much better :) Skip to the Terraform the log buckets and the logging sink (or sinks) section for the Terraform code.

First, create the log bucket for your logs (skip this section if you use Terraform):

Creating a new logging bucket in the Google Cloud console, with Log Analytics enabled and a BigQuery dataset linked to the bucket. The bucket will contain aggregated logs from the prod folder

In the console:

1. Select the project where you want to store the logs
2. Go to Logging -> Log Storage -> Create Log Bucket
3. Give the bucket a meaningful name
4. Click Upgrade to Log Analytics and, optionally, Create a new BigQuery dataset (this does not affect the analytics functionality)
5. Select the region: currently global, us, eu, us-central1, us-west1, and asia-south1 are available
6. Click Next
7. Enter the log retention period in days
8. Click Create Bucket

Using gcloud:

# Create new logging bucket with analytics enabled
gcloud logging buckets create audit-folder-logs --location=global \
       --enable-analytics \
       --retention-days=90 \
       --project=my-audit-logs

# Optional - link a BigQuery dataset
gcloud logging links create audit_folder_logs --bucket audit-folder-logs \
       --location=global \
       --project=my-audit-logs

Terraforming the log buckets and the logging sink (or sinks)

Setting up the logging sink(s) requires the following pieces of terraform:

1. A logbucket with Log Analytics enabled
2. The sink itself: google_logging_folder_sink, google_logging_organization_sink or google_logging_project_sink
3. IAM permissions for the sink identity to write logs to the destination

Here is an example for a folder-level sink setup for audit logs. This creates a sink for My Audited Folder folder, which is a top-level folder inside the GCP org.

# Find the folder number that you want to sink logs from
data "google_active_folder" "audited_folder" {
  display_name = "My Audited Folder"
  parent       = data.google_organization.org.name
}

# Create the log bucket with Log Analytics enabled
resource "google_logging_project_bucket_config" "audited_folder_logbucket" {
  project          = "my-audit-logs"
  location         = "global"
  enable_analytics = true
  bucket_id        = "audit-folder-logs"
}

# Creare a logging sink for the folder
resource "google_logging_folder_sink" "audited_folder" {
  name             = "audited-folder-logs"
  description      = "Sink for My Audited Folder, including all children"
  folder           = data.google_active_folder.audited_folder.name
  destination      = "logging.googleapis.com/${google_logging_project_bucket_config.audited_folder_logbucket.id}"
  include_children = true

  filter = "protoPayload.@type = type.googleapis.com/google.cloud.audit.AuditLog"
}

# Allow each sink's service account to write logs into the audit logs project
resource "google_project_iam_member" "folder-log-writer" {
  project  = "my-audit-logs"
  role     = "roles/logging.bucketWriter"
  member   = google_logging_folder_sink.audited_folder.writer_identity
}

What to do with your logs

Once the logs are sinked to the designated logging buckets, their view is available in Log Analytics, complete with BigQuery-like schemas

Now that your logs are all nicely streamed into a central place, there are a few things you can do with them:

1. Organization-wide log-based security alerts

One of the coolest things you can do with your centralised logs in GCP is setting up log-based alerts. For example, you might want to alert when a folder- or organization-wide IAM role is assigned to a service account.

To create such alert in Terraform, use the following:

# Create a log-based metric that sniffs the folder logging bucket 
resource "google_logging_metric" "folder_iam_changes_for_serviceaccounts" {
  project     = "my-audit-logs"
  name        = "folder_iam_changes_for_serviceaccounts"
  bucket_name = "projects/my-audit-logs/locations/global/buckets/audit-folder-logs"
  filter      = "protoPayload.methodName=\"SetIamPolicy\" protoPayload.resourceName=~\"folders/\" protoPayload.serviceData.policyDelta.bindingDeltas.action=\"ADD\" protoPayload.serviceData.policyDelta.bindingDeltas.member=~\"serviceAccount\""
  description = "Find when afolder-level IAM role is assigned to a service account"
  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
  }
}

# Set up a policy to alert every time a folder-level IAM role is assigned to a service account
resource "google_monitoring_alert_policy" "org_iam_changes_for_serviceaccounts" {
  project      = "my-audit-logs"
  display_name = "Alert when a folder-level IAM role is assigned to a service account"
  combiner     = "OR"
  conditions {
    display_name = "folder-level IAM role assigned to SA"
    condition_threshold {
      filter          = "metric.type=\"logging.googleapis.com/user/folder_iam_changes_for_serviceaccounts\""
      duration        = "0s"
      comparison      = "COMPARISON_GT"
      threshold_value = "0"
    }
  }

2. Easier debugging

If you’ve ever run data transformation jobs on Composer with data distributed across different GCP project and VPC SC perimeters, you’ll know how tricky it is to find where the access error occurs. Now, you can search throughout all your logs — with logging queries or SQL. Nice.

3. Resource usage and cost analytics

This is not new — people have been building BigQuery cost dashboards based on the totalQueriedBytes and totalBilledBytes fields in BQ access logs for years. Now, however, you don’t need to sink those logs to BQ in order to get access to such analytics.

4. Incident resolution

Set up log sinks to stream all your application logs as well as audit logs to a central location, and you will have a wonderful source of information on everything that’s going on in your systems. When an incident inevitably occurs, you will be able to trace events across all of your cloud resources and applications, using SQL, to get to the root cause. Lovely.