Request Annotation with Cloud Audit Logging and Monitoring on GCP

salmaan rashid
Jan 18 · 3 min read

A simple way to embed/associate audit logged metadata to GCP API calls.

Or in other words, whenever you make any GCP API call like GCS, PubSub, you can automatically attach metadata to the call which will show up in audit logs. One usecase is to attach information about why the request is being made. For example, if you run a SaaS service that uses GCP APIs and you make calls to services on behalf of your customers, you can attach information to that call indicating who it was made for.

Also included in this repo is the basic ‘helloworld’ for OpenCensus. With this, you can also emit a metric/counter to any exporter OpenCensus supports that indicates who the call is for. Basically, you can have a metric that shows requests/second or any other metric per customer.

Logging

Sending optional metadata to any GCP API call is actually already there but just not advertized.

Simply send the value as a string within theX-Goog-Request-Reason header.

Thats is.. Just use that header.

Infact, Cloud libraries already supports that parameter:

In golang, when you initialize the client, simply define option.WithRequestReason():

Since these entries show up in audit logs, first enable them:

Once you send in the header, take a look at the structured payload data in the audit logs:

From there, you can even define Log to Metrics as a counter value.

Update 1/31/21: You can also export the logs to BigQuery and filter on the customer

For the really impatient, use curl:

The protos defining this parameter is in audit_log.proto: RequestMetadata.request_attributes attribute_context.proto Request.reason

One thing to point out is each client you initialize will reuse the same reason. Meaning, if you initialize a client, set the reason and then use the client for N API calls for different customer, they will all be attached to the same reason. You can override per rpc but its a bit awkward to setup. The following is one implementation for PubSub using gRPC: per RPC Request Reason

Monitoring

Similar to logging events per customer, a SaaS provider may also want to emit metrics per customer. In the following trivial sample, we will emit a counter simulating a customer request handled and then render the data using prometheus and Grafana.

The snippet is nothing new, its just the ‘helloworld’ of Opencensus.

Code provided in monitoring/main.go will run numerous go routines for all customers and then 3minutes in, a spike in a single customer requests.

To use

Start prometheus and grafana:

Launch the Grafana console, (default username/password is admin/admin).

Configure grafana for a prometheus datasource:

then graph the metric apps_gcs_requests using rate():

Note tht there was a sudden spike in requests sent in by customer-G

Thats about it…just to let you know about this variable.

if interested in grpc metrics, see

gRPC per method observability with envoy, Istio, OpenCensus and GKE

Google Cloud - Community

Google Cloud community articles and blogs