Instrumenting Cloud Spanner Go Applications with OpenCensus

This tutorial will show you how to enable the session-related metrics in a demo application with the Cloud Spanner client library in Go. These metrics allow for better monitoring of the session pool and help us to quickly identify session issues:

• max_allowed_sessions: The maximum number of sessions allowed. Configurable by the user.
• num_sessions_in_pool: The number of sessions currently in the session pool, including in-use and unused.
• max_in_use_sessions: The maximum number of sessions in use during the last 10 minute interval.
• get_session_timeouts: The number of get sessions timeouts due to pool exhaustion.
• num_acquired_sessions: The number of sessions acquired from the session pool.
• num_released_sessions: The number of sessions released by users.

The content will include:

• Create a spanner table & a demo web application
• Enable metric views & export to Cloud Monitoring
• Visualizing metrics in Cloud Monitoring

If you are using the Cloud Spanner client library in Java, you might be interested in this article.

Important: OpenCensus is being replaced by OpenTelemetry (currently entering beta), but the migration should not require much change to use the API. Read more here.

Prerequisite

Create a spanner table

Create a table “users” in your spanner database using the following DDL schema:

CREATE TABLE users (
  id STRING(MAX) NOT NULL,
  email STRING(MAX) NOT NULL,
  first_name STRING(MAX) NOT NULL,
  last_name STRING(MAX) NOT NULL,
) PRIMARY KEY (id)

Create a simple web app

This web app supports two APIs:

• GET localhost/users: List all users in the table and the return data is in json format.
• POST localhost/users: Create a new user with submitted form fields.

The code is shown in webapp_enable_oc.go.

Enable metrics exporting

In order to see metric views, you need to decide two things:

1. Which views do we like to see in the backend?
2. Which backend should we export views to?

You can register all or subset of the views defined in the Spanner Client library using register api. In this tutorial, we will use the convenience API from Spanner client library that registers all views.

// Enable all default views.
spanner.EnableStatViews()

Therefore, all metric views will be seen in the backend.

Next, we need to choose which backend we send metrics to. Opencensus supports exporting metrics to many different backends and you can find supported exporters here. In this tutorial, we use Cloud Monitoring (previously, Stackdriver) as an example:

// Set up the stackdriver exporter.
sd, err := stackdriver.NewExporter(stackdriver.Options{
        ProjectID: projectID,
        ReportingInterval: 60 * time.Second,
})
if err != nil {
        log.Fatalf(“Failed to create the StackDriver exporter: %v”, err)
}
defer sd.Flush()
sd.StartMetricsExporter()
defer sd.StopMetricsExporter()

We create a stackdriver exporter and configure it to report every 60 seconds.

Visualize metrics in Cloud Monitoring

First, we need to run the web app:

$ go run webapp_enable_oc.go // The default port is 8080.
// Or, specify a port.
$ go run webapp_enable_oc.go -port=8080

In order to create some traffic, we can use wrk (a HTTP benchmarking tool):

$ wrk -t12 -c200 -d5m http://127.0.0.1:8080/users
Running 5m test @ http://127.0.0.1:8080/users
  12 threads and 200 connections
  Thread Stats Avg Stdev Max +/- Stdev
    Latency 100.74ms 114.93ms 1.50s 87.70%
    Req/Sec 222.75 64.84 430.00 70.69%
  791833 requests in 5.00m, 379.09MB read
  Socket errors: connect 0, read 53, write 0, timeout 0
Requests/sec: 2638.51
Transfer/sec: 1.26MB

This runs a benchmark for 5 minutes, using 12 threads, and keeping 200 HTTP connections open.

Then, you can navigate to Metric explorer in GCP Cloud Monitoring. You should be able to find metrics with this prefix: “custom.googleapis.com/opencensus/cloud.google.com/go/spanner/”. All related metrics should be under this prefix. If you add all metrics and use “sum” aggregator, you will see a diagram that is similar to this:

All session-related metrics

Note that get_session_timeouts is not shown in the diagram because we have not got any timeout error yet.

Each metric has been tagged with some common attributes:

• client_id: Differentiate spanner clients in a single process. For the same database, it starts with “client-1”, “client-2”, and so on.
• database: the name of database.
• instance_id: the name of instance.
• library_version: a date version of google-cloud-go in YYYYMMDD format (spanner is a submodule).

We can use these attributes to group metrics, e.g., in max_in_use_sessions,

tags of max_in_use_sessions

Specially, num_sessions_in_pool has a type attribute that indicates the type of sessions. The type attribute has four values:

• num_in_use_sessions: sessions that are currently in use by users.
• num_sessions_being_prepared: sessions that are being prepared for writes.
• num_read_sessions: read sessions in the session pool.
• num_write_prepared_sessions: write sessions in the session pool.

The sum of them is the total number of sessions in the session pool, including in-use and unused. A stacked area diagram of num_sessions_in_pool is shown as follows:

A stacked area diagram of num_sessions_in_pool

Conclusion

This tutorial introduces how to enable session-related metrics when using Cloud Spanner Go client library. Six OpenCensus metrics have been added to the client library and can be used for debugging session issues. The metrics can be exported to different backends for storage and visualization easily. These metrics enable us to have a better monitoring of the session pool and locate the session issue quickly.