OpenCensus w/ Prometheus & Stackdriver

2018–02–13: Since this post was written, the Golang SDK is evolved and the bug mentioned herein is addressed. I’ve written a new post to reflect @JDB’s updates to the library. Please read that post. Thanks

JBD

!

An architect at one of my customers educated me on the existence of OpenCensus. I thought I’d check it out while I was deploying Kubernetes to my Raspberry Pi cluster (see Alex Ellis’ excellent post for that).

OpenCensus is a Google-initiated project “that automatically collects traces and metrics from your app, displays them locally, and sends them to any analysis tool”.

I’m going to show you this working two ways using Golang with Prometheus and with Stackdriver… It’s a small cheat because the difference is effectively a single line change but this demonstrates the utility in this tool.

Setup

PROJECT=[[YOUR-PROJECT]] # Only needed for Stackdriver
DIR="/tmp/OpenCensus"
mkdir -p ${DIR}/go
export GOPATH=${DIR}/go
export PATH=${PATH}:${GOPATH}/bin
go get -u go.opencensus.io/...

OpenCensus

There’s a bug in the Golang SDK that arises when multiple measures are created. For the time being, here’s the Prometheus sample reduced to using a single measure; this is a trivial tweak to the sample so that it works:

You can create this file anywhere you wish under ${DIR}/go/src. Convention is that you create it under github.com/[YOUR-GITHUB] if you have one. But, if you wish, you may simply create a file called prometheus.go under ${DIR}/go/src.

From within that directory:

go run [YOUR-FILENAME].go

Should report:

2018/01/21 16:02:37 Serving at :9999

and then you may curl or browse to the endpoint:

curl localhost:9999/metrics
# HELP opencensus_video_cum processed video size over time
# TYPE opencensus_video_cum histogram
opencensus_video_cum_bucket{le="0"} 0
opencensus_video_cum_bucket{le="65536"} 0
opencensus_video_cum_bucket{le="4.294967296e+09"} 0
opencensus_video_cum_bucket{le="+Inf"} 11
opencensus_video_cum_sum 3.753271169191e+19
opencensus_video_cum_count 11

This (metrics) data is in Prometheus’ exporter format and this means that we can point a Prometheus server to this endpoint. So, let’s create a Prometheus configuration and run a Prometheus server. Please leave your code running.

Prometheus

Prometheus is configured (by convention) with a prometheus.yml file. The following file is a simple configuration. It defines two scrape_configs. The first is for Prometheus to monitor itself (localhost:9090). The second is for Prometheus to monitor the OpenCensus Measure that we just created. I suggest you create this file in the same directory that you created the Go file.

Now, from that directory, run the following Docker command to spin up a Prometheus server that uses this configuration:

docker run \
--net=host \
--publish=9090:9090 \
--volume=$PWD/prometheus.yml:/etc/prometheus/prometheus.yml \
prom/prometheus

You should then be able to browse (don’t use curl for this) to the Prometheus server:

http://localhost:9090

Prometheus “Graph”

To confirm that the Prometheus server is monitoring itself and the OpenCensus endpoint, click “Status” and then “Target” or:

http://localhost:9090/targets

Prometheus “Targets”

This shows that Prometheus is correctly (State==”UP”) monitoring the “opencensus” target (on localhost:9999/metrics) and itself “prometheus” (on localhost:9090/metrics). You may click either of these URLs to see, for example, the OpenCensus metrics:

http://localhost:9999/metrics

OpenCensus “Sample”

The Golang sample includes a go-routine that Measures randomly-sized videos after a random period of milliseconds. Every time you refresh the localhost:9999/metrics, you will see, for example, the opencensus_video-cum_count increase. We’ll use this value to show Prometheus’ in-built graphing.

Return to http://localhost:9090/graph and, start typing “opencensus”. Select “opencensus_video_cum_count”:

Prometheus: Filter “opencensus”

Click “Execute” and choose “Graph”:

Prometheus “opencensus_video_cum_count”

When you are finished you may CTRL-C both the Prometheus server and your Golang sample to terminate them.

Stackdriver

Stackdriver (accounts) must be created through the console. So, we’ll take a different trajectory and use the Google Cloud Console (console.cloud.google.com) to create a project and then create a Stackdriver account. We’ll then change a few lines in the Golang sample, rerun it and show the Measures being reported to Stackdriver.

Create a Google Cloud Platform project:

https://console.cloud.google.com/projectcreate

Google Cloud Console ‘[[YOUR-PROJECT]]’

And then create a Stackdriver account:

https://console.cloud.google.com/monitoring?project=[[YOUR-PROJECT]]

Once the Stackdriver console is ready, select “Resources” and “Metric Explorer”:

Stackdriver: Metrics Explorer

This time, we’re going to use the OpenCensus Stackdriver sample as the basis for our code. We will revise the Golang imports (“stackdriver” instead of “prometheus”) and create a stackdriver.NewExporter instead of a prometheus.NewExporter. The Stackdriver exporter needs to be configured with our GCP Project ID:

exporter, err := stackdriver.NewExporter(stackdriver.Options{
  ProjectID: "[[YOUR-PROJECT]]",
 })

NB The code for Measures and Views remains unchanged which is what we’d expect.

There’s one other difference between the Prometheus and Stackdriver models. Prometheus is a poll-based solution. Prometheus expects to poll endpoints (e.g. localhost:9999) for Metrics data. Stackdriver is a push-based solution.

So, we’ll also remove the http server used in the Prometheus sample, that provides metrics on-demand, and have the Stackdriver sample run for some period of time Recording Measures to Stackdriver as they’re generated.

The code below runs for 15 minutes:

NB Don’t forget to replace the value for ProjectID in line #18 with your Google Cloud Platform Project ID.

As before, you may run the sample:

go run [YOUR-FILENAME].go

Once the Stackdriver sample is running, return to the Stackdriver console, refresh the page and search for a Metric beginning “OpenCensus”:

Stackdriver Metrics Explorer “OpenCensus”

And, after sufficient time has passed for some metrics to become available, click refresh and you should see data on the graph:

Stackdriver “OpenCensus/video_cum”

Conclusion

OpenCensus provides a compelling solution for instrumenting your code for Metrics and Traces (we did not cover tracing here). The solution supports multiple languages (including Golang) and supports Prometheus and Stackdriver for monitoring. It also supports Zipkin and Stackdriver for tracing. Check it out!

As always, feedback is welcome.