Google Trillian for Noobs (1c)

The Missing Manuals series

Last week I documented what I hope is the simplest possible Trillian personality. Yesterday, I documented adding an inclusion proof. Earlier today, I documented building a gRPC-based client and server for the personality. Here is a small addition that adds metrics (stats) and traces.

OpenCensus Exporter

With the addition of a straightforward configuration for an OpenCensus Exporter using the OpenCensus Agent, we have the ability to configure the Agent to convert incoming stats|traces into a wide selection of 3rd-party services.

Here’s the Basic Personality server configuration:

oc, err := ocagent.NewExporter(
 ocagent.WithAddress(*ocagEndpoint),
 ocagent.WithInsecure(),
 ocagent.WithReconnectionPeriod(10*time.Second),
 ocagent.WithServiceName(serviceName),
)
if err != nil {
 log.Fatal(err)
}
defer oc.Stop()

And here’s the Agent’s configuration to receive incoming stats|traces on :55678 and export the stats to Prometheus (on :9100) and the route the traces to thezipkin service on its port :9411:

receivers:
  opencensus:
    address: ":55678"
exporters:
  jaeger:
    collector_endpoint: "jaeger:14268/api/traces"
  prometheus:
    address: ":9100"
  zipkin:
    endpoint: "zipkin:9411/api/v2/spans"
zpages:
    port: 9999

NB A potential source of confusion. These DNS names are provided by Docker Compose to the services running on the network it creates. These names are accessible to other services within the solution. In this case, the OpenCensus Agent (itself known as opencensus-agent) is able to refer to these other services by their service names (jaefer ,zipkin) but only within the solution. Externally, we need to port map these services onto an available port on localhost.

OpenCensus Agent

The OpenCensus Agent export as a Prometheus Exporter on the host on :9100 so you can query http://localhost:9100/metrics and hopefully see:

We can confirm the OpenCensus Agent by checking its logs:

docker-compose \
--file=deployment/docker-compose.yml \
logs opencensus-agent
Attaching to deployment_opencensus-agent_1
opencensus-agent_1          | {"level":"info","ts":1562357412.0980296,"caller":"config/config.go:490","msg":"Trace Exporter enabled","exporter":"zipkin"}
opencensus-agent_1          | {"level":"info","ts":1562357412.0983222,"caller":"config/config.go:497","msg":"Metrics Exporter enabled","exporter":"prometheus"}
opencensus-agent_1          | 2019/07/05 20:10:13 Running OpenCensus Trace and Metrics receivers as a gRPC service at ":55678"
opencensus-agent_1          | 2019/07/05 20:10:13 Running zPages on port 9999

The OpenCensus Agent is configured (:9999) to show zPages so we can interrogate that endpoint:

/debug/tracez

NB I would have expected to see upstream gRPC traces reflected by the Agent but they’re not. So this is one problem.

Prometheus

The OpenCensus Agent is now a scrape target (via the above Exporter) for the Prometheus server. So the Basic Personality client and server gRPC metrics are exported to the OpenCensus Agent, which exports them as a Prometheus Exporter and they’re then captured by this server:

Zipkin

For some reason — to be explored — Zipkin either isn’t receiving or isn’t able to process the traces routed through the OpenCensus Agent that are generated by the automatic ocgrpc handler.

You can confirm the Zipkin service logs too though these are verbose and not included here:

docker-compose \
--file=deployment/docker-compose.yml \
logs zipkin

Jaeger

When Zipkin wasn’t showing traces, I added Jaeger (another trace monitoring tool) to see whether it was Zipkin, the upstream OpenCensus Agent, or something else. Jaeger is reported its own traces but no others. So, there’s something either in my configuration or the OpenCensus Agent that’s not working:

Conclusion

An ‘amuse bouche’ for integration stats and traces into the Basic Personality. It would be more awesome if it worked perfectly ;-)