Make Your Service Observable

Distributed Services with Go — by Travis Jeffery (46 / 84)

👈 Three Types of Telemetry Data | TOC | What You Learned 👉

Let’s make your service observable by adding metrics, structured logs, and traces. When you deploy your services to production, you’ll usually configure your metrics, structured logs, and traces to go to external services like Prometheus,[36] Elasticsearch,[37] and Jaegar. To keep things simple, we’ll just log our observability pieces to files and see what the data looks like.

OpenTelemetry[38] is a Cloud Native Computing Foundation (CNCF) project that provides robust and portable APIs and libraries that we can use for metrics and distributed tracing in our service. (OpenCensus and OpenTracing merged to form OpenTelemetry, which is backward-compatible with existing OpenCensus integrations.) OpenTelemetry’s Go gRPC integration supports traces but not metrics, so we’ll use the OpenCensus libraries in our service since OpenCensus’s gRPC integration supports them both. Unfortunately, neither OpenTelemetry nor OpenCensus support logging yet. OpenTelemetry should support logging at some point — a special interest group[39] is planning OpenTelemetry’s logging specification. In the meantime, we’ll use Uber’s Zap logging library.[40]

Most Go networking APIs support middleware, so you can wrap request handling with your own logic. This is where I recommend beginning making your service observable by wrapping all requests with metrics…