OpenTracing, Zipkin and Jaeger for backend in Golang

According to “Google Dapper” a Large-Scale Distributed Systems Tracing Infrastructure, there are some more extensive open source projects can help us to using it within very convenient steps.

What does “Google Dapper” going to solve?
We do need trace latency and root-cause of frontend and backend services even errors. Dapper planed to solve tracing problem of remote procedure calls and then summary to trace tree as image left part to right part.

And then OpenTracing comes in. OpenTracing implements APIs and interfaces base on “Dapper” to help programers add trace code with low coupling library and produce public format tracing data. After all we can get tracing result as picture.

The OpenTracing Data Model

Causal relationships between Spans in a single Trace


        [Span A]  ←←←(the root span)
            |
     +------+------+
     |             |
 [Span B]      [Span C] ←←←(Span C is a `ChildOf` Span A)
     |             |
 [Span D]      +---+-------+
               |           |
           [Span E]    [Span F] >>> [Span G] >>> [Span H]
                                       ↑
                                       ↑
                                       ↑
                         (Span G `FollowsFrom` Span F)

Common function calls

def top_level_function():
    span1 = tracer.start_span('top_level_function')
    try:
        . . . # business logic
    finally:
        span1.finish()
def function2():
    span2 = get_current_span().start_child('function2') \
        if get_current_span() else None
    try:
        . . . # business logic
    finally:
        if span2:
            span2.finish()

In fact, with OpenTracing’s workout, we did can start our tracing tasks, but there are “Zipkin” and “Jaeger” to be introduced.

Zipkin, it is a distributed tracing system too, has dependency-free library, and spring-boot server. But could be roll as a “Tracer” to OpenTracing.

// All data are recorded against the same endpoint, associated with your service graph
localEndpoint = Endpoint.newBuilder().serviceName("tweetie").ip("192.168.0.1").build()
span = Span.newBuilder()
    .traceId("d3d200866a77cc59")
    .id("d3d200866a77cc59")
    .name("targz")
    .localEndpoint(localEndpoint)
    .timestamp(epochMicros())
    .duration(durationInMicros)
    .putTag("compression.level", "9");

// Now, you can encode it as json
bytes = SpanBytesEncoder.JSON_V2.encode(span);

Zipkin supply lots of modules to fit sorts of platform and programing languages supported includes AWS, GCP, Azure, Java, .NET, Node.js, Go, Fingle(Twitter rpc), Python, Spark, Php, js. Zipkin-UI support web interface for standard query support.

For example, with Docker-Zipkin and Zipkin .Net client Lib, we can illustrates well done job by Zipkin.

## to start zipkin docker and UI
$ docker-compose -f docker-compose.yml -f docker-compose-ui.yml up
## start compiled .net example frontend and backend web server
dotnet zipkin4net-master/Examples/aspnetcore/frontend.dll
dotnet zipkin4net-master/Examples/aspnetcore/backend.dll

with Zipkin library enhanced server/service

Jaeger\ˈyā-gər\, inspired by Dapper and OpenZipkin, is a distributed tracing system released as open source by Uber Technologies. Uber Engineering starts use Zipkin and evolving new distributed tracing system for high scalability, to serve thousands of micro-services. So Jaeger extends more complex architecture for larger scale of requests and performance.

In particular, the Zipkin model did not support two important features available in the OpenTracing standard and Jaeger client libraries: a key-value logging API and traces represented as more general directed acyclic graphs rather than just trees of spans.

example image from LightStep [x]PM (Product)

When really steps in, there still some better choice, good example from the post “Tracing http request in go with opentracing”. Following are some digests when implement to backend server with Golang.

Entry Point (Service backend)

// Jaeger tracer can be initialized with a transport that will
// report tracing Spans to a Zipkin backend
// zipkinURL point to server
transport, err := zipkin.NewHTTPTransport(
      *zipkinURL,
      zipkin.HTTPBatchSize(1),
      zipkin.HTTPLogger(jaeger.StdLogger),
)
if err != nil {
      log.Fatalf("Cannot initialize HTTP transport: %v", err)
}
// create Jaeger tracer
tracer, closer := jaeger.NewTracer(
      "TracerName",
      jaeger.NewConstSampler(true), // sample all traces
      jaeger.NewRemoteReporter(transport, nil),
)
// Close the tracer to guarantee that all spans that could
// be still buffered in memory are sent to the tracing backend
defer closer.Close()

Other just useful method for tag , log and error.

// Adds a tag to the span. 
// 
// Tag values can be numeric types, strings, or bools. 
// The behavior of other tag value types is undefined at the 
// OpenTracing level.
SetTag(key string, value interface{}) Span
// LogFields is an efficient and type-checked way to record
// key:value 
// logging data about a Span, though the programming interface 
// is a little more verbose than LogKV()
LogFields(fields ...log.Field)
// set error flag of span
ext.Error.Set(span, true)

We don’t need to care about span relations for HTTP requests, and can get good enough tracing event and detail by Zipkin-UI. Just remember to take care about privacy or secret data leaks if hosted at public domains.