Distributed Tracing with Open telemetry and Zipkin

Distributed tracing is a method for tracking requests as they flow through a complex system made up of multiple servers and services. It enables developers to understand the behavior of their distributed systems by providing visibility into the path of a request as it traverses different components of the system.

Distributed tracing works by instrumenting the code of each component in the system to generate a trace that captures timing and other metadata related to each request. These traces are then aggregated and visualized in a way that makes it easy to identify performance issues and diagnose errors.

The benefits of distributed tracing include:

1. Improved system performance: By providing a detailed view of the behavior of each request, distributed tracing can help developers identify and eliminate bottlenecks in the system.
2. Faster problem resolution: When something goes wrong in a distributed system, it can be difficult to pinpoint the source of the problem. Distributed tracing provides developers with the context they need to quickly identify the root cause of an issue.
3. Better understanding of system behavior: By analyzing trace data over time, developers can gain a deeper understanding of the behavior of their system and make informed decisions about how to improve it.

Overall, distributed tracing is a powerful tool for understanding and improving the performance and reliability of complex distributed systems.

Before we can use OpenTelemetry and OpenZipkin distributed tracing tools in a .NET project, we need to add the necessary NuGet packages. This step is crucial as it ensures that our project has access to the required libraries and dependencies to function properly.

OpenTelemetry is a popular open-source observability framework that can be used for distributed tracing. It provides APIs in multiple languages, including .NET, to instrument applications and generate trace data. OpenZipkin is a distributed tracing system that can be used to collect, analyze, and visualize trace data.

Here’s a step-by-step guide on how to use OpenTelemetry and OpenZipkin in a .NET project:

1. Add the OpenTelemetry and OpenTelemetry.Exporter.Zipkin packages to your project using the NuGet Package Manager ( OpenTelemetry.Extensions.Hosting, OpenTelemetry.Extensions.Hosting, OpenTelemetry.Instrumentation.AspNetCore, OpenTelemetry.Instrumentation.Http, OpenTelemetry.Exporter.Zipkin, OpenTelemetry.Exporter.OpenTelemetryProtocol, OpenTelemetry.Exporter.Console ).
2. In your code, create an OpenTelemetry TracerProviderBuilder and add the ZipkinExporter to export trace data to OpenZipkin.

Project Contents:

1 — Distributed Tracing Extension

public static void AddDistributedTracingServies(this IServiceCollection services, IConfiguration configuration)
    {
        services.Configure<OpenTelemetryViewModel>(configuration.GetSection("OpenTelemetryConfig"));
        var ZipkinUrl = configuration.GetSection("ZipkinConfig")["ZipkinAddress"];
        var OpenTelemetry = configuration.GetSection("OpenTelemetryConfig").Get<OpenTelemetryViewModel>();

        services.AddStackExchangeRedisCache(options =>
        {
            options.Configuration = $"localhost:6379,password=12345,ssl=false,abortConnect=false";
        });
        services.AddSingleton(TracerProvider.Default.GetTracer(OpenTelemetry.ServiceName));
        services.AddScoped<ICounter, CounterService>();

        services.AddOpenTelemetryTracing(tracerProviderBuilder =>
        {
            _ = tracerProviderBuilder
            .AddOtlpExporter(opt =>
             {
                 opt.Protocol = OtlpExportProtocol.HttpProtobuf;
             })

            .AddSource(OpenTelemetry.ServiceName)
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
            .AddService(serviceName: OpenTelemetry.ServiceName, serviceVersion: OpenTelemetry.ServiceVersion))
            .AddHttpClientInstrumentation(options =>
                options.Enrich = (Activity, eventName, rowObject) =>
                {
                    if (eventName == "OnStartActivity" && rowObject is HttpRequestMessage request)
                    {
                        if (request.Method == HttpMethod.Get)
                            Activity.SetTag("GetRequest", "get-request");
                        if (request.Method == HttpMethod.Post)
                            Activity.SetTag("PostRequest", "post-request");
                        if (request.Method == HttpMethod.Put)
                            Activity.SetTag("PutRequest", "put-request");
                        if (request.Method == HttpMethod.Delete)
                            Activity.SetTag("DeleteRequest", "delete-request");
                    }
                })
            .AddAspNetCoreInstrumentation(options =>
            {
                options.Filter = httpContext =>
                    !httpContext.Request.Path.Value.StartsWith("/swagger") &&
                    !httpContext.Request.Path.Value.StartsWith("/_framework") &&
                    !httpContext.Request.Path.Value.StartsWith("/_search") &&
                    !httpContext.Request.Path.Value.StartsWith("/_vs");
            }
             )
            .AddConsoleExporter()
            .AddZipkinExporter(options =>
            {
                options.Endpoint = new Uri(ZipkinUrl);
            });
        });
    }

The AddDistributedTracingServices function is an extension method of IServiceCollection which registers several services and dependencies required for distributed tracing using OpenTelemetry and Zipkin in an ASP.NET Core application.

The AddHttpClientInstrumentation() method adds instrumentation for HTTP client calls, allowing you to capture trace data for incoming and outgoing HTTP requests.

The AddAspNetCoreInstrumentation() method adds instrumentation for ASP.NET Core applications, allowing you to capture trace data for incoming HTTP requests.

The AddZipkinExporter() method adds an exporter to send trace data to a Zipkin server for storage and analysis.

Overall, this function provides an easy way to set up and configure distributed tracing in an ASP.NET Core application using OpenTelemetry and Zipkin.

2 — OpenTelemetry View Model & Zipkin View Model

public class OpenTelemetryViewModel
{
    public string? ServiceName { get; set; }
    public string? ServiceVersion { get; set; }
    public string? ServiceId { get; set; }
}

public class ZipkinViewModel
{
    public string? ZipkinAddress { get; set; }
}

3— Set Activte Middleware

    internal class SetActivteMiddleware
    {
        private readonly IServiceProvider ServiceProvider;
        private readonly RequestDelegate Next;

        public SetActivteMiddleware(RequestDelegate next, IServiceProvider serviceProvider )
        {
            Next = next;
            ServiceProvider = serviceProvider;
        }

        public async Task Invoke(HttpContext context, Tracer tracer)
        {
            using var scope = ServiceProvider.CreateScope();
            var scopedCounter = scope.ServiceProvider.GetRequiredService<ICounter>();

            var count = await scopedCounter.GetCounter();
            if (count > 3)
                count = await scopedCounter.SetCounter();

            using var parent = tracer.StartActiveSpan("Middleware Service 1 is started !!");
            parent.SetAttribute("count.value", count);

            await Next(context);
            using var stopped = tracer.StartActiveSpan("Middleware Service 1 is stopped !!");
        }
    }

This is a middleware component in a .NET application that sets a counter and uses a tracing library (in this case, OpenTelemetry) to track the execution of the middleware. When a request comes in, the middleware creates a new scope for the request and retrieves a counter value from a service. If the count is greater than 3, it sets the count to a new value. It then starts an active span with the tracing library to mark the beginning of the middleware’s execution, passing the count value as an attribute. After the middleware completes its processing, it stops the active span to mark the end of its execution.

4— docker compos file

services:
  openzipkin:
   image: openzipkin/zipkin
   container_name: openzipkin
   ports:
    - 9411:9411

How to use:

1 — We will need to add a few lines of code to the Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddDistributedTracingServies(builder.Configuration);

var app = builder.Build();

app.UseMiddleware<SetActivteMiddleware>();

2 — To implement distributed tracing with Open Telemetry and Zipkin, update the appsettings.json file with their configurations. .

"OpenTelemetryConfig": {
    "ListenerName": "Service1",
    "ServiceName": "Service1.API",
    "ServiceVersion": "1.0.0"
  },
  "ZipkinConfig": {
    "ZipkinAddress": "http://localhost:9411/api/v2/spans"
  },

Result in Zipkin:

Result in Zipkin / dependency

Like of the project in GitHub:

GitHub - odeyAbdulrahman/Distributed.Tracing: Distributed tracing is a technique used to monitor…