Jaeger Tracing On Kubernetes with .NET Core
I wanted to explore distributed tracing on our Kubernetes cluster with Jaeger so I setup a simple proof of concept. This tutorial demonstrates how to setup the Jaeger all-in-one deployment on a non-prod cluster with traces generated from a .NET core application. I included a quick overview of Jaeger at the end of this article.
Kubernetes Deployment
You can get the all-in-one deployment yaml from here to which I made a few modifications. I still wanted some degree of security despite this being in our non-prod cluster, so instead of exposing a load balancer jaeger-query service I wrote an ingress that uses basic auth.
There are three components to Jaeger that are aggregated into one pod for the all-in-one deployment. The Jaeger Collector which receives traces from Jaeger agents and validates, indexes, performs transformations, and stores them. The Jaeger Agent runs as a network daemon or a sidecar and distributes traces to the collector, and finally Jaeger Query which retrieves traces from storage and displays them in the UI. These components should be split up in a production environment, and each one can be configured separately.
The absence of a service type will result in a default ClusterIP service, and we can expose jaeger-query (the Jaeger UI) via a Traefik ingress. Notice that we also include the argument query.base-path=/jaeger in the Jaeger deployment so that we can use the ingress path myhost.com/jaeger . You can read more about that here: https://github.com/jaegertracing/jaeger-ui/issues/42.
.NET Core App Jaeger Instrumentation
You need two NuGet package to use Jaeger.
OpenTracing.Contrib.NetCore
JaegerWe will use dependency injection to get an instance of an ITracer
We create an ITracer from a Jeager.Configuration. This allow us to tweak the Jaeger service name, agent host, agent port, sampler type, and sampler host/port via environment variables. The following are the environment variables required for the service deployment.
JAEGER_SERVICE_NAME: my-service-name
JAEGER_AGENT_HOST: jaeger-agent.monitoring.svc
JAEGER_AGENT_PORT: 6831
JAEGER_SAMPLER_TYPE: const
JAEGER_SAMPLER_MANAGER_HOST_PORT: jaeger-agent.monitoring.svc:5778We rely on our cluster DNS to resolve the hostname jaeger-agent.monitoring.svc .We can use an ITracer to create spans . Spans represent a unit of work or time in your application. They can be nested and encapsulate tags and logs.
Use DI in your controller to access the ITracer singleton public MyController(ITracer tracer) and create a span with a span builder. Spans implement IDisposable and will automatically be marked as finished.
Or you can manually start and stop a span with the following. We can also make use of Tags which are a way to organize spans and provide additional metadata.
That’s all that’s required to setup tracing. You should see start seeing them show up as you make requests to your services.
Quick Jaeger Overview
Jaeger is a CNCF distributed tracing system released by Uber that enables debugging, monitoring, and analysis of your services, and is based on OpenTracing api. It uses distributed context propagation, which essentially is the basis of distributed tracing, to assign metadata to requests as they propagate through your system. Here are some advantages of using Jaeger:
- Performance and latency analysis
- Service Dependency analysis; you can view a DAG of your system in the UI
- Logs associated with each span
- Organization of logs into calling hierarchy
- Cost attribution
- Useful UI and libraries in Go, Node, C#, and others.
- Different levels of sampling can be configured to reduce load: constant, probabilistic, rate limiting, and remote.
Other Cool Things
- Traefik has a Jaeger integration that you can find here: https://docs.traefik.io/configuration/tracing/
- Can output prometheus metrics on number of spans started, finished, etc.
- Working autogenerated DAGs
