Implement distributed tracing with Jaeger & Opentelemetry on Kubernetes

6 min readAug 18, 2022

Kubernetes and the microservice architectures that it enables can create very efficient and scalable systems. But problems arise when one of these microservices develops performance problems. Typically, we first notice that response times from our customer-facing services are getting longer and longer. The problem might be with one of the backend services, or perhaps a database that is beyond its optimal capacity. To discover the root of our problem, we need to implement distributed tracing.

This Blog is about, how To Implement Distributed Tracing for your application running on Kubernetes using Open-telemetry & Jaeger.
We will brief here, how to enable API trace with an easy setup in you application.

In the Following diagram, I will show you how the flow will be between your application, OpenTelemetry, and Jaeger.

Before starting this blog,you can go through the below doc for More details

OpenTelemetry Reference Architecture

Jaeger Components

Install Jaeger

Jaeger is a distributed tracing system.

We will first install jaeger on k8s cluster using jaeger Helm charts: Link.

Step: 1 First, install the cert-manager on the k8s cluster

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml

Step: 2 Clones the code

git clone https://github.com/jaegertracing/helm-charts.git 
cd helm-charts/charts/

Step : 3 Install the jaegerhelm charts

helm install jaeger ./jaeger -n jaeger

Note :As per Jaeger documentation, for large scale production deployment the Jaeger team recommends Elasticsearch backend over Cassandra, as such the default backend may change in the future and it is highly recommended to explicitly configure storage.

In our case we are using ES as backend stroage for Jaeger.

Step : 4 Once you will install the helm charts verify the component running fine for jaeger or not.

$ kubectl get po -n jaegerNAME                                READY   STATUS    RESTARTS   AGE
elasticsearch-master-0              1/1     Running   0          5d
elasticsearch-master-1              1/1     Running   0          4d23h
elasticsearch-master-2              1/1     Running   0          5d1h
jaeger-agent-48tzz                  1/1     Running   0          5d
jaeger-agent-5wr7z                  1/1     Running   0          5d1h
jaeger-agent-b2nfz                  1/1     Running   0          4d23h
jaeger-collector-659cc57fdd-gqh6l   1/1     Running   0          5d
jaeger-query-68c6f979cd-g9knl       2/2     Running   0          5d####################$ kubectl get  svc  -n jaeger
NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                         AGE
elasticsearch-master            ClusterIP   10.241.3.227   <none>        9200/TCP,9300/TCP                               20d
elasticsearch-master-headless   ClusterIP   None           <none>        9200/TCP,9300/TCP                               20d
jaeger-agent                    ClusterIP   10.241.2.166   <none>        5775/UDP,6831/UDP,6832/UDP,5778/TCP,14271/TCP   20d
jaeger-collector                ClusterIP   10.241.1.236   <none>        14250/TCP,14268/TCP,14269/TCP                   20d
jaeger-query                    ClusterIP   10.241.2.142   <none>        80/TCP,16685/TCP,16687/TCP                      20d

Open the Jaeger UI :

$kubectl port-forward svc/jaeger-query 8091:80 -n jaeger

Now open Jaeger UI link on browser:

$http://localhost:8091

Now We will install Opentelementry Operator.

Install Opentelementry Operator

The Helm chart installs OpenTelemetry Operator in Kubernetes cluster. The OpenTelemetry Operator is an implementation of a Kubernetes Operator. At this point, it has OpenTelemetry Collector as the only managed component.

Github Link : opentelemetry-helm-charts/charts/opentelemetry-operator at main · open-telemetry/opentelemetry-helm-charts

Step : 1 Create the namespace for the OpenTelemetry Operator and the secret :

kubectl create namespace opentelemetry-operator-system

Step : 2 Now install helm charts opentelemetry-operator

helm install --namespace opentelemetry-operator-system \ 
     my-opentelemetry-operator open-telemetry/opentelemetry-operator

The Operator itself doesn’t mean we have an opentelemetry-operator working. This is where the Custom Resource Definitions come into play. We need to create a resource describing the opentelemetry-collector instance we want the Operator to manage.

Step : 3 verfiy the resoruce running :

$kubectl get po -n opentelemetry-operator-system 2 3NAME                                                         READY   STATUS    RESTARTS   AGE 4opentelemetry-operator-controller-manager-667b47d8cd-vc4d2   2/2     Running   0          5d1h

Install OpenTelemetry Collector

Once OpenTelemetry operator installed , we can deploy OpenTelemetry-collector .

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

See OpenTelemetry website for more details about the Collector

The Collector can be deployed as one of four modes:
a. Deployment

b. DaemonSet

c. StatefulSet

d. Sidecar

Note: The default mode is Deployment. In open telemetry-operator,they introduce the benefits and use cases of each mode as well as give an example for each.

In our case, we have deployed the Otel collector as Daemon set , with the below configure.

Step : 1 Create otel-collector namespace :

kubectl create ns otel-collector

Step : 2 Create k8s manfiest for deployment of otel collector :

opentelemetry-collector-demonset.yaml :

---
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector-daemonset
  namespace: otel-collector
#  namespace: opentelemetry-operator-system 
spec:
  mode: daemonset
  config: |
    receivers:
      # Make sure to add the otlp receiver.
      # This will open up the receiver on port 4317
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"  
    processors:
    extensions:
      health_check: {}
    exporters:
      jaeger:
        endpoint: jaeger-collector.jaeger.svc.cluster.local:14250
        tls:
          insecure: true
      prometheusremotewrite:
        endpoint: "http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090/api/v1/write"
        external_labels:
          cluster: prod
      logging:service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [jaeger]
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [prometheusremotewrite, logging]

Step : 3 Now create & apply above manifest

kubectl apply -f opentelemetry-collector-demonset.yaml

Step : 4 Just verfiy all otel collector resource are fine or not :

$ kubectl get po -n otel-collectorNAME                                       READY   STATUS    RESTARTS   AGE
otel-collector-daemonset-collector-6qdkq   1/1     Running   0          5d1h
otel-collector-daemonset-collector-dxrdt   1/1     Running   0          5d
otel-collector-daemonset-collector-ql7km   1/1     Running   0          5d2h$ kubectl get svc  -n otel-collectorNAME                                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
otel-collector-daemonset-collector              ClusterIP   10.241.2.182   <none>        4317/TCP,4318/TCP   12d
otel-collector-daemonset-collector-headless     ClusterIP   None           <none>        4317/TCP,4318/TCP   12d
otel-collector-daemonset-collector-monitoring   ClusterIP   10.241.0.205   <none>        8888/TCP            12d

Now all resources are running fine & verfied.

we can use the below endpoint to pushing the metric & traces to otel collector :

Otel-collector endpoint

otel-collector-daemonset-collector.otel-collector.svc.cluster.local:4317

Adding Instrumentation into application code :

Once the above component & the setup are done, We need to Add the Opentelemertry SDK to our application code to generate a trace & parse it to otel collector.

Opentelemetery provides a library for different languages (java, python,golang etc.)

Need to add SDK for both , follow official documentation

You can follow above documentation & add otel SDK into y our application.

Let Test the Setup

Let test the Setup with sample app :

The application is based on an example at GitHub — rbaumgar/otelcol-demo-app: Quarkus demo app to show OpenTelemetry with Jaeger.

Deploying a sample application monitor-demo-app end expose a route:

test-app.yaml :

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: otelcol-demo-app
  name: otelcol-demo-app
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otelcol-demo-app
  template:
    metadata:
      labels:
        app: otelcol-demo-app
    spec:
      containers:
      - image: quay.io/rbaumgar/otelcol-demo-app-jvm
        imagePullPolicy: IfNotPresent
        name: otelcol-demo-app
        env:
        - name: OTELCOL_SERVER
          value: "http://otel-collector-daemonset-collector.otel-collector.svc.cluster.local:4317"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: otelcol-demo-app
  name: otelcol-demo-app
  namespace: test
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
    name: web
  selector:
    app: otelcol-demo-app
  type: LoadBalancer

You can add an environment variable with the name OTELCOL_SERVER if you need to specify a different url for the OpenTelemetry Collector. Default: http://my-otelcol-collector:4317

Test Sample Application

Check the router url with /hello and see the hello message with the pod name. Do this multiple times.

$ curl $URL/hello
hello 
$ curl $URL/sayHello/demo1
hello: demo1
$ curl $URL/sayRemote/demo2
hello: demo2 from http://endpoint/

Now Go on Jaeger UI ,Reload by pressing F5.
Under Service select my-service.
Find Traces…

Conclusion :

In this way, you can easily set up distributed tracing for monitoring, network profiling, and troubleshooting the interaction between components in modern, cloud-native, microservices-based applications with Jaeger, Opentelemetery on k8s.

Hope this blog was helpful for you & please open to share your feedback.