Enriching Prometheus metrics with exemplars for easier observation of a distributed system (Part 2)

Using OpenTelemetry tracing (instead of Zipkin/Brave) with the new OpenTelemetry collector in Jaeger

In Part 1 we looked at how to add exemplars to our metrics using Spring-Cloud-Zipkin which uses the brave tracing format. In this post, we are going to look at replacing with OpenTelemetry tracing format and use the new OpenTelemetry collector in the latest version on Jaeger (1.36.0)

OpenTelemetry

OpenTelemetry is an open source project and unified standard for service instrumentation, or a way of measuring performance. Sponsored by the Cloud Native Computing Foundation (CNCF), it replaces OpenTracing and OpenCensus. The goal is to standardize how you collect and send telemetry data to a backend platform. The OpenTelemetry project consists of specifications, application programming interfaces (APIs), libraries, and integrations, including software development kits (SDKs) for various languages like Java, Go, and Python. It also defines a centralized collector service that you can use for collecting telemetry data from your applications and services. It includes exporters to send that data to the observability platform that you choose.

— https://www.cncf.io/blog/2021/08/06/what-is-opentelemetry-and-why-is-it-the-future-of-instrumentation/

Setup

1. Update Jaeger to use 1.36.0

Since Jaeger 1.35.1, Jaeger now has native support for receiving OpenTelemetry traces. You can read more here: https://medium.com/jaegertracing/introducing-native-support-for-opentelemetry-in-jaeger-eb661be8183c

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
spec:
  strategy: allInOne
  allInOne:
    image: jaegertracing/all-in-one:1.36.0
    options:
      collector:
        otlp:
          enabled: true

— Note: The otlp collector needs to be enabled. All configuration options can be found here: https://www.jaegertracing.io/docs/1.36/cli/

2. Update build.gradle to exclude spring-cloud-sleuth-brave and import spring-cloud-sleuth-otel-autoconfigure and opentelemetry-exporter-otlp-trace

repositories {
   mavenCentral()
   maven {
      url "https://repo.spring.io/snapshot"
   }
   maven {
      url "https://repo.spring.io/milestone"
   }
   maven {
      url "https://repo.spring.io/release"
   }
}

dependencyManagement {
   imports {
      mavenBom "org.springframework.cloud:spring-cloud-dependencies:2021.0.3"
      mavenBom "org.springframework.cloud:spring-cloud-sleuth-otel-dependencies:1.1.0-M6" // Still an experimental project: https://github.com/spring-projects-experimental/spring-cloud-sleuth-otel
   }
}
dependencies {
   ...
   implementation("org.springframework.cloud:spring-cloud-starter-sleuth") {
      exclude group: 'org.springframework.cloud', module: 'spring-cloud-sleuth-brave'
   }
   implementation "org.springframework.cloud:spring-cloud-sleuth-otel-autoconfigure"
   implementation 'io.opentelemetry:opentelemetry-exporter-otlp-trace:1.14.0'
   ...
}

— Note: At time of writing, spring-cloud-sleuth-otel is experimental. With the release of Spring Boot 3.0 Sleuth will be replaced with Micrometer Tracing (https://github.com/micrometer-metrics/tracing/)

3. Update your application.yml to export to the Jaeger OpenTelemetry collector

spring:
  application:
    name: exemplar-demo
  sleuth:
    otel:
      exporter:
        otlp:
          endpoint: http://simplest-collector.default.svc.cluster.local:4317
      config:
        trace-id-ratio-based: 1.0

— Note: spring.cloud.sleuth.otel.config.trace-id-ratio-based needs to be set to 1.0if you want 100% of your requests to be sampled and sent to Jaeger.

Your log line format will have changed to include[spring.application.name, traceId, spanId]

2022-07-25 12:08:13.340  INFO [exemplar-demo,b61ea677596d469a969f4863c55eeb4b,f599f0f0b479c5be] 87949 --- [nio-8080-exec-1] c.g.exemplardemo.HelloWorldController    : Hello world

Sample pull request

A sample pull request showing all the changes needed is available here.

Conclusion

Your traces will still be available in Jaeger and your exemplars in Prometheus, but now they will be using the OpenTelemetry standard.

Sources

OpenTelemetry

Spring Cloud Sleuth OTel Reference Documentation

Introducing native support for OpenTelemetry in Jaeger