Distributed Request Tracing — Spring Boot 3, Micrometer Tracing with OpenTelemetry
The Spring Observability Team has added observability support for Spring Framework 6 and Spring Boot 3. It includes numerous auto-configurations for improved metrics with Micrometer and distributed tracing support with Micrometer Tracing.
In this post, we are going to examine how we can leverage Micrometer Tracing and OpenTelemetry together to instrument a Spring Boot app to enable distributed request tracing. You can read more about distributed request tracing and what role OpenTelemetry plays in that context in my previous article.
I’ve created a sample application with two services to model some service interactions. You can find the source code here. Request path in our sample app would look like this.
- A client places an order through the order service via an HTTP call.
- The order service processes the order and calls the payment service asynchronously over Kafka to handle the payment.
- The payment service listens on the order-events topic to process the payments.
- Upon processing the payment, the payment service calls the order service via an HTTP call to update the order status.
First, let’s see it in action..
Follow the instructions in the README file in the repository to run the necessary components and place a request to the order service by visiting the http://localhost:8081/swagger-ui/index.html#/order-controller/placeOrder.
Then navigate to http://localhost:16686 to access the jaeger UI and examine the request tracing path. You will be able to see the request path in a waterfall diagram like the one below.
If you click on one of the operations, you will be able to find more details about that particular operation.
Interesting stuff, isn’t it?
Let’s explore how easy it is to enable request tracing in a Spring Boot application. As I mentioned earlier, with the latest Spring Boot 3 version, the Spring team has enabled distributed request tracing support through Micrometer Tracing. How can Micrometer Tracing make our lives easier?
Micrometer Tracing
Micrometer Tracing provides a simple facade for the most popular tracer libraries, letting you instrument your JVM-based application code without vendor lock-in. It is designed to add little to no overhead to your tracing collection activity while maximizing the portability of your tracing effort
This is the first level of abstraction we are dealing with when it comes to tracing. Think of it this way: Micrometer Tracing simply provides an interface so it can support multiple tracers.
What is a tracer, you may ask?
Tracer: A library that handles the lifecycle of a span. It can create, start, stop, and report spans to an external system via reporters / exporters.
- https://docs.micrometer.io/tracing/reference/glossary.html
Micrometer Tracing supports widely supported tracers. As of June 2024, we can use either OpenZipikin Brave or OpenTelemetry.
In order to enable tracing with Micrometer Tracing in a Spring Boot application, we need to add a few dependencies to our classpath.
- Spring Boot Actuator provides dependency management and auto-configuration for Micrometer Tracing.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>- In our case, we will be using OpenTelemetry, and hence we’ll be adding micrometer-tracing-bridge-otel. When you add a bridge, the micrometer-tracing library is added transitively.
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>Another advantage of using Micrometer is that a lot of projects have already been instrumented, so we don’t need to write any additional code for request tracing. For instance, all the components we are using for inter-service communication in this example, such as Spring Kafka and HTTP clients, have already been instrumented.
OpenTelemetry
OpenTelemetry is the second level of abstraction we are dealing with in this example. OpenTelemetry is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. It’s also a widely adopted standard that we can utilize to instrument our code to avoid vendor lock-in.
Once we instrument our Spring Boot app to emit OTel spans, then we can export these spans to an array of span stores. There are several ways that you can export these signals to a observability backend of your choice.
- Export directly to a observability backend
- Use an OTel collector in between
- Use OpenTelemetry Protocol(OTLP)
OpenTelemetry Protocol
OpenTelemetry Protocol (OTLP) is a telemetry data delivery protocol that we can leverage to export the telemetry data to an observability backend. This specification defines how OTLP is implemented over gRPC and HTTP. You have to add the following dependency to the classpath in order to export your telemetry data via OTLP.
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>We will be using Jaeger, an open source observability platform originated in Uber. It can consume OTLP natively. You can find the list of vendors who can consume OTLP natively here. As we are using an all-in-one image from Jaeger we will be exposing port 4318 to accept OTLP over HTTP. (port 4317 is for gRPC and 16686 for UI)
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686"
- "4318:4318"
- "4317:4317"In our properties file we will be setting management.otlp.tracing.endpoint to enable HTTP export of telemetry data to Jaeger.
The default URL path for requests that carry trace data is
/v1/traces(for example the full URL when connecting to “example.com” server will behttps://example.com/v1/traces). The request body is a Protobuf-encodedExportTraceServiceRequestmessage.
management.otlp.tracing.endpoint=http://localhost:4318/v1/traces(You can find how Spring Boot will take its opinionated view based on properties and classes available by analyzing OtlpAutoConfiguration.java class.)
Intercepting a request and enabling tracing is an expensive operation with the amount of data we need to capture and the number of network calls. So, in production environments, we often downsample the trace signals to reduce the impact. We can control the sampling rate in Spring Boot by the property
management.tracing.sampling.probability=1.0By default, Spring Boot samples only 10% of requests and we have chnaged it to 100% so that every request is sent to the trace backend. This in only for development purposes and in production you need to consider about downsampling your signals. You can read more on signal downsampling here.
Micrometer Observation API
As you already know, observability is not just about distributed tracing. There are other signals such as metrics, logs, events, profiles, etc. Instead of instrumenting these signals using different APIs, Micrometer has come up with the idea of instrumenting against a single API and enabling support for multiple signals (tracers, metrics, logs) from that.
Instrument code once, and get multiple benefits out of it.
https://docs.micrometer.io/micrometer/reference/observation.html
Let’s explore how we can leverage this API to instrument some code that we have written. Let’s say we want to trace what happened in the processPayment method in the PaymentService class.
As a first step, you could use the ObservationAPI in the following way:
public class PaymentService {
@Autowired
ObservationRegistry observationRegistry;
public void processPayment(OrderPayment orderPayment) {
Observation.createNotStarted("payment:processPayment", observationRegistry).observe(()->{
// payment process logic
}
}
}or you could use the @Observed annotation to create observations.
public class PaymentService {
@Observed(name = "payment:processPayment")
public void processPayment(OrderPayment orderPayment) {
// payment process logic
}
}
As you can see, we have more information on the request tracing path now, including the operations of the processPayment method. There are some default tags that have been added, and we can add our custom tags too.
You can read more about how we can utilize this API to instrument our codebases here. There’s an example of how we can instrument Apache HTTP Client using this Observation API in the official documentation as well.
Well, that pretty much sums up what I wanted to convey…
From here, I recommend you to play around with these tools on your own. You could try a different observability backend or a different tracer library. Alternatively, you could experiment with different signals like metrics and logs along with the request tracing and see how we can correlate them to enhance the debugability.
Cheers !!!
