Getting Jaeger’s Java Client internal metrics into Prometheus

We recently integrated Micrometer into the internal metrics collection mechanism for the Jaeger Java Client, making it easier to get them into Prometheus.

Prometheus showing the internal metrics from the Jaeger Java Client

The Jaeger Java Client, like other Jaeger clients, already had an internal metrics mechanism for collecting data such as “number of spans started”. With the support for Micrometer included in the client version 0.25.0, it’s now easier to get this data fed into a range of backend metrics platforms supported by Micrometer, like Prometheus, JMX and/or StatsD, among others.

To demonstrate this feature, we developed a simple Vert.x application that would just accept a request on a given port, create a span, write back a “Hello” message and finish the span.

Our example makes use of the Prometheus Registry as the concrete backend for Micrometer, available under the artifact coordinates io.micrometer:micrometer-registry-prometheus.

Vert.x application demonstrating the Micrometer integration with Jaeger Java Client internal metrics

For reference, this is the build.gradle file for this project. It’s pretty much the same as the starter project for Vert.x, with the addition of Jaeger and Micrometer libraries.

Gradle configuration file for our example

Run the example above with ./gradlew run and it should print out a log entry like this:

13:45:47.634 [vert.x-eventloop-thread-0] WARN  io.vertx.starter.MainVerticle - Registered tracer: GlobalTracer{Tracer(...)}
INFO: Succeeded in deploying verticle

After that, just hit the port 8080to create spans:

$ curl http://localhost:8080
Hello Vert.x!

The metrics collected are exposed to Prometheus on port 8081:

$ curl http://localhost:8081
# HELP jaeger:sampler_updates_total 
# TYPE jaeger:sampler_updates_total counter
jaeger:sampler_updates_total{result=”err”,} 0.0
jaeger:sampler_updates_total{result=”ok”,} 0.0
# HELP jaeger:baggage_restrictions_updates_total 
# TYPE jaeger:baggage_restrictions_updates_total counter
jaeger:baggage_restrictions_updates_total{result=”err”,} 0.0
jaeger:baggage_restrictions_updates_total{result=”ok”,} 0.0
# HELP jaeger:span_context_decoding_errors_total 
# TYPE jaeger:span_context_decoding_errors_total counter
jaeger:span_context_decoding_errors_total 0.0
# HELP jaeger:sampler_queries_total 
# TYPE jaeger:sampler_queries_total counter
jaeger:sampler_queries_total{result=”ok”,} 0.0
jaeger:sampler_queries_total{result=”err”,} 0.0
# HELP jaeger:baggage_updates_total 
# TYPE jaeger:baggage_updates_total counter
jaeger:baggage_updates_total{result=”err”,} 0.0
jaeger:baggage_updates_total{result=”ok”,} 0.0
# HELP jaeger:baggage_truncations_total 
# TYPE jaeger:baggage_truncations_total counter
jaeger:baggage_truncations_total 0.0
# HELP jaeger:traces_total 
# TYPE jaeger:traces_total counter
jaeger:traces_total{sampled=”n”,state=”started”,} 0.0
jaeger:traces_total{sampled=”n”,state=”joined”,} 0.0
jaeger:traces_total{sampled=”y”,state=”started”,} 3.0
jaeger:traces_total{sampled=”y”,state=”joined”,} 0.0
# HELP jaeger:reporter_spans_total 
# TYPE jaeger:reporter_spans_total counter
jaeger:reporter_spans_total{result=”dropped”,} 0.0
jaeger:reporter_spans_total{result=”err”,} 0.0
jaeger:reporter_spans_total{result=”ok”,} 0.0
# HELP jaeger:started_spans_total 
# TYPE jaeger:started_spans_total counter
jaeger:started_spans_total{sampled=”y”,} 3.0
jaeger:started_spans_total{sampled=”n”,} 0.0
# HELP jaeger:finished_spans_total 
# TYPE jaeger:finished_spans_total counter
jaeger:finished_spans_total 3.0

At this point, you just need to get Prometheus to scrape the port 8081. A minimalist configuration file could look like this:

Simplistic Prometheus YAML configuration

Then, start Prometheus using this file:

$ prometheus --config.file=/tmp/prometheus.yml

Prometheus should be available at http://0.0.0.0:9090 , ready to display the metrics reported by the Jaeger Client Java.

Summary

In this blog post, we’ve seen that it’s now very easy to get the internal metrics from the Jaeger Client for Java published by a backend supported by Micrometer, like Prometheus. Give it a try and don’t forget to give us your feedback!