The OpenMetrics (Prometheus) metrics format allows metrics providers like Helidon to add optional strings to the metrics output. These strings describe exemplars — samples that are, in some way, typical of all the samples represented by the metric. In particular, an exemplar string for a metric often identifies a trace for an actual use of a service in the application that triggered an update of that metric.
Beginning with Helidon 2.3.0, you can add exemplars to Helidon OpenMetrics output for histograms, counters, timers, and simple timers by adding a few dependencies to the pom.xml
of your Helidon SE or MP application. You do not have to change your application code or your configuration.
In this article, we assume you have an existing Helidon application, such as the Helidon QuickStart application created using the Helidon CLI. You can easily add exemplar support to your application and see it in action:
- Add a few dependencies to your
pom.xml
. - Run Zipkin locally.
- Run and access your application.
- Browse the Zipkin traces identified by the new metrics exemplars.
How Helidon implements exemplars
When you enable exemplar support, each time Helidon updates a metric with a data value it also records the trace ID and the current system time for the sample.
Later, when Helidon formats an OpenMetrics (Prometheus) response to a request sent to your application’s/metrics
endpoint, it selects one representative sample — one exemplar — for each relevant metric value and adds information about that sample to the output for that metric value. (See the examples below.)
Some metric values, such as min
or max
, correspond directly to at least one sample and Helidon uses such a sample for the exemplar. Other metric values are aggregates of multiple samples, such as mean
. For aggregate metrics, Helidon selects for the exemplar a sample with a value as close as possible to the reported metric value.
In many cases, the exemplar’s value will not be the same as an aggregate metric value. For example, consider a metric that is the mean of samples with values 3, 4, and 8. No sample has a value equal to the mean — 5 — so Helidon would choose as the exemplar for this metric the sample with value 4 because that sample has value closest to the mean.
Adding exemplar support to your application
Update your pom.xml
Add dependencies for exemplar support, tracing support, and Zipkin support:
<dependency>
<groupId>io.helidon.metrics</groupId>
<artifactId>helidon-metrics-trace-exemplar</artifactId>
<scope>runtime</scope>
</dependency><dependency>
<groupId>io.helidon.microprofile.tracing</groupId>
<artifactId>helidon-microprofile-tracing</artifactId>
<scope>runtime</scope>
</dependency><dependency>
<groupId>io.helidon.tracing</groupId>
<artifactId>helidon-tracing-zipkin</artifactId>
<scope>runtime</scope>
</dependency>
You could add Helidon Jaeger support instead of Zipkin if you prefer.
Start Zipkin
If you do not start Zipkin, your application will still function correctly and Helidon will add exemplars to the metrics output, but Helidon will log a warning message when it cannot connect to the Zipkin server to report the tracing spans (and of course you will be unable to browse the traces in Zipkin).
docker run --name zipkin -d -p 9411:9411 openzipkin/zipkin
Try the revised application
Build and run your application:
mvn package
java -jar target/quickstart-mp.jar
Access some of the application’s services (responses in italics):
curl -X GET http://localhost:8080/greet
{"message":"Hello World!"}
curl -X GET http://localhost:8080/greet/Joe
{"message":"Hello Joe!"}
curl -X PUT -H "Content-Type: application/json" -d '{"greeting" : "Hola"}' http://localhost:8080/greet/greeting
curl -X GET http://localhost:8080/greet/Jose
{"message":"Hola Jose!"}
curl -X GET http://localhost:8080/greet
{"message":"Hola World!"}
Retrieve metrics
Now, retrieve the metrics and note the exemplar that now appears in the output. Exemplars actually appear as comments at the end of metrics output, as prescribed in the OpenMetrics spec:
curl -s -X GET http://localhost:8080/metrics/vendor
# TYPE vendor_requests_count_total counter
# HELP vendor_requests_count_total Each request (regardless of HTTP method) will increase this counter
vendor_requests_count_total 6 # {trace_id="acc807bee31f8b96"} 1 1619637182.784000
...
Each exemplar shows
- the trace ID,
- the value which that sample contributed to the metric (in this case, the count), and
- the time in epoch seconds when Helidon recorded that sample.
Browse the Zipkin Traces
Access Zipkin in your browser athttp://localhost:9411
.
Click RUN QUERY
to see the scans your Helidon application reported to Zipkin.
Then click EXPAND ALL
to reveal the trace IDs.
Find the trace ID listed in the metrics output. Click SHOW
in that row to see the trace details.
Explore the details of the selected trace and its various sub-spans.
Bottom Line
Without changing your application code, and by adding just a few dependencies to your project’s pom.xml
, you can enable exemplars in your Helidon application’s metrics output.
You can use the trace IDs from the exemplars in the metrics output to identify representative traces for the metric values and use the trace utility (e.g., Zipkin) to find out details about each of those traces.
More Information
Please read the documentation about metrics exemplar support for Helidon SE and Helidon MP applications for more information.
The OpenMetrics spec describes exemplars.