Using Exemplars in Helidon Metrics

The OpenMetrics (Prometheus) metrics format allows metrics providers like Helidon to add optional strings to the metrics output. These strings describe exemplars — samples that are, in some way, typical of all the samples represented by the metric. In particular, an exemplar string for a metric often identifies a trace for an actual use of a service in the application that triggered an update of that metric.

Beginning with Helidon 2.3.0, you can add exemplars to Helidon OpenMetrics output for histograms, counters, timers, and simple timers by adding a few dependencies to the pom.xml of your Helidon SE or MP application. You do not have to change your application code or your configuration.

In this article, we assume you have an existing Helidon application, such as the Helidon QuickStart application created using the Helidon CLI. You can easily add exemplar support to your application and see it in action:

• Add a few dependencies to your pom.xml.
• Run Zipkin locally.
• Run and access your application.
• Browse the Zipkin traces identified by the new metrics exemplars.

How Helidon implements exemplars

When you enable exemplar support, each time Helidon updates a metric with a data value it also records the trace ID and the current system time for the sample.

Later, when Helidon formats an OpenMetrics (Prometheus) response to a request sent to your application’s/metrics endpoint, it selects one representative sample — one exemplar — for each relevant metric value and adds information about that sample to the output for that metric value. (See the examples below.)

Some metric values, such as min or max, correspond directly to at least one sample and Helidon uses such a sample for the exemplar. Other metric values are aggregates of multiple samples, such as mean. For aggregate metrics, Helidon selects for the exemplar a sample with a value as close as possible to the reported metric value.

In many cases, the exemplar’s value will not be the same as an aggregate metric value. For example, consider a metric that is the mean of samples with values 3, 4, and 8. No sample has a value equal to the mean — 5 — so Helidon would choose as the exemplar for this metric the sample with value 4 because that sample has value closest to the mean.

Adding exemplar support to your application

Update your pom.xml

Add dependencies for exemplar support, tracing support, and Zipkin support:

<dependency>
    <groupId>io.helidon.metrics</groupId>
    <artifactId>helidon-metrics-trace-exemplar</artifactId>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>io.helidon.microprofile.tracing</groupId>
    <artifactId>helidon-microprofile-tracing</artifactId>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing-zipkin</artifactId>
    <scope>runtime</scope>
</dependency>

You could add Helidon Jaeger support instead of Zipkin if you prefer.

Start Zipkin

If you do not start Zipkin, your application will still function correctly and Helidon will add exemplars to the metrics output, but Helidon will log a warning message when it cannot connect to the Zipkin server to report the tracing spans (and of course you will be unable to browse the traces in Zipkin).

docker run --name zipkin -d -p 9411:9411 openzipkin/zipkin

Try the revised application

Build and run your application:

mvn package
java -jar target/quickstart-mp.jar

Access some of the application’s services (responses in italics):

curl -X GET http://localhost:8080/greet
{"message":"Hello World!"}

curl -X GET http://localhost:8080/greet/Joe
{"message":"Hello Joe!"}

curl -X PUT -H "Content-Type: application/json" -d '{"greeting" : "Hola"}' http://localhost:8080/greet/greeting

curl -X GET http://localhost:8080/greet/Jose
{"message":"Hola Jose!"}

curl -X GET http://localhost:8080/greet          
{"message":"Hola World!"}

Retrieve metrics

Now, retrieve the metrics and note the exemplar that now appears in the output. Exemplars actually appear as comments at the end of metrics output, as prescribed in the OpenMetrics spec:

curl -s -X GET http://localhost:8080/metrics/vendor
# TYPE vendor_requests_count_total counter
# HELP vendor_requests_count_total Each request (regardless of HTTP method) will increase this counter
vendor_requests_count_total 6 # {trace_id="acc807bee31f8b96"} 1 1619637182.784000
...

Each exemplar shows

• the trace ID,
• the value which that sample contributed to the metric (in this case, the count), and
• the time in epoch seconds when Helidon recorded that sample.

Browse the Zipkin Traces

Access Zipkin in your browser athttp://localhost:9411.

Click RUN QUERYto see the scans your Helidon application reported to Zipkin.

Zipkin Dashboard

Then click EXPAND ALL to reveal the trace IDs.

Zipkin Trace Overview

Find the trace ID listed in the metrics output. Click SHOW in that row to see the trace details.

Zipkin Trace Expended Overview

Explore the details of the selected trace and its various sub-spans.

Zipkin Trace Detail

Bottom Line

Without changing your application code, and by adding just a few dependencies to your project’s pom.xml, you can enable exemplars in your Helidon application’s metrics output.

You can use the trace IDs from the exemplars in the metrics output to identify representative traces for the metric values and use the trace utility (e.g., Zipkin) to find out details about each of those traces.

More Information

Please read the documentation about metrics exemplar support for Helidon SE and Helidon MP applications for more information.

The OpenMetrics spec describes exemplars.