Peek Inside Coherence With OpenTracing, Part 2
Trace REST API requests through Coherence with JAX-RS
If you haven’t read the previous article, it is highly recommended that you do that first, as it covers the details of how tracing works in Coherence and is a prerequisite for this article.
This article builds on the previous article by adding support for tracing to the REST API of the coherence-demo application. It does that by creating a trace for each JAX-RS request within the application. It also exposes Coherence Management over REST, so we can dynamically configure Coherence tracing within the application.
Let’s begin with the JAX-RS span generation.
Adding Tracing Support to REST Endpoints
In order to add tracing support to JAX-RS resources in the demo application, we’ll first need to add the dependencies necessary to support OpenTracing.
The following are the Maven coordinates for what is required:
Implementing Tracing Filter
The easiest way to add tracing support to JAX-RS is to create a TracingFilter class that implements bothjavax.ws.rs.container.ContainerRequestFilter and javax.ws.rs.container.ContainerResponseFilter interfaces. This will allow JAX-RS runtime to create the outermost span for each incoming request, and close it after the response has been sent to the client.
Let’s take a look at the two methods of primary interest within theTracingFilter:
The code above executes when a request is in-bound, creating a span using the class and method name of the JAX-RS resource. Along with the operation name, captured metadata includes the component generating the span (jaxrs), that the span is server-side, the HTTP method, and the request URL.
The code then stores the span and its associated scope into the ContainerRequestContext, so it can be accessed after the response is processed, as shown below:
We obtain the references for the span and scope to set the metadata pertaining to the result of the HTTP request (status code, error status) and then finish the span and close the scope.
To use the TracingFilter, a simple change to the demo’s ServiceResourceConfig constructor is required to register it with JAX-RS.
With the above in place, it’s now possible to easily generate spans for all REST resources of the demo application. Well, at least the REST resources in the same ResourceConfig. The same filter could be registered with the demo ApplicationResourceConfig to generate spans for all static resource requests. We’ll leave that as an exercise for the reader.
Enabling Coherence Management over REST
To implement the ability to dynamically enable, disable, or configure tracing within the cluster, we have two options. The first is using JMX directly; however, as we want to be able to do this from the browser, that would mean adding custom REST endpoints to make the JMX calls. Since Coherence already provides Management over REST, we can instead register the Coherence Management resource along with the other resources already defined for the demo:
Additionally, by defining the Management over REST resources here, we eliminate the need for dealing with CORS restrictions.
We won’t go into many details on the UI itself — it’s available on GitHub for those interested in the details — but we do want to show some example usages of the Coherence Management over REST endpoints enabling dynamic configuration of OpenTracing within the cluster:
Analyzing Captured Traces with Jaeger UI
If we now run the application against a three member cluster, the tracing spans created by Coherence should be merged into the root tracing span created for our JAX-RS resource by the TracingFilter.
To verify, we can look at the details for one of the traces we captured using Jaeger UI console:
We can see a span generated for the demo’s ChartDataResource.getChartData endpoint as well as spans generated by Coherence by calls within ChartDataResource.getChartData. Also note that because the cluster has three members, we can see the Size and AggregateFilter requests each being processed in parallel by the cluster.
Taking a closer look at getChartData, we should see the metadata added by the TracingFilter:
Yes, we can see the component is jaxrs and we have useful values for http.url and http.method.
Going back to the span listing, we can take a quick peek to see what Coherence is generating. Let’s take a look at the AggregateFilter operation. This is composed of three parts: request, dispatch, and process.
Taking a look at AggregateFilter.request:
We can see the request was initiated by member 1. The span, in this case, is considered a client span in the Coherence context. Next, the request is dispatched to the appropriate member(s) for processing:
We can see from the dispatch, the current member is 2 and the initiator of the request (member.source) is 1. We also can see which thread is processing the request, and the internal operation type. Also, note, span.kind has changed from client to server. The process span includes similar details as the dispatch span, but also includes details on the filter type, the partitions that contain the data, as well as the owning cache.
Simplifying JAX-RS Tracing
While rolling out a custom solution similar to the TracingFilter is certainly a viable option to capture traces from JAX-RS endpoints, there are easier ways.
For example, Eclipse Microprofile implementations such as Helidon provide built-in support for tracing of JAX-RS resources. All you need to do to enable it is a single line in a config file, and a dependency on the tracing client you want to use.
You can also add tracing support for any other CDI bean that is used by the JAX-RS resources, such as domain services or repository implementations, by simply annotating the methods you want to trace (or the class, in which case all the methods will be traced) with the @Traced annotation.
Summary
That wraps up our JAX-RS and Coherence tracing integration discussion.
We’ve shown how to easily generate spans for an application’s REST endpoints and how to integrate Coherence Management over REST to dynamically change the tracing configuration of the Coherence cluster.
We also looked at how the captured traces look like in the Jaeger UI, and what type of metadata is provided by both our custom JAX-RS filter and Coherence itself.
In the next article, we’ll move from the API layer of the application to the persistence layer, and show you how to add tracing spans from the JPA and JDBC calls that Coherence makes to the database via a cache store.
