Protecting the collection of spans

The Jaeger Collector is the component responsible for receiving the spans that were captured by the tracer and writing them to a persistent storage like Cassandra or Elasticsearch. This component is usually deployed on a server remote to the instrumented application, sometimes even deployed on a distant data center.

For scenarios where the collector might be exposed to the public internet, it’s important to place an authentication proxy in front of the Collector. Until now, however, there wasn’t an easy way to let the client send authentication data along with the HTTP request. A new feature just added to the Java Client changes that. By exporting a few extra environment variables, you can make use of this new feature without changes to your code, as long as you are using the version 0.22.0 or higher of the client.

We’ll demonstrate this by placing a Keycloak Authentication Proxy in front of our Jaeger Collector and creating a Service Account for our instrumented application.

Preparing Keycloak

For this demo, we’ll use Keycloak on Docker, but the instructions are quite similar to Red Hat SSO. If you already have a Keycloak or Red Hat SSO instance running, you can skip this first step:

$ docker run \
-d \
--name keycloak-server \
-e KEYCLOAK_USER=admin \
-e KEYCLOAK_PASSWORD=password \
-p 8080:8080 \
jboss/keycloak

After a few seconds, an instance of Keycloak should be up and running. Login using and into http://YOUR_IP:8080/auth/admin/master/console , replacing by your a local IP that can be accessed from outside of the Docker internal network. This is probably an IP like .

Let’s create a new realm, called , one role and two clients.

To create the realm, simply place the mouse pointer over the “Master” realm name on the top-left part of the screen and a blue “Add realm” button will show up. Click on it and enter “jaeger” as the name.

The role we will create should be named and will represent applications (not users) across our realm. The idea is that each one of our microservices would be a client, all having a role . We could use this distinction to allow only to talk to our Jaeger Collector, while blocking applications from, say, accessing the Jaeger Query UI. To create this role, click on the option located at the menu on the left-hand side, under “Configure”, then click “Add Role” and enter “application” as the “Role Name”.

We then create our OAuth Clients: the first should be named and should look like this:

proxy-jaeger client on Keycloak

Once it’s created, open the tab, select and copy its contents into . Note that this JSON contains a field: we’ll need this later!

Then, we’ll create another client that will represent our instrumented application (microservice). We’ll name it , but you should probably name it after your service’s name. It’s very similar to the first one, except that we’ll turn on the option “Service Accounts Enabled”. In the end, this is how it looks like:

Keycloak Client representing the instrumented business application

Open the “Service Account Roles” tab and assign the role “application” to this client, by selecting it on the “Available Roles” box and clicking on “Add selected”, so that it gets added into the “Assigned Roles” box.

Finally, open the “Credentials” tab and copy the “Secret”. We’ll need it later as well.

Preparing the proxy

At this point, you should have a configuration file located at , which is the Keycloak Open ID Connect (OIDC) JSON file for the client. Let’s create a proxy configuration as well, placed in the same directory:

:

{
"target-url": "http://YOUR_IP:14268",
"bind-address": "0.0.0.0",
"http-port": "8080",
"applications": [
{
"base-path": "/",
"adapter-config": {
"realm": "jaeger",
"auth-server-url": "http://192.168.178.20:8080/auth",
"ssl-required": "external",
"resource": "proxy-jaeger",
"credentials": {
"secret": "81e7f607-3949-4c19-be9b-1fb1b1f92ae6"
}
}
,
"constraints": [
{
"pattern": "/*",
"roles-allowed": [
"application"
]
}
]
}
]
}

Note that both the and the values should match the ones from the . The represents our Jaeger Collector, which we’ll start in the next steps. The port is the Collector’s HTTP port for receiving spans in Jaeger’s format.

With the configuration files in place, let’s start the Keycloak Auth Proxy:

docker run \
-d \
--name=keycloak-proxy \
-p 8180:8080 \
-v /tmp/conf:/opt/jboss/conf \
jboss/keycloak-proxy

After a couple of seconds, the Keycloak Auth Proxy should have started.

On a real production scenario, this proxy might also do the SSL termination, so that the client would securely transmit the span and authentication data over an encrypted channel. For simplicity, we are leaving this out of this blog post. Alternatively, the proxy itself could sit behind a service doing the SSL termination, like an OpenShift secure route.

Starting the Jaeger Collector

To keep this demo simple, we’ll start the Jaeger Docker image:

docker run \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
--name=jaeger \
jaegertracing/all-in-one:latest

After a few seconds, all Jaeger components should be up and running.

Configuring the client

To configure the client, all we need is to export some environment variables, like this:

export JAEGER_ENDPOINT=http://YOUR_IP:8180/api/traces 
export JAEGER_AUTH_TOKEN=THE_TOKEN

With this configuration, the Jaeger Java Client will use the and set as a Bearer token within the HTTP header on requests sent to the endpoint, which is set to the port of our proxy (8180). Non-authenticated requests will be blocked, while requests with valid tokens should reach the collector.

We’ll obtain a valid token by running the following command:

curl \
-X POST \
-u instrumented-application:THE_SECRET \
http://YOUR_IP:8080/auth/realms/jaeger/protocol/openid-connect/token \
-d 'grant_type=client_credentials'

In a production environment, you might have a long-lived OAuth token obtained by a bootstrap script or supplied by a tool like Ansible, perhaps destroying “old” pods from time to time, getting a new token each time.

Make sure to replace by the value we copied from the “Credentials” tab for the client and to replace with your IP.

The command should return a JSON including a property named (the very first one in the JSON). This is the token we need for our .

With the appropriate environment variables in place, you should now start your application! In the boot log, there should be an entry like this when the tracer is initialized:

15:30:07,677 INFO  [com.uber.jaeger.Configuration] (ServerService Thread Pool -- 64) Initialized tracer=Tracer(version=Java-0.21.0-SNAPSHOT, serviceName=opentracing-cdi-example, reporter=CompositeReporter(reporters=[RemoteReporter(queueProcessor=RemoteReporter.QueueProcessor(open=true), sender=HttpSender(), maxQueueSize=100, closeEnqueueTimeout=1000), LoggingReporter(logger=org.slf4j.impl.Slf4jLogger(com.uber.jaeger.reporters.LoggingReporter))]), sampler=ConstSampler(decision=true, tags={sampler.type=const, sampler.param=true}), ipv4=-1062686188, tags={hostname=carambola, jaeger.version=Java-0.21.0-SNAPSHOT, ip=192.168.178.20}, zipkinSharedRpcSpan=false, baggageSetter=com.uber.jaeger.baggage.BaggageSetter@7fbb331c)

Note that the property is set to , indicating that the endpoint environment variable was recognized and an appropriate sender was selected. Try calling your endpoints and you should see traces on Jaeger, as usual. Then, try stopping your application, unset the environment variable and try again. No traces should arrive at the server now, indicating that the Jaeger Auth Proxy blocked the request.

Summary

In a production deployment, the collector might sit in a different data center as the agent or the target application, or there might be several collectors in a multitenant scenario. For such cases, controlling who sends data to the collector is not only useful, but required.

This recently added feature to the Jaeger Java Client is the first step in adding authentication to the “client” side of the communication. The second step is to let the Agent have a similar behavior, so that the Client can still send spans via UDP to a local agent and the agent making a secure connection to the collector.

As always, join our mailing list or Gitter channel and let us know if you use this feature and whether your use case is covered by it.

JaegerTracing

Open source distributed tracing platform at Cloud Native…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store