Protecting the collection of spans

Published in

JaegerTracing

6 min readDec 4, 2017

The Jaeger Collector is the component responsible for receiving the spans that were captured by the tracer and writing them to a persistent storage like Cassandra or Elasticsearch. This component is usually deployed on a server remote to the instrumented application, sometimes even deployed on a distant data center.

For scenarios where the collector might be exposed to the public internet, it’s important to place an authentication proxy in front of the Collector. Until now, however, there wasn’t an easy way to let the client send authentication data along with the HTTP request. A new feature just added to the Java Client changes that. By exporting a few extra environment variables, you can make use of this new feature without changes to your code, as long as you are using the version 0.22.0 or higher of the client.

We’ll demonstrate this by placing a Keycloak Authentication Proxy in front of our Jaeger Collector and creating a Service Account for our instrumented application.

Preparing Keycloak

For this demo, we’ll use Keycloak on Docker, but the instructions are quite similar to Red Hat SSO. If you already have a Keycloak or Red Hat SSO instance running, you can skip this first step:

$ docker run \
    -d \
    --name keycloak-server \
    -e KEYCLOAK_USER=admin \
    -e KEYCLOAK_PASSWORD=password \
    -p 8080:8080 \
    jboss/keycloak

After a few seconds, an instance of Keycloak should be up and running. Login using admin and password into http://YOUR_IP:8080/auth/admin/master/console , replacing YOUR_IP by your a local IP that can be accessed from outside of the Docker internal network. This is probably an IP like 192.168.178.x .

Let’s create a new realm, called jaeger, one role and two clients.

To create the realm, simply place the mouse pointer over the “Master” realm name on the top-left part of the screen and a blue “Add realm” button will show up. Click on it and enter “jaeger” as the name.

The role we will create should be named application and will represent applications (not users) across our realm. The idea is that each one of our microservices would be a client, all having a role application. We could use this distinction to allow only applications to talk to our Jaeger Collector, while blocking applications from, say, accessing the Jaeger Query UI. To create this role, click on the Roles option located at the menu on the left-hand side, under “Configure”, then click “Add Role” and enter “application” as the “Role Name”.

We then create our OAuth Clients: the first should be named proxy-jaeger and should look like this:

Once it’s created, open the Installation tab, select Keycloak OIDC JSON and copy its contents into /tmp/conf/keycloak.json. Note that this JSON contains a secret field: we’ll need this later!

Then, we’ll create another client that will represent our instrumented application (microservice). We’ll name it instrumented-application, but you should probably name it after your service’s name. It’s very similar to the first one, except that we’ll turn on the option “Service Accounts Enabled”. In the end, this is how it looks like:

Keycloak Client representing the instrumented business application

Open the “Service Account Roles” tab and assign the role “application” to this client, by selecting it on the “Available Roles” box and clicking on “Add selected”, so that it gets added into the “Assigned Roles” box.

Finally, open the “Credentials” tab and copy the “Secret”. We’ll need it later as well.

Preparing the proxy

At this point, you should have a configuration file located at /tmp/conf/keycloak.json, which is the Keycloak Open ID Connect (OIDC) JSON file for the proxy-jaeger client. Let’s create a proxy configuration as well, placed in the same directory:

proxy.json:

{
          "target-url": "http://YOUR_IP:14268",
          "bind-address": "0.0.0.0",
          "http-port": "8080",
          "applications": [
              {
                  "base-path": "/",
                  "adapter-config": {
                    "realm": "jaeger",
                    "auth-server-url": "http://192.168.178.20:8080/auth",
                    "ssl-required": "external",
                    "resource": "proxy-jaeger",
                    "credentials": {
                      "secret": "81e7f607-3949-4c19-be9b-1fb1b1f92ae6"
                    }
                  }
            ,
            "constraints": [
                      {
                          "pattern": "/*",
                          "roles-allowed": [
                              "application"
                          ]
                      }
                  ]
              }
          ]
      }

Note that both the auth-server-url and the credentials.secret values should match the ones from the keycloak.json. The target-url represents our Jaeger Collector, which we’ll start in the next steps. The port 14268 is the Collector’s HTTP port for receiving spans in Jaeger’s format.

With the configuration files in place, let’s start the Keycloak Auth Proxy:

docker run \
    -d \
    --name=keycloak-proxy \
    -p 8180:8080 \
    -v /tmp/conf:/opt/jboss/conf \
    jboss/keycloak-proxy

After a couple of seconds, the Keycloak Auth Proxy should have started.

On a real production scenario, this proxy might also do the SSL termination, so that the client would securely transmit the span and authentication data over an encrypted channel. For simplicity, we are leaving this out of this blog post. Alternatively, the proxy itself could sit behind a service doing the SSL termination, like an OpenShift secure route.

Starting the Jaeger Collector

To keep this demo simple, we’ll start the all-in-one Jaeger Docker image:

docker run \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 16686:16686 \
    -p 14268:14268 \
    --name=jaeger \
    jaegertracing/all-in-one:latest

After a few seconds, all Jaeger components should be up and running.

Configuring the client

To configure the client, all we need is to export some environment variables, like this:

export JAEGER_ENDPOINT=http://YOUR_IP:8180/api/traces 
export JAEGER_AUTH_TOKEN=THE_TOKEN

With this configuration, the Jaeger Java Client will use the HttpSender and set THE_TOKEN as a Bearer token within the Authorization HTTP header on requests sent to the endpoint, which is set to the port of our proxy (8180). Non-authenticated requests will be blocked, while requests with valid tokens should reach the collector.

We’ll obtain a valid token by running the following command:

curl \
    -X POST \
    -u instrumented-application:THE_SECRET \
    http://YOUR_IP:8080/auth/realms/jaeger/protocol/openid-connect/token \
    -d 'grant_type=client_credentials'

In a production environment, you might have a long-lived OAuth token obtained by a bootstrap script or supplied by a tool like Ansible, perhaps destroying “old” pods from time to time, getting a new token each time.

Make sure to replace THE_SECRET by the value we copied from the “Credentials” tab for the instrumented-application client and to replace YOUR_IP with your IP.

The curl command should return a JSON including a property named access_token (the very first one in the JSON). This is the token we need for our JAEGER_AUTH_TOKEN .

With the appropriate environment variables in place, you should now start your application! In the boot log, there should be an entry like this when the tracer is initialized:

15:30:07,677 INFO  [com.uber.jaeger.Configuration] (ServerService Thread Pool -- 64) Initialized tracer=Tracer(version=Java-0.21.0-SNAPSHOT, serviceName=opentracing-cdi-example, reporter=CompositeReporter(reporters=[RemoteReporter(queueProcessor=RemoteReporter.QueueProcessor(open=true), sender=HttpSender(), maxQueueSize=100, closeEnqueueTimeout=1000), LoggingReporter(logger=org.slf4j.impl.Slf4jLogger(com.uber.jaeger.reporters.LoggingReporter))]), sampler=ConstSampler(decision=true, tags={sampler.type=const, sampler.param=true}), ipv4=-1062686188, tags={hostname=carambola, jaeger.version=Java-0.21.0-SNAPSHOT, ip=192.168.178.20}, zipkinSharedRpcSpan=false, baggageSetter=com.uber.jaeger.baggage.BaggageSetter@7fbb331c)

Note that the sender property is set to HttpSender , indicating that the endpoint environment variable was recognized and an appropriate sender was selected. Try calling your endpoints and you should see traces on Jaeger, as usual. Then, try stopping your application, unset the JAEGER_AUTH_TOKEN environment variable and try again. No traces should arrive at the server now, indicating that the Jaeger Auth Proxy blocked the request.

Summary

In a production deployment, the collector might sit in a different data center as the agent or the target application, or there might be several collectors in a multitenant scenario. For such cases, controlling who sends data to the collector is not only useful, but required.

This recently added feature to the Jaeger Java Client is the first step in adding authentication to the “client” side of the communication. The second step is to let the Agent have a similar behavior, so that the Client can still send spans via UDP to a local agent and the agent making a secure connection to the collector.

As always, join our mailing list or Gitter channel and let us know if you use this feature and whether your use case is covered by it.