In a production deployment of Jaeger, it may be advantageous to restrict access to Jaeger’s Query service, which includes the UI. For instance, you might have internal security requirements to allow only certain groups to access trace data, or you might have deployed Jaeger into a public cloud. In a true microservices way, one possible approach is to add a sidecar to the Jaeger Query service, acting as a security proxy. Incoming requests hit our sidecar instead of reaching Jaeger’s Query service directly and the sidecar would be responsible for enforcing the authentication and authorization constraints.
Note: This blog post is an update of a similar one published last year, with the instructions updated to catch up with the software updates that happened since, as well as with the changes made to the OpenShift template for Jaeger.
For demonstration purposes we’ll make use of Keycloak as our security solution, but the idea can be adapted to work with any security proxy. This demo should also work without changes with Red Hat SSO. For this exercise, we’ll need:
- A Keycloak (or Red Hat SSO) server instance running.
- An OpenShift cluster, where we’ll run Jaeger backend components. It might be as easy as
oc cluster up
- A local clone of the Jaeger OpenShift Production template
Note that we are not trying to secure the communication between the components, like from the Agent to the Collector. For this scenario, there are other techniques that can be used, such as mutual authentication via certificates, employing Istio or other similar tools.
For this demo, we’ll run Keycloak via Docker directly on the host machine. This is to stress that Keycloak does not need to be running on the same OpenShift cluster as our Jaeger backend.
The following command should start an appropriate Keycloak server locally. If you already have your own Keycloak or Red Hat SSO server, skip this step.
Once the Keycloak server is up and running, let’s create a realm for Jaeger:
- Login into Keycloak (http://<YOUR_IP>:8080/auth/admin/master/console) with
adminas the username and
passwordas the password
- In the top left corner, mouse over the
Masterrealm name and click
Add realmthat appears below to that. Name it
- On the left hand side menu, under
proxy-jaegeras the client ID and save it
- Set the
Valid Redirect URIsand save it. You might want to fine tune this in a production environment, otherwise you might be open to an attack known as “Unvalidated Redirects and Forwards”.
- Open the
Installationtab and select
Keycloak OIDC JSONand copy the JSON that is shown. It should look like this, but the
secretwill have different values.
The last property,
confidential-port, will need to be manually removed later on.
And finally, let’s create a role and a user, so that we can log into Jaeger’s Query service:
- Under the
Configureleft hand side menu, open the
Rolespage and click
- As role name, set
- Under the
Manageleft hand side menu, open the
Userspage and click
- Fill out the form as you wish and set
ONand click on
- Open the
Credentialstab for this user and set a password (temporary or not).
- Open the
Role mappingstab for this user, select the role
Available Roleslist and click
For this demo, we assume you have an OpenShift cluster running already. If you don’t, then you might want to check out tools like
minishift. If you are running a recent version of Fedora, CentOS or Red Hat Enterprise Linux you might want to install the package
origin-clients and run
oc cluster up. This should get you a basic OpenShift cluster running locally, available at https://127.0.0.1:8443 . You can login using
developer as both the login and the password.
To make it easier for our demonstration, we’ll create a privileged Jaeger namespace and set our
developer user as admin:
Preparing the Jaeger OpenShift template
We’ll use the Jaeger OpenShift Production template as the starting point: either clone the entire repository, or just get a local version of the template.
The first step is to add the sidecar container to the
query-deployment object. Under the
containers list, after we specify the
jaeger-query, let’s add the sidecar:
Note that container specifies a
security-proxy-configuration-volume: we’ll use it to store the proxy’s configuration file. You should add the volume under the existing
jaeger-configuration ConfigMap. It will look like this:
Now, we need to specify the contents of the
ConfigMap, with the proxy’s configuration entry. Given that our
production/configmap-elasticsearch.ymlalready has a
ConfigMap specified, we just need to add a new entry to it, after the last entry, at the same level as the
query node. It will look like this:
Note that we are only allowing users with the role
user to log into our Jaeger UI. In a real world scenario, you might want to adjust this to fit your setup. For instance, your user data might come from LDAP, and you only want to allow users from specific LDAP groups to access the Jaeger UI.
secret within the
credentials should match the secret we got from Keycloak at the beginning of this exercise, as well as the
Attention: The value for the
auth-server-urlshould be a location that is reachable by both your browser and by the sidecar, like your host’s LAN IP (192.x, 10.x).
localhost/127.x is not going to work.
As a final step, we go back to the main template file (
jaeger-production-template.yml), as we need to change the query service to direct requests to the port
8080 (proxy) instead of
16686. This is done by changing the property
targetPort on the service named
query-service, setting it to
Now that we have everything ready, let’s deploy Jaeger into our OpenShift cluster. Run the following commands from the same directory you cloned the OpenShift template repository:
During the first couple of minutes, it’s OK if the pods
jaeger-collector fail, as Elasticsearch will still be booting. Eventually, the service should be up and running, as shown in the following image.
Once it is ready to serve requests, click on URL for the route (https://jaeger-query-jaeger.127.0.0.1.nip.io). You should be presented with a login screen, served by the Keycloak server. Login with the credentials you set on the previous steps, and you should reach the regular Jaeger UI.
In this exercise, we’ve seen how to add a security proxy to our Jaeger Query pod as a sidecar. All incoming requests go through this sidecar and all features available in Keycloak can be used transparently, such as 2-Factor authentication, service accounts, single sign-on, brute force attack protection, LDAP support and much more.