Secure your Red Hat SSO/Keycloak Operator instance with your own keystore

Problem description

Published in

virtualsciences

8 min readOct 14, 2022

Red Hat Single Sign-On is a Red Hat distribution of Keycloak. You can install it in several ways on OpenShift. For instance, there is a set of templates, but you can also install it using the Red Hat SSO Operator. And the Red Hat SSO Operator is the Red Hat distribution of the upstream Keycloak Operator, based on JBoss Wildfly. The upstream Keycloak Operator is deprecated and is superseded by a Quarkus based Keycloak since version 20.

Using the operator you’ll get a preconfigured and operator-managed version of Keycloak. And in the context of this article it will install and manage the following resources:

a Stateful Set for the Keycloak instance.
either an embedded Postgress database or references to an external postgress database.
secrets with the database credentials and keycloak admin credentials.
a service that exposes the https port of the JBoss Wildfly Undertow server.
and a route that will have TLS Termination on re-encrypt.

On startup the Keycloak instance will generate a self-signed certificate using the OpenShift internal certification manager, and creates a keystore and truststore with it:

Keycloak instance generating keystore and truststore

After the startup of the keycloak instance it provides two URLs for accessing the authorization server:

externalURL: https://keycloak-rhsso.apps.oscluster.yourdomain.nl'
internalURL: https://keycloak.rhsso.svc:8443

You’ll find this in the status of the keycloak instance YAML:

The problem with these is that regarding the external URL, in combination with the TLS Termination set to reencrypt, it relies on the certificate of the OpenShift environment. When this is a so-called wildcard certificate, then this would work fine. For instance:

However, if this turns out to be a specific certificate then this would lead to a hostname verification error. It would state that the hostname does not exist in the list of Subject Alternative Names. And that is correct because you would not update your certificate every time you add a service to the cluster.

For the internal URL to work you would need to have the OpenShift CA in your truststore.

Implement your own keystore

As we see it, the solution would be to use your own key and truststore and set the TLS Termination to passthrough. I have consulted several articles and documents, that would suggest a solution. However, I failed to get those working. Combining them I did find a working solution. At the end of the article, I’ll provide a list of consulted pages.

Attach a custom keystore and truststore to your Keycloak instance

One of the articles describes how to add certificates to the RH-SSO truststore. I started with that, and figured that I could extend it to also provide a keystore.

I’m not going to describe how to create a keystore and truststore here. In the article the central point is to add the specific environment variables and a volume mount to add the truststore to the Keycloak Custom Resource:

This suggests you have a secret containing the truststore, which is mounted as a volume and setting the SSO_TRUSTSTORE* variables to point to that truststore. This seems to work, but adding a keystore with corresponding variables didn’t.

The solution above refers to the Keycloak instance Custom Resource Definition that states that you can add a keycloakDeploymentSpec node and underneath that an experimental node. The latter node may contain an env node and a volumes node, amongst other options:

So, what I need is:

environment variables referencing both the keystore and truststore with their properties
a secred with the keystore and truststore mounted as a volume.
a script that reconfigures keycloak and that is delivered as a ConfigMap, mounted in the proper folder so that it is automatically run.

In my solution, it turns out that I couldn’t use the SSO_TRUSTSTORE* variables because I found that on startup of the pod they in some mystical way did not contain the values I set, but reflect the proper values not earlier than when the server was running. So, I used my own custom environments. The environment variables I use are as follows:

keycloak instance environment variables

The volume mount of the keystore and truststore secret is:

keycloak instance keystore volume mount

Under volumes, there is a list of items. Every item in the list expects:

name: name of the volume mount
mountPath: folder path in which the secret is mounted
secrets: a list of secrets that are mounted. Apparently, this is a list, but we need only one.
items: a list of key-path pairs of every item in the secret that needs to be placed in the folder. The key refers to the item in the secret. The path is the name of the file in the mountPath folder.

Then lastly, the tlsTermination must be set to passtrough under externalAccess:

Set tlsTermination to passthrough

Postconfigure script

Having the keystore and truststore attached to the keycloak instance, the server needs to be reconfigured to have it use the custom keystore and truststore. The Keycloak JBoss Wildfly server uses the file standalone-openshift.xml for the configuration. In it, the Undertow HTTP Server is configured as:

It refers to the security-realm “ApplicationRealm”, which is defined at the top of the file as:

The keystore here reflects the values of my changes but originally contain the generated self-signed keystore.

The truststore is defined under the Keycloak server subsystem:

Again here my changes are reflected.

To have these changes performed, I created a postconfigure.sh script, that I posted on GitHub.

It consists of the following snippets. First, it starts the JBoss CLI and starts an embedded server:

Since the Keycloak server isn’t started yet, you’ll need an embedded server to have access to the configuration. In doing so, you use the APIs of JBoss Wildfly that implement the appropriate business rules to have the configuration consistent. Unless you would want to change the configuration manually or by string replacements risking an inconsistent configuration. As can be seen, the embedded server is started with the standalone-openshift.xml as the server-config.

Next, the keystore is set by navigating to the proper setting. The JBoss CLI uses a XPath-alike language to select the appropriate node from the standalone-openshift.xml on which operations can be performed:

Updating the JBoss Wildfly Undertow server keystore

By the way, to find out the proper path, you can start the JBoss CLI in the terminal of the pod as /opt/eap/bin/jboss-cli.sh and navigate to the appropriate node using cd, ls, and pwd, as follows:

Navigating through the JBoss Wildfly configuration

Here you can connect to the running Wildfly server. The pwd command will provide the proper path. Add the command you want to perform on the node separated with a colon ‘:’. The write-attribute(name, value) command will set the particular attribute with the appropriate value. The $KEYSTORE_PATH variable is set earlier in the script as:

In a similar way the truststore is set and the embedded server is stopped:

Set truststore and stop the embedded server

Mount postconfigure script

For the postconfigure script I created a configmap as:

Create configmap keycloak-postconfigure

Since I had to test and therefore update the script several times, I created a handy script for it: replace_cm_keycloak-postconfig.sh.

This script is to be placed in the /opt/eap/extensions folder of the keycloak pod/container. This can be done in the same way as the secret for the keystore and truststore. Add the following snippet to the Keycloak Custom Resource as an item under volumes:

Mount postconfigure script

This is quite similar to the mount of the keystore and truststore secret. However, instead of secrets it uses configMaps to refer to the keycloak-postconfigure configmap. It mounts it on the folder /opt/eap/extensions from where it is picked up by the JBoss Wildfly startup to be executed automatically. One other important thing is the mode element of the postconfigure.sh item. It needs have the executable bit set. It is done by setting the file attributes to octal 0777 (decimal 511 in this example). This could be made more strict of course.

Completion

I placed an example keycloak custom resource on GitHub: kc_keycloak.yaml. Having all in place, a re-apply of this custom resource will restart the keycloak-0 pod. In the log you will see the following snippets:

Final Remarks

In this solution the postconfigure.sh script is mounted using a ConfigMap to the prescribed extensions folder of the JBoss Wildfly home. Doing so this folder is “hijacked” by this solution. The easiest way to add other extensions is to add those to the configmap. If you want to use an external, managed Postgress database, you might need to install another JDBC Driver to support TLS. To me, it does not seem appropriate to add binaries as drivers to a config map. You could solve this by mounting an “emptyDir” volume to the folder and using an init container to copy the driver and the postconfigure.sh script to that folder. Under the experimental node in the Keycloak Custom Resource, you could also add the command and args element to execute a modified script in stead of the default entrypoint command of image. However, I did not test this yet.

Update, November 25th, 2022: If you have implemented the solution described in this article, please check out my follow-up article Red Hat Single Sign-on Operator and custom Keystore revisited. Here I adapt the solution, to prevent the Operator from failing.

Secure your Red Hat SSO/Keycloak Operator instance with your own keystore

Problem description

Implement your own keystore

Attach a custom keystore and truststore to your Keycloak instance

Postconfigure script

Mount postconfigure script

Completion

Final Remarks

Consulted Articles

Written by Martien van den Akker