Secure Kafka with Keycloak: SASL OAuth Bearer

This post will do a step-by-step configuration of the strimzi-operator (Apache Kafka) on Openshift. Expose an external listener on the Openshift platform as a route over TLS and Secure the Kafka Cluster using Keycloak using SASL OAuth Bearer.

Suppose you don’t want to do a bunch of configurations. An easy option for you would be Openshift Streams for Apache Kafka https://www.redhat.com/en/blog/introducing-red-hat-openshift-streams-apache-kafka.

Let’s begin the Journey of securing the Kafka kittens.

Suppose you aren’t familiar with the concepts like SASL for OAuth. Let’s go over the details of how SASL OAuth Bearer works.

SASL OAuth Bearer:

https://datatracker.ietf.org/doc/html/rfc7628 [Simple Authentication and Security Layer (SASL) Mechanisms for OAuth]

OAuth 2.0 Protocol Flow

Overview

Running Apache Kafka deployed using the Strimzi Operator and Keycloak Operator on Openshift. Two Quarkus-based Clients: producer and consumer applications running on an external local system.

• During the terraforming of the Kafka cluster, it will fetch the JWKs certificate from the Keycloak.
• Configure a service account in Keycloak for the producer/consumer.
• Using the service account sends a request to the Keycloak (OpenID-connect end-point) for fetching the token.
• The producer/consumer sends the token with the message or consumes the message.
• The broker verifies the token and, authenticates the user & allows them to produce/consume.

Setup

• CRC or Openshift Instance
• strimzi-operator
• keycloak operator
• Kafka CLI tools/Java
• kubectl/oc tools

Installation: OLM: One-click install

• Keycloak Operator

keycloak-operator

• Strimzi Operator

strimzi-operator

Strimzi Local Installation

git clone https://github.com/strimzi/strimzi-kafka-operator.git
cd strimzi-kafka-operator

Update the namespace in which you wanted to deploy the Operator.

sed -i '' 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml

Login & Create the namespace/project (kafka-demo)

oc login <cluster>
oc new-project <projectname> //kafka-demo as project name

Create the Cluster-Operator in the namespace: Kafka

kubectl apply -f install/cluster-operator -n kafka-demo

Check the status of the operator pod

oc get pods -n kafka-demo

Keycloak Local Installation

git clone https://github.com/keycloak/keycloak-operator.git
cd keycloak-operator

Login & Create the namespace/project

oc login <cluster>
oc new-project <projectname> //kc

• Run make cluster/prepare
• Run kubectl apply -f deploy/operator.yaml
• Run kubectl apply -f deploy/examples/keycloak/keycloak.yaml

Once Keycloak is up and running. Create a realm: demo

Create Realm: demo

Creating Kafka Cluster

Whether you have used OLM or followed manual steps, we can now create the Instance.

git clone https://github.com/akoserwal/keycloak-integrations.git
cd kafka-sasl-oauth-keycloak/CR

Update the authentication spec in the Kafka Custom Resource. Open my-cluster.yaml in your preferred editor.

Configure the keycloak end-points:

• Fetch the Keycloak route

export NAMEPSACE=<>

export KEYCLOAK_ROUTE=$(oc get route keycloak -n $NAMESPACE --template='{{ .spec.host }}')

• Fetch the keycloak route SSL certificate.

echo "" | openssl s_client -servername $KEYCLOAK_ROUTE -connect $KEYCLOAK_ROUTE:443 -prexit 2>/dev/null| openssl x509 -outform PEM > keycloak.crt

• Create a secret with keycloak this certificate

oc create secret generic ca-keycloak  \
--from-file=keycloak.crt=keycloak.crt

• Create a truststore that can be used by the Producer/Consumer application

keytool -keystore keycloak.jks -alias root -import -file keycloak.crt -storepass password -noprompt

Authentication Listener Spec:

• update <keycloak-host>
• realm: demo

listeners:
       - name: external
         port: 9094
         tls: true
         type: route
         authentication:
          checkIssuer: true
          jwksEndpointUri: >-
            https://<keycloak-host>/auth/realms/demo/protocol/openid-connect/certs
          userNameClaim: preferred_username
          checkAccessTokenType: true
          accessTokenIsJwt: true
          enableOauthBearer: true
          validIssuerUri: >-
            https://keycloak-host/auth/realms/demo
          tlsTrustedCertificates:
            - certificate: keycloak.crt
              secretName: ca-keycloak
          type: oauth

Create the Kafka cluster in the same or different namespace. In this case, using namespace: kafka-demo

oc create -f keycloak-integrations/kafka-sasl-oauth-keycloak/CR/my-cluster.yaml

Accessing the Cluster using routes

$ oc get -n kafka-demo routes my-cluster-kafka-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}'
//output:kafka broker bootstrap url
eg: my-cluster-kafka-bootstrap-kafka-demo.<openshift-cluster-domain-url>

Fetch the ca-cert from the kafka-demo namespace

oc extract -n kafka-demo secret/my-cluster-cluster-ca-cert --keys=ca.crt

Create a truststore for the Producer/Consumer App

keytool -import -trustcacerts -alias root -file ca.crt -keystore truststore.jks -storepass password -noprompt

Producer/Consumer

cd kafka-sasl-oauth-keycloak/producer 
or
cd kafka-sasl-oauth-keycloak/consumer

Open both applications in your favorite editor as Maven-based projects.

Service Account in Keycloak

Create a service account in Keycloak: kafka-client-service-acc (any other name). Make sure to enable the flag: “Service Account Enabled: ON.”

Copy the ClientId and Credentials

Export properties or update values in the application.properties

export KEYCLOAK_HOST=https://<keycloak-host>
export KAFKA_BOOSTRAP_HOST=my-cluster-kafka-bootstrap-kafka-demo.apps.<domain>:443
export KEYCLOAK_CLIENT_SECRET=
export KEYCLOAK_CLIENT_ID=kafka-client-service-acc
export KEYCLOAK_REALM=demo
export KEYCLOAK_TRUSTSTORE=keycloak.jks
export KAFKA_TRUSTSTORE=truststore.jks
export KAFKA_TOPIC=new-topic

Run the producer & consumer app

./mvnw quarkus:dev

Producer/Consumer

Congratulation!! If you can see the above output. Now, you can produce/consume messages secured by SASL over the Oauth mechanism.

Security Isolation Concern

Our setup works great. But any service account created in that Keycloak(realm: demo) can authenticate against your Kafka cluster.

How to restrict authentication for a set of service accounts?

A new spec called Custom Claim Check was recently added to the authentication listener spec in strimzi-operator. Which allows you to add a custom claim in Token. Which will be checked during the authentication flow.

eg, customClaimCheck: @.userId == '123'

Let’s update the CR and redeploy the cluster.

listeners:
      - authentication:
          checkIssuer: true
          jwksEndpointUri: >-
            https://keycloak-kc.apps.akoserwa.ipt.integrations.rhmw.io/auth/realms/demo/protocol/openid-connect/certs
          userNameClaim: preferred_username
          clientId: kafka-broker
          checkAccessTokenType: true
          accessTokenIsJwt: true
          checkAudience: false
          enableOauthBearer: true
          validIssuerUri: >-
            https://keycloak-kc.apps.akoserwa.ipt.integrations.rhmw.io/auth/realms/demo
          tlsTrustedCertificates:
            - certificate: keycloak.crt
              secretName: ca-truststore
          type: oauth
          clientSecret:
            key: kafka-broker
            secretName: clientsecret
          customClaimCheck: '''kafka-user'' in @.realm_access.roles'
        name: external
        port: 9094
        tls: true
        type: route

Now try to run the producer or consumer application. You will get this error:

org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed due to an invalid token: io.strimzi.kafka.oauth.validator.TokenValidationException: Token validation failed: Custom claim check failed

Authentication failed! For getting authentication. We need to add a realm role in our service account configuration.

Create a Role in the Keycloak(demo realm)

Now add the realm role to our service account.

Suppose you try to check the token claim. You can see the role: “kafka-user” added to the roles array in the realm_access object.

”realm_access”: {
 “roles”: [“kafka-user”, “..”]
 }

Custom claim check rule will be like: ‘kafka-user’ in @.realm_access.roles’

Let’s try to run the producer app again.

Great, we are now able to authenticate, and any other service account without that “kafka-user” role won’t be able to authenticate.

Conclusion

We managed to deploy a secure Kafka Cluster using Keycloak with the help of SASL over the Oauth mechanism. It's easier to control access using the Keycloak Admin interface. Also, addressed the advanced security challenges around restricting the set of service accounts to authenticate against a Kafka instance.

If you like this post, give it a Cheer!!!

Follow the Collection: Keycloak for learning more…

Happy Secure Coding ❤