How to setup OAuth2 mechanism to a Kafka Broker

With the commit of KIP-255 (Kafka Improvement Proposal) at version 2.0.0 of Kafka, now we can use SASL (Simple Authentication and Security Layer) OAUTHBEARER to authenticate clients to the broker or interbroker authentication.

Prerequisites

• Docker
• Docker-compose
• Git

A Quick Demo

If you want only run this setup, just copy this gist file and execute with docker-compose up
This compose will run:

• OAuth2 Server
• Kafka Broker
• Producer
• Consumer

Implementation of Java Classes to support OAuth mechanism

With the documentation of KIP-255 in hands, we need to implement 2 classes to use an external OAuth2 server to authenticate our clients or brokers.

The first class implements AuthenticateCallbackHandler and will serve for clients or broker that needs to authenticate.

The second class implements the same class and will serve to broker can make the validation of the send token using OAuth token introspection.

We will implement two another helper classes, one for handle our http requests (Get Token and Introspect Token) and a class to set our JWT properties. The kafka-oauth project is hosted at GitHub and you can clone to see or modify all classes.

Configuring Kafka Broker

For this configuration, we need to do three steps:

• Create an JAAS configuration file setting the login module that we will use
• Add some properties to server.properties file
• Add our jar file to Kafka lib folder.

JAAS Configuration file

At this file we will configure the login module will be used by our broker.

server.properties file

At this file we will set the classes that will be used to make the login and validation in our OAuth2 Server. The complete server.properties file is at GitHub repository.

Starting an OAuth2 Server and our Kafka Broker

This project requires an OAuth2 server to provide the token and validation of our client or broker. An easy and open source alternative is use ORY Hydra, that is a Certified OAuth2 server written in Go. For this example, we use a docker-compose file that setup server and create 3 accounts:

• consumer-kafka: for consumer container
• producer-kafka: for producer container
• broker-kafka: for interbroker authentication

For start our OAuth2 server and our Kafka broker we need to clone the kafka-playground GitHub repository and run the docker-compose file at root folder. Before running docker-compose we need to set an environment variable called HOST_IP.

HOST_IP=XXX.XXX.XXX.XXX docker-compose up

Configure our clients (Producer/Consumer/Stream) to use this mechanism

Now we can use our jar file to configure clients using Java or Scala. For javascript developer there is an option called kafka-node-oauth. This is an npm module that implements this mechanism.

Java Example

At the cloned repository, we have a folder called kafka-using-java, that contains one producer example, using our .jar file.
To run this example you will need to set HOST_IP environment variable wich contains IP address of running machine.

HOST_IP=XXX.XXX.XXX.XXX docker-compose up

Javascript Example

At the cloned repository, we have a folder called kafka-using-node, that contains one producer and one consumer example, using our modified kafka-node-oauth package.
To run this example you will need to set HOST_IP environment variable wich contains IP address of running machine.

HOST_IP=XXX.XXX.XXX.XXX docker-compose up

Closing words

I hope this article help you to setup a Kafka broker with OAuth2 authentication ready. If you have any doubts or improvement please write a comment that i will really happy to help!