Kafka Multi-Tenancy Architecture: SSL client authentication

Itamar Yerushalmi
Nov 7, 2017 · 4 min read

Kafka is a distributed streaming platform. As such, we would want many consumers and producers to write to the same Kafka cluster.

In one of our latest projects in a large scale cyber-security company, we created a large scale deployment of a Kafka cluster that needed to serve approximately 10TB of digested data each day, split across over 100 million events. This was done in order to limit each customer’s ability to access only their topic in a multi-tenancy architecture.

There are a few guides over the internet to enable SSL client authentication on Kafka. Some of them are outdated or incomplete, while the others include overkill Kerberos implementation. So I created this one, which is based on Kafka’s official documentation at this link, hopefully you’ll find it useful.

Producer can write to all topics. Consumer can access secured topic only with right certificate.

For this example, I have one Kafka server and 2 clients (consumers).

Server Side:

Make sure that you have truststore and keystore JKSs for each server.

  1. In case you want a self signed certificate, you can use the following commands:

Make sure that you configure the certificate properly — this will be used later in the ACL configuration.

2. On Kafka servers — server.properties — add the following lines:

3. Add the following to the advertised.listeners and the listeners keys SSL values on port 9092:

4. Change

To

5. For logging — enable DEBUG on Kafka authentication by changing the line in log4j.properties —

From:

To:

6. On the client servers add the following lines to consumer.properties and producer.properties:

Notice that you are using different files for the different servers.

7. Produce messages to a new topic:

You should now be able to write to the topic from both servers — if you look at the kafka-authorizer.log on the server you will see the following messages:

8. Using the kafka-acl command — set permissions for one of the users — by default, it will remove permissions from the other user:

9. Now, if you try to write or read from the granted user, you will succeed —

and from the other user — permission denied.

Talking Tech Around

Talking about startups and diffrent types of technologies…

Talking Tech Around

Talking about startups and diffrent types of technologies that interests me. Most of the technologies will be in the new type of information technology for ex. Cloud Services , SaaS , PaaS , IaaS , Containers , Machine Learning & AI in IT and of course how IoT will be there also.

Itamar Yerushalmi

Written by

Cloud Solution Architect @TeraSky LTD

Talking Tech Around

Talking about startups and diffrent types of technologies that interests me. Most of the technologies will be in the new type of information technology for ex. Cloud Services , SaaS , PaaS , IaaS , Containers , Machine Learning & AI in IT and of course how IoT will be there also.