Kafka Multi-Tenancy Architecture: SSL client authentication

Itamar Yerushalmi
Nov 7, 2017 · 4 min read

Kafka is a distributed streaming platform. As such, we would want many consumers and producers to write to the same Kafka cluster.

In one of our latest projects in a large scale cyber-security company, we created a large scale deployment of a Kafka cluster that needed to serve approximately 10TB of digested data each day, split across over 100 million events. This was done in order to limit each customer’s ability to access only their topic in a multi-tenancy architecture.

There are a few guides over the internet to enable SSL client authentication on Kafka. Some of them are outdated or incomplete, while the others include overkill Kerberos implementation. So I created this one, which is based on Kafka’s official documentation at this link, hopefully you’ll find it useful.

Image for post
Image for post
Producer can write to all topics. Consumer can access secured topic only with right certificate.

For this example, I have one Kafka server and 2 clients (consumers).

Server Side:

Make sure that you have truststore and keystore JKSs for each server.

  1. In case you want a self signed certificate, you can use the following commands:

Make sure that you configure the certificate properly — this will be used later in the ACL configuration.

Image for post
Image for post

2. On Kafka servers — server.properties — add the following lines:

3. Add the following to the advertised.listeners and the listeners keys SSL values on port 9092:

4. Change

To

5. For logging — enable DEBUG on Kafka authentication by changing the line in log4j.properties —

From:

To:

6. On the client servers add the following lines to consumer.properties and producer.properties:

Notice that you are using different files for the different servers.

7. Produce messages to a new topic:

You should now be able to write to the topic from both servers — if you look at the kafka-authorizer.log on the server you will see the following messages:

Image for post
Image for post

8. Using the kafka-acl command — set permissions for one of the users — by default, it will remove permissions from the other user:

9. Now, if you try to write or read from the granted user, you will succeed —

Image for post
Image for post

and from the other user — permission denied.

Image for post
Image for post

Talking Tech Around

Talking about startups and diffrent types of technologies…

Itamar Yerushalmi

Written by

Cloud Solution Architect @TeraSky LTD

Talking Tech Around

Talking about startups and diffrent types of technologies that interests me. Most of the technologies will be in the new type of information technology for ex. Cloud Services , SaaS , PaaS , IaaS , Containers , Machine Learning & AI in IT and of course how IoT will be there also.

Itamar Yerushalmi

Written by

Cloud Solution Architect @TeraSky LTD

Talking Tech Around

Talking about startups and diffrent types of technologies that interests me. Most of the technologies will be in the new type of information technology for ex. Cloud Services , SaaS , PaaS , IaaS , Containers , Machine Learning & AI in IT and of course how IoT will be there also.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store