Running Kafka in Docker Compose

So today we will be talking about Kafka deployment with Docker Compose Services. I was working with Kafka in my project and initially, I was having issues as this was my first time working with Kafka.

As I was new to this Kafka, looking over multiple sources I finally figured what I need to know to make it work. So let see what issue I was facing before and you may have the same issue.

The issue I Faced Initially

When I ran the compose, it was working with my app using the container network only internally, but I missed this.

So when I tried to deploy the Kafka on one server and connect to another server running the app, It failed to connect. I thought of like server issue if blocked port or network, but no. The issue was with the Kafka configuration.

Kafka needs the listeners to be configured such that the access can be enabled differently. The types are like

1. Docker network Internal Access

It means the apps running in the container network can access the Kafka, but not outside can access. This is when you need to just make the Kafka communicate internally without external access.

Listener be configured like

ex: PLAINTEXT://kafka:9092

where kafka is the container name and 9092 is the access port

2. Host-Only Access

It means the apps running in the container network, as well as app running on a host external to docker network, can access the Kafka, but no external host can access. This is when you need to just make the Kafka communicate with services running on localhost only.

Listener be configured like

ex: PLAINTEXT://localhost:9092

where localhost is the local network and 9092 is the access port

3. External Host Access

It means the apps running in the container network, as well app running on the host or some another host (using IP of the host deployed) can access the Kafka with IP. This is when you need to make the Kafka communicate with external access to host other than local.

Listener be configured like

ex: PLAINTEXT://IP:9092

where IP is the IP address where kafka is deployed and 9092 is the access port

So, we talked about three ways of making the kafka access available.

What if you need to make multiple access for like, let say

You need to enable local access for some apps and also like external access using IP, so what you can do

You may use the IP for localhost as well, but this is just an example of multiple listeners, easy way

• configure one listener for localhost and one for Ip based
  like PLAINTEXT://localhost:9092 and PLAINTEXT://IP:9092

but you will get error for using same port. let us change the port and try again.

• like PLAINTEXT://localhost:9092 and PLAINTEXT://IP:29092

there will be one error again. This is related to the Listener PROTOCOL, we need to create a listener with a different name and declare PROTOCOL later.

• So we now add in a format like LISTENER_NAME:IP:port, where LISTENER_NAME is the name of the listener like, let say EXTERNAL or some name like JOHN and another be like BEN.
  Here we replaced the PLAINTEXT:// with the name which we need to mention with another property, which we will check next.
• So our listeners will be now, like JOHN://ip:29092 and BEN://localhost:9092 so there will be no error now
• Next, we need to define PLAINTEXT as the protocol for these like BEN:PLAINTEXT and JOHN:PLAINTEXT
• You will think like we just replaced the listeners to some variables and using now, but this is the way to declare multiple listeners with same protocol.

Important Properties for Compose for Kafka(Just the ones we using now)

1. KAFKA_ADVERTISED_LISTENERS

here we need to mention our listeners that we talked about above. Ex PLAINTEXT://IP:9092 or JOHN://IP:9092

2. KAFKA_LISTENER_SECURITY_PROTOCOL_MAP

if we use multiple listeners using names, then we need to add this property to define PROTOCOLS like JOHN://PLAINTEXT

Checkout official site for other properties and details https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html

Solution for same

So, we have seen the issues as well as the solution above. But here I will tell you what needs to be done in a simple way in summarised steps.

1. Create a compose file
2. Add one Kafka service
   Configure Listener as needed, like let say PLAINTEXT://localhost:9092 for enabling listener access to host applications only.
3. Add one zookeeper service

4. Add volumes optionally(for data persistence)

5. Ran the compose where you need to deploy.

Compose file For Kafka & Zookeeper Single node

Kafka Compose file

Also, do follow my other blog for tech-related posts as well as Guides.

4 Ways Performance Testing Can Streamline Digital Transformation

That is all for today, Hope you like the post. Show your support by clapping for the same. Also, let me know your thoughts below, I will try to improve/add more if required. Happy reading and deploying.