Simplifying Kafka Setup with Docker: A Step-by-Step Guide

Kafka With Docker

In today’s interconnected digital landscape, real-time data processing is crucial for powering various applications, from e-commerce platforms to IoT devices. Message streaming architectures play a pivotal role in enabling real-time communication and data processing.

Apache Kafka has become a cornerstone technology in the world of real-time data processing and streaming applications. Its robust architecture enables handling massive volumes of data with high throughput, fault tolerance, and scalability.

However, setting up Kafka traditionally can be a daunting task, requiring careful configuration and management of dependencies. In this guide, we’ll explore how Docker simplifies the installation process, allowing you to spin up Kafka and related services with ease.

Prerequisites:

Before we begin, ensure you have Docker installed on your machine. If not, you can download and install Docker from the official website: Docker.

Step 1: Setting up Docker Compose:

Docker Compose is a tool for defining and running multi-container Docker applications. It allows you to manage complex setups with ease. If you haven’t already installed Docker Compose, follow the instructions here to install it.

Step 2: Writing the Docker Compose File:

We’ll start by creating a docker-compose.yml file to define our Kafka setup. Below is a sample configuration that includes Kafka and Kafdrop:

version: "3"
services:
  kafdrop:
    image: obsidiandynamics/kafdrop
    restart: "no"
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "kafka:29092"
    depends_on:
      - "kafka"
  kafka:
    image: obsidiandynamics/kafka
    restart: "no"
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

In this configuration:

• We have two services: Kafka and Kafdrop.
• Kafka is exposed on ports 2181 (for Zookeeper) and 9092 (for external communication).
• Kafdrop, a web UI for monitoring Kafka topics, is exposed on port 9000.

Step 3: Starting Kafka and Kafdrop:

To start Kafka and Kafdrop, navigate to the directory containing your docker-compose.yml file and run the following command:

docker-compose up -d

This command will pull the necessary Docker images (if not already available) and start the Kafka and Kafdrop containers in the background.

Step 4: Verifying Installation:

Once the containers are up and running, you can verify the installation by accessing the Kafdrop UI in your web browser. Navigate to http://localhost:9000 view the Kafdrop dashboard. From here, you can monitor Kafka topics, partitions, and messages in real-time.

Kafdrop

Topic Messages

Additionally, you can interact with Kafka using command-line tools like kafka-console-producer and kafka-console-consumer. These tools come bundled with the Kafka distribution and can be accessed from within the Kafka container.

Step 1: Accessing the Kafka Container:

Use docker exec to access the shell of the Kafka container.

docker exec -it <kafka_container_id> /bin/bash

Replace <kafka_container_id> with the ID or name of your Kafka container.

Step 2: Running kafka-console-producer and kafka-console-consumer:

Once you're inside the Kafka container's shell, locate the Kafka binaries directory. It's typically located at /opt/kafka/bin/. You can run the following commands to produce and consume messages:

Producer:

# Producer
kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic

Consumer:

# Consumer
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning

By providing the full path to the Kafka binaries, you should be able to run kafka-console-producer.sh and kafka-console-consumer.sh without encountering the "command not found" error.

Pub/Sub Kafka

Search for Kafka Binaries:

You can use the find command to search for the Kafka binaries within the container. For example:

find / -name kafka-console-consumer.sh

Conclusion:

By leveraging Docker and Docker Compose, you can streamline the installation and management of Kafka, making it easier to develop, test, and deploy Kafka-based applications. The docker-compose.yml file provided in this guide serves as a starting point for building more complex Kafka setups tailored to your specific requirements.

That is all for this post, Apache Kafka is quite an extensive topic to cover in a single post, and I’m excited to see you in the next couple of posts with the following topics.

1. Implementation of Apache Kafka in Node.js
2. Kafka vs RebitMQ
3. Many more….

If you found this blog post useful then clap, comment, and follow.

🤝 Let’s connect on LinkedIn: Raheel Butt