Running Apache Kafka on Docker

I am going to give a talk in pycontw 2017 about Apache Kafka. In the talk, I plan to arrange a live demonstration on my MacBook Pro. However, the environment is always the first item on my to-do list.

There are many ways to install and run Apache Kafka. It is getting more popular to deploy applications and application infrastructure by Docker. I’d like to share how I use docker to run Apache Kafka via Docker.

OUTLINE

• Overview
• Pre-requisites
• Two docker compose YML files
• Running Apache Kafka on Docker
• Verify whether Apache Kafka started successfully

Overview

Usually, we need two containers at least : one for Kafka, the other for Zookeeper. So far (2017/05/26), there is no official image for Kafka yet in Docker Hub. I used two different Apache Kafka images as my test environments.

Pre-requisites

• Docker (Docker machine, docker compose)
• (opt) Python for verifying Kafka broker

Two docker compose YML files

The docker compose YML files can be found in docker-compose_ap-kafka.yml and docker-compose_cp-kafka.yml.

There are several variables should be reviewed and modified based on the environment.

❶ Images source: wurstmeister/zookeeper and wurstmeister/kafka:0.10.1.0–1; confluent/zookeeper and confluent/kafka:0.10.0.0-cp1

❷ Environment variables:

KAFKA_ADVERTISED_HOST_NAME : the hostname/ip

KAFKA_ZOOKEEPER_CONNECT: URL to Zookeeper

❸ Ports: 2181 for Zookeeper ; 9092 for Kafka

Running Apache Kafka on Docker

Create a docker-machine for kafka image from wurstmeister

After modifying the compose yml file ( host ip), please type the follow command to build docker containers for one node Apache Kafka.

docker-machine create —-driver virtualbox —-virtualbox-memory 6000 ap-kafka
eval $(docker-machine env ap-kafka)
docker-compose -f docker-compose_ap-kafka.yml up -d

Verify whether it started successfully

Now a Kafka broker is up and running, we should verify whether it started successfully. Although there are few third-party apps for monitoring and managing Kafka, we can simply use the following two ways to check it.

• via Apache Kafka shell script (from the official website )
• via benchmark.py (need to install python and necessary libraries)

benchmark.py for testing performance of different python client libraries for Apache Kafka

Create a docker-machine for confluent kafka

The procedure for confluent kafka image is similar with the above.

docker-machine create —-driver virtualbox —-virtualbox-memory 6000 cp-kafka
eval $(docker-machine env confluent)
docker-compose -f docker-compose_cp-kafka.yml up -d

This small scale of Apache Kafka would be useful for Kafka API and client testing.

本文介紹了我如何在Mac上透過Docker compose 建立Apache Kafka 測試環境，依照docker images 的來源，有兩個獨立的Docker compose YML可以用來建立single node 的 Kafka Broker。

Ref:

1. Apache Kafka official website
2. Python kafka benchmark
3. wurstmeister Docker Kafka image
4. Docker image installation from Confluent