Running Apache Kafka on Docker

I am going to give a talk in pycontw 2017 about Apache Kafka. In the talk, I plan to arrange a live demonstration on my MacBook Pro. However, the environment is always the first item on my to-do list.

There are many ways to install and run Apache Kafka. It is getting more popular to deploy applications and application infrastructure by Docker. I’d like to share how I use docker to run Apache Kafka via Docker.


  • Overview
  • Two docker compose YML files
  • Running Apache Kafka on Docker
  • Verify whether Apache Kafka started successfully


  • Docker (Docker machine, docker compose)
  • (opt) Python for verifying Kafka broker


Usually, we need two containers at least : one for Kafka, the other for Zookeeper. So far (2017/05/26), there is no official image for Kafka yet in Docker Hub. I used two different Apache Kafka images as my test environments.

Two docker compose YML files

The docker compose YML files can be found in docker-compose_ap-kafka.yml and docker-compose_cp-kafka.yml.

There are several variables should be reviewed and modified based on the environment.

Images source: wurstmeister/zookeeper and wurstmeister/kafka:–1; confluent/zookeeper and confluent/kafka:

Environment variables:



Ports: 2181 for Zookeeper ; 9092 for Kafka

Running Apache Kafka on Docker

Create a docker-machine for kafka image from wurstmeister

After modifying the compose yml file ( host ip), please type the follow command to build docker containers for one node Apache Kafka.

docker-machine create —-driver virtualbox —-virtualbox-memory 6000 ap-kafka
eval $(docker-machine env ap-kafka)
docker-compose -f docker-compose_ap-kafka.yml up -d

Verify whether it started successfully

Now a Kafka broker is up and running, we should verify whether it started successfully. Although there are few third-party apps for monitoring and managing Kafka, we can simply use the following two ways to check it. for testing performance of different python client libraries for Apache Kafka

Create a docker-machine for confluent kafka

The procedure for confluent kafka image is similar with the above.

docker-machine create —-driver virtualbox —-virtualbox-memory 6000 cp-kafka
eval $(docker-machine env confluent)
docker-compose -f docker-compose_cp-kafka.yml up -d

This small scale of Apache Kafka would be useful for Kafka API and client testing.

本文介紹了我如何在Mac上透過Docker compose 建立Apache Kafka 測試環境,依照docker images 的來源,有兩個獨立的Docker compose YML可以用來建立single node 的 Kafka Broker。


  1. Apache Kafka official website
  2. Python kafka benchmark
  3. wurstmeister Docker Kafka image
  4. Docker image installation from Confluent