Running Apache Kafka on Docker

Suci Lin

I am going to give a talk in pycontw 2017 about Apache Kafka. In the talk, I plan to arrange a live demonstration on my MacBook Pro. However, the environment is always the first item on my to-do list.

There are many ways to install and run Apache Kafka. It is getting more popular to deploy applications and application infrastructure by Docker. I’d like to share how I use docker to run Apache Kafka via Docker.


  • Overview
  • Pre-requisites
  • Two docker compose YML files
  • Running Apache Kafka on Docker
  • Verify whether Apache Kafka started successfully


Usually, we need two containers at least : one for Kafka, the other for Zookeeper. So far (2017/05/26), there is no official image for Kafka yet in Docker Hub. I used two different Apache Kafka images as my test environments.


  • Docker (Docker machine, docker compose)
  • (opt) Python for verifying Kafka broker

Two docker compose YML files

The docker compose YML files can be found in docker-compose_ap-kafka.yml and docker-compose_cp-kafka.yml.

There are several variables should be reviewed and modified based on the environment.

Images source: wurstmeister/zookeeper and wurstmeister/kafka:–1; confluent/zookeeper and confluent/kafka:

Environment variables:



Ports: 2181 for Zookeeper ; 9092 for Kafka

Running Apache Kafka on Docker

Create a docker-machine for kafka image from wurstmeister

After modifying the compose yml file ( host ip), please type the follow command to build docker containers for one node Apache Kafka.

docker-machine create —-driver virtualbox —-virtualbox-memory 6000 ap-kafkaeval $(docker-machine env ap-kafka)docker-compose -f docker-compose_ap-kafka.yml up -d

Verify whether it started successfully

Now a Kafka broker is up and running, we should verify whether it started successfully. Although there are few third-party apps for monitoring and managing Kafka, we can simply use the following two ways to check it. for testing performance of different python client libraries for Apache Kafka

Create a docker-machine for confluent kafka

The procedure for confluent kafka image is similar with the above.

docker-machine create —-driver virtualbox —-virtualbox-memory 6000 cp-kafkaeval $(docker-machine env confluent)docker-compose -f docker-compose_cp-kafka.yml up -d

This small scale of Apache Kafka would be useful for Kafka API and client testing.

本文介紹了我如何在Mac上透過Docker compose 建立Apache Kafka 測試環境,依照docker images 的來源,有兩個獨立的Docker compose YML可以用來建立single node 的 Kafka Broker。


  1. Apache Kafka official website
  2. Python kafka benchmark
  3. wurstmeister Docker Kafka image
  4. Docker image installation from Confluent

Suci Lin

Written by

Suci Lin

Data Engineer, focus on stream processing and IoT. Passionate about data storytelling with Data Visualization and engineering culture within an organization.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade