Running Apache Kafka on Docker

I am going to give a talk in pycontw 2017 about Apache Kafka. In the talk, I plan to arrange a live demonstration on my MacBook Pro. However, the environment is always the first item on my to-do list.

There are many ways to install and run Apache Kafka. It is getting more popular to deploy applications and application infrastructure by Docker. I’d like to share how I use docker to run Apache Kafka via Docker.


OUTLINE

  • Overview
  • Two docker compose YML files
  • Running Apache Kafka on Docker
  • Verify whether Apache Kafka started successfully

PRE-REQUISITES

  • Docker (Docker machine, docker compose)
  • (opt) Python for verifying Kafka broker

Overview

Usually, we need two containers at least : one for Kafka, the other for Zookeeper. So far (2017/05/26), there is no official image for Kafka yet in Docker Hub. I used two different Apache Kafka images as my test environments.


Two docker compose YML files

The docker compose YML files can be found in docker-compose_ap-kafka.yml and docker-compose_cp-kafka.yml.

There are several variables should be reviewed and modified based on the environment.

Images source: wurstmeister/zookeeper and wurstmeister/kafka:0.10.1.0–1; confluent/zookeeper and confluent/kafka:0.10.0.0-cp1

Environment variables:

KAFKA_ADVERTISED_HOST_NAME : the hostname/ip

KAFKA_ZOOKEEPER_CONNECT: URL to Zookeeper

Ports: 2181 for Zookeeper ; 9092 for Kafka


Running Apache Kafka on Docker

Create a docker-machine for kafka image from wurstmeister

After modifying the compose yml file ( host ip), please type the follow command to build docker containers for one node Apache Kafka.

docker-machine create —-driver virtualbox —-virtualbox-memory 6000 ap-kafka
eval $(docker-machine env ap-kafka)
docker-compose -f docker-compose_ap-kafka.yml up -d

Verify whether it started successfully

Now a Kafka broker is up and running, we should verify whether it started successfully. Although there are few third-party apps for monitoring and managing Kafka, we can simply use the following two ways to check it.

benchmark.py for testing performance of different python client libraries for Apache Kafka

Create a docker-machine for confluent kafka

The procedure for confluent kafka image is similar with the above.

docker-machine create —-driver virtualbox —-virtualbox-memory 6000 cp-kafka
eval $(docker-machine env confluent)
docker-compose -f docker-compose_cp-kafka.yml up -d

This small scale of Apache Kafka would be useful for Kafka API and client testing.


本文介紹了我如何在Mac上透過Docker compose 建立Apache Kafka 測試環境,依照docker images 的來源,有兩個獨立的Docker compose YML可以用來建立single node 的 Kafka Broker。

Ref:

  1. Apache Kafka official website
  2. Python kafka benchmark
  3. wurstmeister Docker Kafka image
  4. Docker image installation from Confluent