I am going to give a talk in pycontw 2017 about Apache Kafka. In the talk, I plan to arrange a live demonstration on my MacBook Pro. However, the environment is always the first item on my to-do list.
There are many ways to install and run Apache Kafka. It is getting more popular to deploy applications and application infrastructure by Docker. I’d like to share how I use docker to run Apache Kafka via Docker.
- Two docker compose YML files
- Running Apache Kafka on Docker
- Verify whether Apache Kafka started successfully
Usually, we need two containers at least : one for Kafka, the other for Zookeeper. So far (2017/05/26), there is no official image for Kafka yet in Docker Hub. I used two different Apache Kafka images as my test environments.
- Docker (Docker machine, docker compose)
- (opt) Python for verifying Kafka broker
Two docker compose YML files
There are several variables should be reviewed and modified based on the environment.
❶ Images source: wurstmeister/zookeeper and wurstmeister/kafka:0.10.1.0–1; confluent/zookeeper and confluent/kafka:0.10.0.0-cp1
❷ Environment variables:
KAFKA_ADVERTISED_HOST_NAME : the hostname/ip
KAFKA_ZOOKEEPER_CONNECT: URL to Zookeeper
❸ Ports: 2181 for Zookeeper ; 9092 for Kafka
Running Apache Kafka on Docker
Create a docker-machine for kafka image from wurstmeister
After modifying the compose yml file ( host ip), please type the follow command to build docker containers for one node Apache Kafka.
docker-machine create —-driver virtualbox —-virtualbox-memory 6000 ap-kafkaeval $(docker-machine env ap-kafka)docker-compose -f docker-compose_ap-kafka.yml up -d
Verify whether it started successfully
Now a Kafka broker is up and running, we should verify whether it started successfully. Although there are few third-party apps for monitoring and managing Kafka, we can simply use the following two ways to check it.
- via Apache Kafka shell script (from the official website )
- via benchmark.py (need to install python and necessary libraries)
Create a docker-machine for confluent kafka
The procedure for confluent kafka image is similar with the above.
docker-machine create —-driver virtualbox —-virtualbox-memory 6000 cp-kafkaeval $(docker-machine env confluent)docker-compose -f docker-compose_cp-kafka.yml up -d
This small scale of Apache Kafka would be useful for Kafka API and client testing.