Getting Started With Kafka
We are presuming that you know about Apache Kafka & its use cases. So without much theory here lets starts with setups. Basically in this guide we will configure a basic Kafka instance in an Ubuntu environment & write a very very basic python producer & consumer.
Step by step
Step 1: Download Kafka & Extract it
$ tar -xzf kafka_2.10-0.10.1.1.tgz
Step 2: Kafka is very much dependent on zookeeper. Start zookeeper.
$ cd kafka_2.10-0.10.1.1/
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Step 3: Start Kafka
$ bin/kafka-server-start.sh config/server.properties
Kafka producer writes everything to a Kafka topic, and consumer will read from it. So before writing the consumer & producer program we will create a topic.
Step 4: Create a topic
bin/kafka-topics.sh --zookeeper localhost:2181 --create \
--replication-factor 1 --partitions 1 --topic gps-logs
So far so good now we will install a python Kafka client. It is also assumed that pip is in your system.
pip install kafka-python
Step 6: Producer
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
Step 7: Consumer
from kafka import KafkaConsumer
consumer = KafkaConsumer('gps-logs',bootstrap_servers=['localhost:9092'])
for msg in consumer:
Thats all for now.