Getting Started With Kafka

We are presuming that you know about Apache Kafka & its use cases. So without much theory here lets starts with setups. Basically in this guide we will configure a basic Kafka instance in an Ubuntu environment & write a very very basic python producer & consumer.

Step by step

Step 1: Download Kafka & Extract it
$ wget http://www-us.apache.org/dist/kafka/0.10.1.1/kafka_2.10-0.10.1.1.tgz
$ tar -xzf kafka_2.10-0.10.1.1.tgz

Step 2: Kafka is very much dependent on zookeeper. Start zookeeper.

$ cd kafka_2.10-0.10.1.1/
$ bin/zookeeper-server-start.sh config/zookeeper.properties

Step 3: Start Kafka

$ bin/kafka-server-start.sh config/server.properties

Kafka producer writes everything to a Kafka topic, and consumer will read from it. So before writing the consumer & producer program we will create a topic.

Step 4: Create a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --create \
--replication-factor 1 --partitions 1 --topic gps-logs

So far so good now we will install a python Kafka client. It is also assumed that pip is in your system.

Step 5:

pip install kafka-python

Step 6: Producer

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('gps-logs', b'another_message')

Step 7: Consumer

from kafka import KafkaConsumer
consumer = KafkaConsumer('gps-logs',bootstrap_servers=['localhost:9092'])
for msg in consumer:
    print (msg)

Thats all for now.