Kafka Queue system with example

  1. Brief Introduction of Kafka Queue System.
  2. Setting up Kafka and configuring with maven projects.
  3. Examples for Kafka Producer and Consumer
  4. Gotchas which I faced in configuration.

Introduction:

Setup & Configuration:

  • Download the latest version of Kafka.
  • Please follow below commands to untar your downloaded file.
tar -xzvf kafka_2.11-0.10.1.0.tgz cd kafka_2.11-0.10.1.0
  • Kafka requires zookeeper server, If you don’t have one you can start using below command
bin/zookeeper-server-start.sh config/zookeeper.properties
  • Next step is to start Kafka service. In this example, I working on standalone Kafka server.
bin/kafka-server-start.sh config/server.properties
  • Next steps are to create the topic, add the message to the topic(produce) and read the message from the queue(consume)
  • Create a topic as test
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Producer and Consumer Examples

  • I wrote simple Producer and Consumer in Java. Please take a look Git project at Kafka101
  • Please run, ProducerDemo to push messages to topic test queue.
  • Then following, Please run ConsumerDemo to read messages.

Gotchas?

  • Implementing Kafka Consumer, there was the problem with Library kafka_clients latest version 0.10.x.x, KafkaConsumer.poll the method was not implemented which will return always null;
  • Please make sure you use 0.9.x.x version of the library or else check with latest version implementation.

--

--

--

Software Engineer , Build Everything required.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sreedhar Bukya

Sreedhar Bukya

Software Engineer , Build Everything required.

More from Medium

Kafka Customer → Get Specific Messages Only

How Apache Camel Facilitates Your Integration Tasks: Part 2

Apache Kafka: Core Concepts and Use Cases

Consume AVRO Messages from Kafka without schema