Quick Installation of Multi-broker Apache Kafka Cluster

Kafka is the next big thing in the pocket of Big Data store. Industries have come up with different kind of use-case for Apache Kafka. Initially, Kafka was developed by Linkedin as a distributed message transfer system. Lately, Industries have been using this tool for not only transferring the data but also for transferring the data between two points. As we are in the world of big data and mostly companies don’t keep the whole data in one place. In the end, if they want to pursue big data analytics then they have to move the data in one place. This movement leads to the our use-case in which we will be using Kafka for transferring the data from one source to other. We will be talking about this use-case in later posts. In this post, we will learn how to install Apache Kafka.

In this tutorial, we are assuming that you don’t have any instance of Zookeeper or Kafka on your machine.

  • First, download the Kafka 0.8.2.0 version using the following link:
http://apache.arvixe.com/kafka/0.8.2.0/kafka_2.10-0.8.2.0.tgz
  • Just Untar the files and move to the folder:
tar -xvzf kafka_2.10–0.8.2.0.tgz
cd kafka_2.10–0.8.2.0.tgz
  • Kafka is based on the zookeeper service and the good thing about Kafka installation is that it offers the installation of the zookeeper service as well. Before you start kafka server you will need to ensure that you have started Zookeeper service.
bin/zookeeper-server-start.sh config/zookeeper.properties
  • Now you can start the Kafka Server
bin/kafka-server-start.sh config/server.properties

You are set to go!! Now if you are willing to setup a multi-broker kafka cluster then you will have to add new nodes to the server without taking the servers down.

You will need to copy the server.properties and change the following properties in the new file:

cp config/server.properties config/server-1.properties

config/server-1.properties:

broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1

Thats it!! You have your multi-broker Kafka Cluster ready for hands on.


Originally published at datascienceandyou.wordpress.com on September 26, 2015.

Show your support

Clapping shows how much you appreciated Abhinav Bhatnagar’s story.