Installation Guide for Kafka Cluster with 3 Zookeeper Nodes and 3 Broker Nodes
To set up a Kafka cluster with 3 Zookeeper nodes and 3 broker nodes, you will need the following requirements:
- Install Java: Kafka requires Java to run, so you should ensure all machines in your Kafka cluster have Java installed.
- Kafka Binary Copy in all 3 machine
Let's start the installation of the Kafka Cluster.
I am using Centos 8 OS for Cluster.
Setup Zookeeper Cluster with High Availability
Configure the DNS entries locally on all servers. If DNS is already configured, there is no need to mention the server IP addresses.
vim /etc/hosts
Download zookeeper tar on all node
wget https://downloads.apache.org/kafka/3.4.0/kafka_2.12-3.4.0.tgz
Extract zookeeper package on all node
tar -zxvf kafka_2.12–3.4.0.tgz
Then Create the same directory name in all nodes and move the binary to that directory.
mkdir /app
mv kafka_2.12–3.4.0 /app
Configure multiple zookeeper.properties on all node
Location : /app/kafka_2.12–3.4.0/config/zookeeper.properties
Create a directory for zookeeper dir
Mkdir –p /app/data/zookeeper
Add values in the zookeeper.properties file
vim zookeeper.properties
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/app/data/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
server.1=192.168.213.133:2888:3888
server.2=192.168.213.134:2888:3888
server.3=192.168.213.135:2888:3888
:wq
After save from file.
Set id in multiple zookeepers, setup each node
on node zookeeper1, add the file
vi /data/zookeeper/myid
1
:wq
on node zookeeper2, add the file
vi /data/zookeeper/myid
2
:wq
on node zookeeper3, add the file
vi /data/zookeeper/myid
3
:wq
To verify the Zookeeper Id in all nodes.
Now Start zookeeper in all node
Command
./bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties
After some time, zookeeper services come up in 30 second
Centos — 1
Centos — 2
Centos — 3
Tested telnet checked zookeeper up and running.
well, it is done, the zookeeper cluster is running well. And then we will set up the Kafka cluster.
Setup Kafka Cluster Multi Broker
Create directory for kafka data on all node
mkdir /app/data/kafka-logs
Setup zookeepers connect on server configuration, exec on all node kafka cluster, edit this file
set unique broker id each node kafka cluster, exec on Centos-1
vim /app/kafka_2.12–3.4.0/config/server.properties
broker.id=0 # unique value
listeners=PLAINTEXT://localhost:9092
delete.topic.enable=true
auto.create.topics.enable=false
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/app/data/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=2
zookeeper.connect=192.168.213.133:2181,192.168.213.134:2181,192.168.213.135:2181
Execute on Centos-2
vim /app/kafka_2.12–3.4.0/config/server.properties
broker.id=1 # unique value
delete.topic.enable=true
auto.create.topics.enable=false
log.dirs=/app/data/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=2
zookeeper.connect=192.168.213.133:2181,192.168.213.134:2181,192.168.213.135:2181
Execute on Centos-3
vim /app/kafka_2.12–3.4.0/config/server.properties
broker.id=2 # unique value
log.dirs=/app/data/kafka-logs
listeners=PLAINTEXT://localhost:9092
delete.topic.enable=true
auto.create.topics.enable=false
advertised.listeners=PLAINTEXT://localhost:9092
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=2
zookeeper.connect=192.168.213.133:2181,192.168.213.134:2181,192.168.213.135:2181
Now Start Broker in all node
Command to start the broker
./bin/kafka-server-start.sh -daemon ./config/server.properties
ss -tulpn | grep java && Check broker connection status.
Well, it is done, the Broker is running well. And then we will set up the Kafka cluster.
All broker node has been connected with the zookeeper.
