Installation of Apache Kafka on Ubuntu 16.04 , 18.04 and 20.04
Before ! lets understand Kafka and why only Apache Kafka.
Introduction :-
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Kafka provides an asynchronous protocol for connecting programs together, but it is undoubtedly a bit different from, say, TCP (transmission control protocol), HTTP, or an RPC protocol. The difference is the presence of a broker. A broker is a separate piece of infrastructure that broadcasts messages to any programs that are interested in them, as well as storing them for as long as is needed. So it’s per‐ fact for streaming or fire-and-forget messaging.
Prerequisites :-
A. To run properly at least 4 GM RAM Server is needed for Apache Kafka
B. Apache Kafka written on JAVA so it will require to have java platform (JVM) on your server. JVM versions above 8.
Step 1 :- Create a User on system
Step 2 :- Download and Install Kafka binaries
Step 3 (Option A) :- if you want to setup only One Kafka broker
Step 3 (Option B) :- if you want to setup three 3 Kafka brokers
Step 4 :- Create a topic , Producer and Consumer
Step 5 :- Connect from Outside (Offset Explorer kafka Tool)
Step 1 :- Create a User on system
create a user called kafka with the useradd
command:
Logged in as your non-root sudo user, create a user called kafka with the useradd
command:
$ sudo useradd kafka
Set the password using passwd
:
$ sudo passwd kafka
Add the kafka user to the sudo
group with the adduser
command
$ sudo adduser kafka sudo
Your kafka user is now ready. Log into this account using su
:
$ su -l kafka
Step 2 :- Download and Install Kafka binaries
create a directory in /home/kafka
called Downloads
to store your downloads:
$ mkdir ~/Downloads
Use curl
to download the Kafka binaries:
$ curl “https://www.apache.org/dist/kafka/2.1.1/kafka_2.11-2.1.1.tgz" -o ~/Downloads/kafka.tgz
Create a directory called kafka
and change to this directory. This will be the base directory of the Kafka installation:
$ mkdir ~/kafka && cd ~/kafka
Extract the archive you downloaded using the tar
command:
$ tar -xvzf ~/Downloads/kafka.tgz — strip 1
Step 3 (Option A) :- if you want to setup only one Kafka broker
First see what are the file present in “Kafka” Directory
modify the server.properties first.. So i add few lines below. you just need to open your properties file , find and modify.
if you are not sure then you can copy of the code from our Repository and make changes according to your requirement.
Link :- https://github.com/YetAnotherOpportunityofLearning/apachekafka
kafka@dev-k8s-master:~/kafka/config$ vi server.properties# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1listeners=PLAINTEXT://0.0.0.0:9191
advertised.listeners=PLAINTEXT://10.10.28.12:9191# A comma separated list of directories under which to store log files
log.dirs=/home/kafka/kafka/data/kafka1# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
Now might be you are confuse with this line
log.dirs=/home/kafka/kafka/data/kafka
I created a Additional directory for store the logs files (Kafka/broker) and store the snapshot (Zookeeper)
log.dirs=/home/kafka/kafka/data/kafka
dataDir=/home/kafka/kafka/data/zookeeper
modify the zookeeper.properties
kafka@dev-k8s-master:~/kafka/config$ vi zookeeper.properties# the directory where the snapshot is stored.
dataDir=/home/kafka/kafka/data/zookeeper# the port at which the clients will connect
clientPort=2181
# admin.serverPort=8080
server=localhost:2888:3888
Lets Start Zookeeper and Kafka Broker
$ cd bin
$ bin/zookeeper-server-start.sh -daemon /home/kafka/kafka/config/zookeeper.properties
$ bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server.properties
Now lets verify all the brokers are running fine or not
$ jps
Output :-
30755 Jps
17162 Kafka (Means Kafka is running)
31774 QuorumPeerMain (Means Zookeeper is running)
One More way to Verify the Broker and Zookeeper
$ bin/zookeeper-shell.sh localhost:2181
result will be look like.
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
$ ls /brokers/ids
output :-
{ 0 } (it is your broker id)
Step 3 (Option B) :- if you want to setup three 3 Kafka broker
First see what are the file present in “Kafka” Directory
I will Recommend to you remove server.properties. and take files from our repository link :- https://github.com/YetAnotherOpportunityofLearning/apachekafka
please verify all the things are matching with your exceptions or not.
Open all the brokers and zookeeper file and verify.
Now might be you are confuse with this line
log.dirs=/home/kafka/kafka/data/kafka
I created a Additional directory for store the logs files (Kafka/broker) and store the snapshot (Zookeeper)
log.dirs=/home/kafka/kafka/data/kafka1 log.dirs=/home/kafka/kafka/data/kafka2 log.dirs=/home/kafka/kafka/data/kafka3
dataDir=/home/kafka/kafka/data/zookeeper
Lets Start Zookeeper and Kafka Brokers
$ cd bin
Lets Start Zookeeper and Kafka Broker
$ cd bin
$ bin/zookeeper-server-start.sh -daemon /home/kafka/kafka/config/zookeeper.properties
$ bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server1.properties
$ bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server2.properties
$ bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server3.properties
Now lets verify all the brokers are running fine or not
$ jps
Output :-
30755 Jps
17162 Kafka (Means Kafka1 is running)
31774 QuorumPeerMain (Means Zookeeper is running)
9954 Kafka (Means Kafka3 is running)
1855 Kafka (Means Kafka2 is running)
One More way to Verify the Brokers and Zookeeper
$ bin/zookeeper-shell.sh localhost:2181
result will be look like.
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
$ ls /brokers/ids
output :-
{ 0 , 1 , 2 } (it is your brokers id)
Step 4 :- Create a topic , Producer and Consumer
Creating Topic
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic medium
Creating Producer
$ bin/kafka-console-producer.sh --broker-list localhost:9091,localhost:9092,localhost:9093 --topic medium
Creating Consumer
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9091,localhost:9092,localhost:9093 --topic medium
Step 5 :- Connect from Outside (Offset Explorer kafka Tool)
A. Open Offset click on New Connection
B. Give your Zookeeper Details “Host IP”
C. Open Advanced Option and give the Bootstrap Server details
D. Now click on Add Button and Cluster will Connect.
Thanks for reading the blog please must try because “Practice make you perfect”
Don’t Forget to give us Clap and share with Other’s.