Installation of Apache Kafka on Ubuntu 16.04 , 18.04 and 20.04

Before ! lets understand Kafka and why only Apache Kafka.

Introduction :-
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Installation of Apache Kafka on Ubuntu

Kafka provides an asynchronous protocol for connecting programs together, but it is undoubtedly a bit different from, say, TCP (transmission control protocol), HTTP, or an RPC protocol. The difference is the presence of a broker. A broker is a separate piece of infrastructure that broadcasts messages to any programs that are interested in them, as well as storing them for as long as is needed. So it’s per‐ fact for streaming or fire-and-forget messaging.

Prerequisites :-
A. To run properly at least 4 GM RAM Server is needed for Apache Kafka
B. Apache Kafka written on JAVA so it will require to have java platform (JVM) on your server. JVM versions above 8.

Step 1 :- Create a User on system
Step 2 :- Download and Install Kafka binaries
Step 3 (Option A) :- if you want to setup only One Kafka broker
Step 3 (Option B) :- if you want to setup three 3 Kafka brokers
Step 4 :- Create a topic , Producer and Consumer
Step 5 :- Connect from Outside (Offset Explorer kafka Tool)

Step 1 :- Create a User on system

create a user called kafka with the useradd command:

Logged in as your non-root sudo user, create a user called kafka with the useradd command:

$ sudo useradd kafka

Set the password using passwd:

$ sudo passwd kafka

Add the kafka user to the sudo group with the adduser command

$ sudo adduser kafka sudo

Your kafka user is now ready. Log into this account using su:

$ su -l kafka

Step 2 :- Download and Install Kafka binaries
create a directory in /home/kafka called Downloads to store your downloads:

$ mkdir ~/Downloads

Use curl to download the Kafka binaries:

$ curl “https://www.apache.org/dist/kafka/2.1.1/kafka_2.11-2.1.1.tgz" -o ~/Downloads/kafka.tgz

Create a directory called kafka and change to this directory. This will be the base directory of the Kafka installation:

$ mkdir ~/kafka && cd ~/kafka

Extract the archive you downloaded using the tar command:

$ tar -xvzf ~/Downloads/kafka.tgz — strip 1

Step 3 (Option A) :- if you want to setup only one Kafka broker

First see what are the file present in “Kafka” Directory

modify the server.properties first.. So i add few lines below. you just need to open your properties file , find and modify.
if you are not sure then you can copy of the code from our Repository and make changes according to your requirement.
Link :- https://github.com/YetAnotherOpportunityofLearning/apachekafka

kafka@dev-k8s-master:~/kafka/config$ vi server.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
listeners=PLAINTEXT://0.0.0.0:9191
advertised.listeners=PLAINTEXT://10.10.28.12:9191
# A comma separated list of directories under which to store log files
log.dirs=/home/kafka/kafka/data/kafka1
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

Now might be you are confuse with this line
log.dirs=/home/kafka/kafka/data/kafka
I created a Additional directory for store the logs files (Kafka/broker) and store the snapshot (Zookeeper)

log.dirs=/home/kafka/kafka/data/kafka
dataDir=/home/kafka/kafka/data/zookeeper

modify the zookeeper.properties

kafka@dev-k8s-master:~/kafka/config$ vi zookeeper.properties
# the directory where the snapshot is stored.
dataDir=/home/kafka/kafka/data/zookeeper
# the port at which the clients will connect
clientPort=2181

# admin.serverPort=8080
server=localhost:2888:3888

Lets Start Zookeeper and Kafka Broker

$ cd bin
$ bin/zookeeper-server-start.sh -daemon /home/kafka/kafka/config/zookeeper.properties
$ bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server.properties

Now lets verify all the brokers are running fine or not

$ jps
Output :- 
     30755 Jps
     17162 Kafka (Means Kafka is running)
     31774 QuorumPeerMain (Means Zookeeper is running)

One More way to Verify the Broker and Zookeeper

$ bin/zookeeper-shell.sh localhost:2181
result will be look like.
  Welcome to ZooKeeper!
      JLine support is disabled
  WATCHER::
  WatchedEvent state:SyncConnected type:None path:null
$ ls /brokers/ids
output :- 
   {​​​​​​ 0 }​​​​​​ (it is your broker id)

Step 3 (Option B) :- if you want to setup three 3 Kafka broker
First see what are the file present in “Kafka” Directory

I will Recommend to you remove server.properties. and take files from our repository link :- https://github.com/YetAnotherOpportunityofLearning/apachekafka
please verify all the things are matching with your exceptions or not.

Open all the brokers and zookeeper file and verify.

Now might be you are confuse with this line
log.dirs=/home/kafka/kafka/data/kafka
I created a Additional directory for store the logs files (Kafka/broker) and store the snapshot (Zookeeper)

log.dirs=/home/kafka/kafka/data/kafka1 log.dirs=/home/kafka/kafka/data/kafka2 log.dirs=/home/kafka/kafka/data/kafka3

dataDir=/home/kafka/kafka/data/zookeeper

Lets Start Zookeeper and Kafka Brokers

$ cd bin

Lets Start Zookeeper and Kafka Broker

$ cd bin
$ bin/zookeeper-server-start.sh -daemon /home/kafka/kafka/config/zookeeper.properties
$ bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server1.properties
$ bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server2.properties
$ bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server3.properties

Now lets verify all the brokers are running fine or not

$ jps
Output :- 
     30755 Jps
     17162 Kafka (Means Kafka1 is running)
     31774 QuorumPeerMain (Means Zookeeper is running)
     9954  Kafka (Means Kafka3 is running)
     1855  Kafka (Means Kafka2 is running)

One More way to Verify the Brokers and Zookeeper

$ bin/zookeeper-shell.sh localhost:2181
result will be look like.
  Welcome to ZooKeeper!
      JLine support is disabled
  WATCHER::
  WatchedEvent state:SyncConnected type:None path:null
$ ls /brokers/ids
output :- 
   {​​​​​​ 0 , 1 , 2 }​​​​​​ (it is your brokers id)

Step 4 :- Create a topic , Producer and Consumer

Creating Topic

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic medium

Creating Producer

$ bin/kafka-console-producer.sh --broker-list localhost:9091,localhost:9092,localhost:9093 --topic medium

Creating Consumer

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9091,localhost:9092,localhost:9093 --topic medium

Step 5 :- Connect from Outside (Offset Explorer kafka Tool)

A. Open Offset click on New Connection

B. Give your Zookeeper Details “Host IP”
C. Open Advanced Option and give the Bootstrap Server details

D. Now click on Add Button and Cluster will Connect.

Thanks for reading the blog please must try because “Practice make you perfect”

Don’t Forget to give us Clap and share with Other’s.