Spring Kafka Tutorial: Setting Up Project

The project we will be building is a Notification System. The Service will expose REST API endpoints to send message and mail using different vendors. The responsibility of the service is to send the mail/message and notify back the caller about mail/message being successfully sent.

We will be creating a multi-module Spring boot project. Different modules would be:

1. DAO
2. Notification Publisher(APIs will be implemented here)
3. Notification Consumer: This sends out the mail/message to user via available vendor.

For the project we need the following configured:

1. IDE of your choice
2. Java 17 or higher: There are multiple open source JDK are out there. Install any one of those, configure it. You might find the installation instruction from where you download. I personally prefer Adoptium.
3. Maven: Follow this: https://maven.apache.org/guides/getting-started/windows-prerequisites.html
4. Kafka: Download Kafka from here: https://kafka.apache.org/downloads
5. Offset Explorer: We will use this to see our events, and the position of consumers. https://kafkatool.com/download.html

Starting up Kafka

1. Extract the file from the downloaded zip to location where you want to have Kafka.
2. Navigate to that folder and open terminal there.

To start Kafka, we first need to start Zookeeper. Zookeeper keeps track of brokers in a cluster, keep them connected and in sync. That’s why, we first start a Zookeeper. Once it is started, we starts Kafka brokers, each pointing to that one Zookeeper.

Go to the config folder in Kafka folder. You will find zookeeper.properties file there. There are two properties of our interest here.

1. dataDir: This defines the location where zookeeper will store the data related to the kafka brokers’ configs and states. You are free to change it, but I suggest to let it be to the default location.
2. clientPort: This is the port on which Zookeeper will be running on. Define any value of your choice. We usually need to change if the default port(2181) is already used by some other process or we need more than one instance of Zookeeper.

Once these are configured, run the below command in terminal:

Unix:

bin/zookeeper-server-start.sh config/zookeeper.properties

Windows

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

This will start the zookeeper on the port defined in properties file.

Now we need to configure two broker configs for our cluster. Both broker will connect to the zookeeper we just started.

Go to config folder, locate server.properties, create two copies of it, for each broker we will be starting. I named them server1.properties and server2.properties.

Both broker need to have two property value to be unique,

1. broker.id: The id for each broker should be unique in a cluster
2. listeners: This defines the protocol, IP and port combination where this broker will be listening to requests. If you don’t set this, it will run on default port, 9092. I have set this to 9092 and 9093.

Set the property zookeeper.connect to ip:port for running zookeeper instance. Run both broker one by one now:

Unix:

bin/kafka-server-start.sh config/server-1.properties

bin/kafka-server-start.sh config/server-2.properties

Windows:

.\bin\windows\kafka-server-start.bat .\config\server-1.properties

.\bin\windows\kafka-server-start.bat .\config\server-2.properties

You have started the Kafka server with two brokers.

Let’s create a topic, publish data, and see it getting consumed. The commands here are in windows, the commands in bash is almost the same, you just have to run the sh file in bin, instead of bat in windows.

 .\bin\windows\kafka-topics.bat --create --topic "my-event" --bootstrap-server localhost:9092

The above command, runs a script(kafka-topics) to create( — create) a topic( — topic) name “my-event”, by connecting to the broker(bootstrap server), running on “localhost:9092”.

You can pass in any broker address. The brokers communicate among each other, and share a metadata, which stores the information of topics(which broker has data for which topic). Let’s start a producer now, and produce some data(event) to this topic we just created.

.\bin\windows\kafka-console-producer.bat --topic "my-event" --bootstrap-server localhost:9093
>First Event
>Second Event

The above command, runs a script(kafka-console-producer) to produce a data to the topic( — topic) “my-event”, by connecting to the broker(bootstrap server), running on “localhost:9092”. Once you press enter, you will be prompt to enter data for the event. Enter any data, and press ctrl-C to exit producer.

Once the events are produced, we start the consumer to consume the events from Kafka brokers.

 .\bin\windows\kafka-console-consumer.bat --topic "my-event" --from-beginning --bootstrap-server localhost:9092

The above command, runs a script(kafka-console-consumer) to consume events in the topic( — topic) “my-event”, and we tell the consumer the position, from where to start reading events( — from-beginning). The consumer will read events by connecting to the broker(bootstrap-server) “localhost:9092”.

Note that we won’t be doing this in the rest of tutorial trail. We will producing and consuming from our application. If you want to know the working of the Kafka behind all these commands, do comment, I will make a detailed blog about the ins and outs.

Points to Ponder

1. Let’s say we have two brokers, running on port 9092 and 9093. We create a topic on one of the broker, and try to produce to another broker, what happens then?
2. Kafka is defined to be fail-safe, what if one of the broker fails, how Kafka ensures we don’t lose data?
3. Why use Kafka and not DB? Since Kafka saves data to drive, won’t it be slower than DB?

Don’t worry, I will be answering these in following blogs.