Kafka on The Microservice Architecture
Kafka Introduction and Its Role In the Microservice World
The rise of Microservice Architecture may help companies build highly scalable products; while at the same time, opening up new points of failure we never had before. The Microservice architectural model dictates that a system should be divided into small but independent services. You might have tens or hundreds of services to maintain instead of a big chunk of monolith application.
Although each service is disconnected, you might still need integration in the form of data passing between each service. Let say you have ten services, then you might end up with up to 45 integrations. Now, imagine having to integrate 100 services; it is a nightmare, isn’t it? Not to mention the different requirement for each service such as different types of connections, types of data format, and so on and so forth.
Kafka provides a solution for this particular problem where this tech product decouples each service with the Message Queue model. Kafka is a message broker between services, meaning the service sends data to Kafka without actually know which service is the recipient, vice versa. A producer, a service that sends messages, could just send Kafka’s data and assume the message was received. This pattern is basically a publish-subscribe messaging pattern which you can read more about here.
With this flow, a service now does not need to wait for a response from the recipient. This led Kafka to enable services to process data in real-time. Kafka also allows a system to be better in scalability rather than traditional service integrations.
This article will introduce you to Kafka components to provide you a basic understanding of what Kafka is so that you can see how simple the system is and help you into the rabbit hole of learning this fantastic tech product.
Kafka Topics, Partition, and Offset
One of the main components, when we are talking about Kafka, is Kafka Topics. This component labels specific message streams into a group based on the topic this data addresses, hence Kafka Topics. You can have as many Kafka Topics you want but make sure its label is unique.
In Kafka Topics, data will be split into several partitions. This partition will be in an ordered manner, and each partition is identified by its number. You need to specify how many partitions you need when creating a partition. Otherwise, It will adhere to the default configuration. But, don’t worry, you can change the partition number later.
Each message stored within a partition has incremental identifiers, called offset. A partition is independent, and an offset between the partition does not contain the same data. A message arrived into a partition guaranteed to be written sequentially (but not across a partition, it also does not always include the same number of offsets). Each message stored in Kafka is immutable, meaning you cannot change already stored data.
Kafka Broker, Producer, and Consumer
Kafka Broker is a server containing only some Kafka Topic partitions. Kafka will assign and spread a partition into a random broker. Kafka maintains this distribution, so you don’t have to implement this action. But, how do you access all the information in Kafka if it is distributed? In Kafka, every broker is a bootstrap server, meaning if you had access to one broker, you had access to the whole cluster of Kafka Broker.
Kafka implements the concept of data replication. Each partition has a backup somewhere in a different random broker so that in case of failure, the system still working. The number of backups it has is specified by the replication factor the user set when creating topics.
Each partition will have one leader at one time and many In-Sync Replicas (ISR). The leader is responsible for receiving and serving data; meanwhile, the ISR is just syncing the data. If a broker that contains a partition leader goes down, then one of the ISR will be the leader until that failed broker is running again.
A service called Zookeeper assigns the leader of a partition in the background, including the case of reassigning leader when a broker failure. Zookeeper is basically a broker management service to facilitate Kafka handle changes such as new topics, partition updates, and others. Importantly, to be able to run a Kafka service requires a running Zookeeper service.
A service that produces and sends information into Kafka Topics is called Producer. A producer does not need to acknowledge which broker they need to send it to; it is only need to include a message key and that message will always goes to the same partition. If producer sent messages without a key, then those messages will be sent round robin between broker. You can read more about round robin approach in the link here.
A producer can choose a message acknowledgment methods that is sent to Kafka. There are three cases of acknowledgment methods that can be selected by setting the acks. First, the Producer doesn’t want to know the acknowledgment (acks = 0). Second, Producer only gets the acknowledgment from the partition leader (acks = 1). Last, the Producer receives the acknowledgment from the partition leader and its ISR (acks = all).
A service that consumes (and subscribes) messages from topics is called Consumer. A Consumer just tell what topics they need to access without specifiying which broker to read from. It can read from multiple partitions and many topics as needed. The reading mechanism of a Consumer is simple; It read offsets sequencially within partition (But, it does not guarantee the reading order across partition). Whenever it read an offset, Kafka will save the last offset a consumer read. So, in case of consumer failure, the consumer will able to read from the last offset once it running again.
Hands-on
This hands-on will guide you to the “Hello, World!” of Kafka. We will run Zookeeper and Kafka service to send simple messages from our CLI.
Prerequisite
- Kafka
- JDK 1.8
Guide
First, make sure to install JDK 1.8 and Kafka first. You can check the requirements is installed by typing this simple command: java -version and kafka-topics.sh --version (it might be different from your machine).
Then, run the Zookeeper service by using the command zookeeper-server-start.sh <zookeeper properties>. First, You need to specify the configuration for the zookeeper. There is a default template file already that you can use in the installation folder.
Then, Open a new terminal, and run the Kafka service by using the command kafka-server-start.sh <kafka server properties>. Also, You need to specify the configuration for Kafka. You can use the default configuration found in the installation folder.
Then, open a new terminal, and create new Topics by executing this command kafka-topics.sh --zookeeper <zookeeper url>:<zookeeper port> --topic <topic name> --create --partitions <desired partition number> --replication-factor <desired partition number>
Next, get into Producer CLI by running this program kafka-console-producer.sh --broker-list <kafka url>:<kafka port> --topic <name of topic>
Last, Open another terminal, then run this command to open Consumer CLI kafka-console-consumer.sh --bootstrap-server <kafka url>:<kafka port> --topic <name of topic>
Result
We have been successfully run the Kafka service. Even though we only run Consumer and Producer from CLI, I think it is a significant baby step for you to explore more about Kafka in the future. The gif below is the result of the message passing process from Producer to Consumer. Congratulations!
Acknowledgement
You can learn in-depth about Kafka by watching this fantastic course from Stephane Maarek. I know a lot from this course, and I think you should too. You can access his teaching by the link below.
This article is the embodiment of my learning progress in Kampus Merdeka x Gojek internship program. Feel free to follow me to receive updates on my learning journey!
