Everything you need to know about Kafka
During our degree-related coursework, Queues, lists, and stacks are some of the basic data structures that we learn about. Additionally, we are learning how those Structures differ from one another. There are software programs like Kafka, RabbitMQ, ActiveMQ, and HiveMQ Message Brokers to deal with when we get into the IT industry. We are now attempting to incorporate everything you have learnt into this product. We don’t receive the entire range of this product as a consequence. Because have know idea how exactly work with this product. Because we are trying to map our fundamentals to these products. When you scale this application, you now have an issue since your topic or queue has many customers. We’re currently in a little bit of difficulty.
What is Message Broker?
A message broker is software that enables applications, systems, and services to communicate with each other and exchange information. The message broker does this by translating messages between formal messaging protocols. This allows interdependent services to “talk” with one another directly, even if they were written in different languages or implemented on different platforms. — IBM
What is Kafka?
Compared to message brokers like RabbitMQ, Kafka delivers substantially greater performance. It is an effective choice for creating queues since it boosts performance through sequential disk I/O. It is essential for big data use cases since it can achieve high throughput with little resource usage.
Here are some scenarios.
01. Scenario 01
In Kafka, a single topic may include several partitions. Employee topic and three partitions are shown in the above diagram. We also have a project consumer group, which counts as a single consumer. Messages are being taken by a single consumer (Consumer 1) from all three parts. We’re scaling the application now.
02. Scenario 02
We add a second consumer for 2nd scenario. Consumer 1 listens to Partitions 0 and 1 in this section. Consumer 2 listening to Partition 2. However, this is a totally random situation where the topics and listeners are related. Let’s go a little further.
03. Scenario 03
Right now, there are three partitions and three consumers. We’ll see what happens if we add one more consumer.
04. Scenario 04
Kafka does not transmit messages to Consumer 4 since the topic has three partitions. Consumer 4 is therefore not in use. As additional consumers are added, they all become idle.
Keep in mind, with respect to the number of consumers, we can have as many consumers as the maximum number of partitions you have.
Let’s consider if the consumer 1 or consumer 2 crashes then there is a one consumer already running so, then Kafka immediately transfer over to the remaining consumers. There is one more use case like this.
05. Scenario 05
If we open another consumer group to the same topic. The First consumer group has 3 consumers and second consumer group has 2 consumers. So each consumer has each partition. So how consumer knows where do I take the messages, so the Kafka support retention. So there is a mechanism they maintain called consumer offset
What is Consumer Offset?
The consumer offset is a way of tracking the sequential order in which messages are received by Kafka topics. Keeping track of the offset, or position, is important for nearly all Kafka use cases and can be an absolute necessity in certain instances, such as financial services.
Based on the consumer offset, Kafka knows particular consumer need to go to assigned partition respectively.
Number of consumers in the consumer group should be equal or less than to the number of partition in the topic
