Async REST services with messaging
Why do we need a messaging system?
The majority of the time, we use asynchronous REST to create microservices. But HTTP, a synchronous protocol, is what we employed in this case. So, REST becomes synchronous when we use microservices. We employ a message method to get around this problem.
RabbitMQ
Using que management protocol
Smart Broker-Routing, content-based routing in the broker
Kafka
Similar as the RabbitMQ but consider as stream processing system
Dumb Broker
For routine queue management, as when we submit something to the queue, take the item that arrived first, process it, provide an output, and take the item that comes next, RabbitMQ is ideal.
When a message arrives at a queue used by Kafka, it copies the message to each queue in the set and can be directed to a specific queue. We choose the queue from the broker based on a few message attributes.
If anything happens, we may apply Kafka to notify others.
We have more data in the queue when we have a processing capacity of 10 messages per minute and a consumer processing speed of 15 seconds per message.
RabbitMQ — Circularly spread these messages to many customers. By expanding the number of consumers, we can accelerate this. If we have certain communications that take longer to process than others, the messages for lower consumers will require less time. Because it must wait for one message to be processed before processing the next, upper consumers form a queue. RabbitMQ will continue sending messages and building a queue even after your process is complete. Due to this, lesser consumers in this case become idle. When prefetch is set to 2, RabbitMQ’s fair dispatch feature, which uses prefetch, will delay sending messages until the first two messages are processed.
If a message is received by a consumer and cannot be processed right away, it is not in a queue and is not anywhere. In order to avoid using ACK, RabbitMQ tells when to delete the message and notifies the other party that they can remove the message since the message has finished processing.
Kafka — Kafka has a consumer group when taking into account the identical scenario as previously. Within a single topic, we can divide it into several sections. whenever we post a message on the subject. We start with a key, multiply it by the number of partitions, and then check to see which partition appears. We can ensure that the same customer’s message is sent into the same partition if we use our customer number as the key. This order is guaranteed to be in the division and not the subject. To make sure that each partition has its own consumer, we may take a consumer and direct it to listen to the partitions. Configuring a consumer group is what it is called. No should have fewer customers than or equal to the number of partitions. If we provide a message from a certain point, Kafka has a consumer offset and knows to give the following message from that point on.
We may utilize Kafka when RabbitMQ is required.
In Kafka, there is typically one topic and several partitions.
Each consumer in this setup is listening to each partition, but if we add more, the additional consumers will be idle. The number of consumers in a consumer group must be equal to or fewer than the number of partitions in a subject, according to a regulation. If we remove one active consumer, the additional partition begins to hear the partition.
There is no assurance of order between partitions in Kafka.
If we provide the producer the key, it will apply the mod algorithm to determine what to do depending on the key. The use of the same key ensured that all went to the same partition. The round robin method used by Kafka if the key is not provided is undesirable since it does not ensure that orders will be fulfilled for the same customer and across messages. We must define the key in order to ensure the order, but there is no order guarantee within the partition.
