Implementing RabbitMQ and Kafka: common use cases
My name is Lior and I’m a Software Engineer at BigPanda. Our product was developed a few years ago, when asynchronous processing solutions were still in their early days. In recent months we encountered scalability challenges which drove us to look for a more appropriate solution than our current one which is rabbitMQ queue system.
When choosing between RabbitMQ and Kafka one should keep in mind that the two operate by different concepts: RabbitMQ is a more intuitive broker system. It leverages added flexibility by using exchanges. Kafka is considered a more complex system, engineered specifically for distributed systems problems and processing ‘big data’.
In this article I want to look at three common use cases, and for each use case I will present an implementation using each technology. For the following use cases I will refer to RabbitMQ’s queue and Kafka’s topic both as a queue for simpler comparison.
So what are the common use cases?
PubSub
Sending messages from one producer to multiple subscribers in a publisher subscriber pattern.
- RabbitMQ distributes events between subscribed consumers. In order to send all events to all subscribers we will need to use an exchange between the producer and the queues:
2. Kafka sends all published events to all subscribed consumers by default. Therefore we just need to configure each consumer group to subscribe to the same topic (“Queue”):
Routing
Routing is the ability to route each message to one or more different queues.
- In this case RabbitMQ is more intuitive than kafka. If we use an exchange the same way we did with the PubSub, we only need to define a routing_key for the exchange :
2. In Kafka there is no quick out of the box capability that enables routing each message to different queues. A possible solution will be to use a dedicated micro service which will consume the messages from “Queue1” (see image) and then will route them (will produce them to “Queue2” and “Queue3”) according to the routing rules we wish to apply. In this case the routing is not handled by the queue system but in the code we implement in “Consumer1” service:
Work Queue
The idea behind work queue is to be able to concurrently handle messages with multiple consumers
- To implement a work queue with RabbitMQ you bind the consumers to the same queue:
2. With Kafka we have a dedicated feature for this ability, You define each of the dedicated consumers as part of the same consumer group. This way the Kafka broker will handle the messages distribution between the consumers on that consumer group. The constraint is that the number of consumers in the consumer group is limited to the number of partitions you have :
Summary:
RabbitMQ and Kafka are useful systems for many use cases. We saw that typically we would prefer using Kafka to implement the PubSub use case, while for Routing it will probably be RabbitMQ. Either way, you should be familiar with the features provided by each system (exchange, consumer group, routing_key etc’) to make the best decision for your product.
