RabbitMQ vs. Kafka: Head-To-Head

Comparing the strengths and weaknesses of each

Eran Stiller
Feb 17 · 13 min read
Featured image — Photo by AbsolutVision on Unsplash

Introduction

As a software architect dealing with a lot of microservice-based systems, I often encounter the ever-repeating question: “Should I use RabbitMQ or Kafka?”

Note 1

If you are not familiar with the internal structure of RabbitMQ and Kafka, I highly recommend reading through part 1 of this post first. If you are not sure, then feel free to skim over the headlines and diagrams to at least get a glimpse of these differences.

Note 2

Following the previous post, some readers asked me about Apache Pulsar. Pulsar is another messaging platform that aims to combine some of the best that RabbitMQ and Kafka have to offer.


Notable Differences Between RabbitMQ and Kafka

RabbitMQ is a message broker, while Apache Kafka is a distributed streaming platform. This difference might seem semantic, but it entails severe implications that impact our ability to implement various use cases comfortably.

Message ordering

Photo by Arshad Pooloo on Unsplash

“Messages published in one channel, passing through one exchange and one queue and one outgoing channel will be received in the same order that they were sent.” — RabbitMQ Broker Semantics

Differently put, as long as we have a single message consumer, it receives messages in order. However, once we have multiple consumers reading messages from the same queue, we have no guarantee regarding the processing order of messages.

An example of lost message ordering when using RabbitMQ

Message routing

Photo by Webaroo on Unsplash

Message timing

Photo by Oliver Hale on Unsplash

Message retention

Photo by chuttersnap on Unsplash

Fault handling

Photo by Sarah Kilian on Unsplash
  1. Persistent failures — Failures that occur due to a permanent issue that cannot be resolved via additional retries. Common causes of these failures are software bugs or an invalid message schema (i.e., a poison message).
Consumer 1 can continue retries on message 1, while other consumers continue processing messages
Messages in the bottom partition are not handled if the consumer is stuck retrying a message

Scale

Photo by Graphic Node on Unsplash

Consumer complexity

Photo by John Barkiple on Unsplash
RabbitMQ consumers efficiently scale-up and scale-down
Kafka partitions cannot be removed, leaving consumers with more work after scaling down

When to Use Which?

Photo by Paulius Dragunas on Unsplash
  1. Message timing control (controlling either message expiry or message delay).
  2. Advanced fault handling capabilities, in cases when consumers are more likely to fail to process messages (either temporarily or permanently).
  3. Simpler consumer implementations.
  1. Message retention for extended periods, including the possibility of replaying past messages.
  2. The ability to reach a high scale when traditional solutions do not suffice.
  1. Availability of a managed cloud solution if applicable.
  2. The operational cost of each solution.
  3. Availability of SDKs for our target stack.

Closing Thoughts

I started this two-post series with the observation that many developers view RabbitMQ and Kafka as interchangeable. I hope that reviewing these posts assisted in gaining insight into these platforms’ implementation, and the technical differences between them.

Better Programming

Advice for programmers.

Thanks to Zack Shapiro

Eran Stiller

Written by

A software architect, consultant, and instructor. CodeValue founder & CTO. Microsoft Regional Director (MRD) & MVP on Azure. (https://stiller.blog)

Better Programming

Advice for programmers.

More From Medium

More from Better Programming

More from Better Programming

More from Better Programming

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade