How to choose a Message Queue

Antoine Duprat
Feb 9, 2018 · 4 min read

I’m working in a team which is developing a mail server: James. This project is a subpart of the OpenPaaS project, involving multiple teams.

In this mail server we need to implement a distributed mail queue. A mail queue is a mandatory component of SMTP servers. It allows uncoupling receiving messages from their processing. The current implementation relies on an embedded ActiveMQ server with decades old JMS implementation. Time had come for a little lift-up!

But a mail queue is a complex system… Not only should it be an efficient work queue, but also many additionnal features should be implemented:

  • Priority: You might want to give a higher priority to your organisation email, compared to spam
  • Delays: Maybe you don’t want to send too much mails at once. Maybe you need to wait a bit before re-sending an email to a remote mail server in case of errors
  • Management: Mail server administrators expects to inspect the content of the mail queue, remove the elements they want, amongst other…

We could not implement it in a straightforward, and production grade fashion on top of RabbitMQ. Thus we decided to look for alternative solutions and message queues.

Every OpenPaaS team being using message queues, we had to choose the one that better fits people needs.

Defining requirements

In order to perform this, we decided to interview each team leaders. We defined a list of topic to discuss:

  • features they implement, and will implement on top of a message queue
  • limitations they currently encounter
  • experience with this message queue technology

We digged quite far, and discussed some hot topics:

  • at least once Vs at most once
  • availability Vs consistency
  • management capabilites: queue size, browse, …

As a conclusion, OpenPaaS is abstracting the message queue technology behind a messaging API. This API allows one to do publish/subscribe to specific topics, to perform broadcasts and to share work using work queues.

OpenPaaS also uses the message queue to allow some services to communicate between them. This includes information:

  • from the Mail server, James
  • from the calendar system, Sabre.
  • from the contacts service

Most of our use cases are relying on at least one delivery, but some of them would benefit from an exactly once semantic. We tend to favor consistency, and want as much management capabilities as we can.

This allows us to extract core requirements:

  • we need basic messaging capabilities
  • we need Consistency, at least one semantic
  • as a bonus, advanced management capabilities

But we also added some criteria which were not subjected to the interviews:

  • maturity of the project
  • community
  • performances, …

Hopfully, after running the interviews, there were no contradictions such as one team needs availability and an other one consistency. Those interviews helped us a lot in order to exclude some implementations from our final choice.

Short list

I will present here the ones that catched our attention the most:

RabbitMQ is the message queue currently being used by OpenPaaS, so no migration would be necessary. It offers a good and mature community. Some problems regarding clustering have been reported, including message loss and manual reconciliation upon partition. The bad point of this solution is that it doesn’t fit the advanced management features needed by James. We selected it for further investigation, and the quality of its documentation have been a decisive factor.

Kafka is a cutting edge streaming platform. It fulfill the requested features. As it exposes a distributed log, some features of a mail queue are easier to implement. Its community is strong and mature, and it is thought to clustering as a primary concern. Replay is a core concept. However, it’s architecture is complex and involve a ZooKeeper quorum. We selected it as well.

RocketMQ is a promissing, newly born Apache project. However, despite good performances, and an impressive feature set, the community is not very mature, mostly centered around the Alibaba company. The project is still under development. So we concidered despite all of its advantages choosing it would not be a wise choice.

Artemis, is the HornetQ, donated to the Apache foundation, and adopted by the ActiveMQ project. A rock-solid message queue, but sadly clustering it is hard. Some old school techonogies (including XML!) are involved. Thus we decided to not investigate further.

NSQ is decentralized messaging system. The messaging patterns we want are supported, but only a hack allows to gain *durability*. Clustering is a primary citizen concept, but at the cost of features. AMQP is not supported, replay neither. We decided it was not worth paying the cost of a migration to it.

And then we add to choose

None of the above presented technologies allows to fulfill James needs in a satisfying manner, while fitting our production standards. Thus we decided to exclude James features from the choice. The mail queue advanced capabilities will be implemented with a message queue/database combinaison.

Our strategy then became exploring with a POC the limitations regarding RabbitMQ use.

  • We provided a POC of OpenPaaS use cases on top of RabbitMQ
  • We conducted experiments on top of a dockerized rabbitMQ cluster: stoping nodes, producing and consuming on differents nodes, declaring exchanges and queues on different nodes.
  • We tested durability, production/consumption order.

A strategic meeting will then tell if we would like to perform the move to Kafka.

Linagora Engineering

We are Open Source Engineers, Hacking Awesome Stuff

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store