A Visual Guide to Message Queues

Data is important. Losing data is bad.

Humans are imperfect. Humans make software to operate on data. Software is imperfect. Software crashes and loses your data. You are now sad.

So, how do we help software to not lose our data, even in the event of an unprecedented crash? This is where the concept of message queues comes into play!

A Simple Scenario

Let’s say you just created a mass-rad article on Message Queues, and you now need to post it and send an update to your 7,673 subscribers. You hit the “send” button, the back-end stores the article in the database, invokes the function to send a notification to your subscribers and…

…while emailing subscriber #27, your application shits the bed for one reason or another. You’re now left with only a handful of subscribers receiving their update. You might be able to start the process over and have the emails send successfully, but even if you do, 26 people are now going to get two copies of the same notification from you.

How can we ensure that a notification gets sent to each subscriber only once, even if something goes wrong along the way?

Enter The Message Queue

A message queue is a service that stores data as “messages” in a list known as — here it comes — a queue. When your application wants a task to be performed on an important piece of information, you provide that information as a “message” to the message queue.

Here we see a representation of the app (left) sending a message to the message queue (right). The message queue is represented as a rabbit to symbolize RabbitMQ; a popular, open source message queue solution.

Programs known as “workers” tell the message queue that they wish to be informed about new messages that they can operate on. Multiple workers can subscribe to a queue and the queue manager will make sure that a message is only given to one worker at a time.

On the left we see a “worker” requesting a message and on the right we have the queue manager.

The message queue then hands off a copy of the message to the next worker in line. The message queue will keep track of when it handed off the message to the worker and will not delete the message until the worker tells the manager that the task was completed.

The worker has suffered some unexpected tragedy and failed to report back to the queue manager. So the queue manager puts the previously assigned message back into the queue to be handled by another worker.

After a period of time, if no updates have come from the worker, the manager will add the message back into the queue to be handled by another worker.

This is why message queues can be such a powerful tool for creating more fault-tolerant systems. By enforcing a worker to be accountable for reporting on its operational state, our queue manager can make an informed assumption that something has gone wrong and can try the request again.

Back To Our Scenario

We publish a new article and want to send it to our many subscribers. Depending on your actual scenario and the technology you’ve chosen for your message queue, there may be more efficient ways to handle bulk records. To stay focused on the concepts and keep things simple, we’re simply going to add a message to the queue for each subscriber that should be notified.

We then have one or more “Send Notification” workers who can get the message off the queue, send the notification to the subscriber and inform the message queue manager that they were able to successfully complete the operation.

With the exception of completely broken code in our workers, we can sleep soundly knowing that our subscribers will each be updated only once about our new article. Even if one of our workers goes down in the process.

One Last Thing

More than ever, applications are being split up into pieces and communication between these components is vital. It doesn’t matter if you’re deploying multiple instances of the same code base across an array of machines or adopting a distributed pattern like micro-services. You’re going to need a reliable way to inform multiple systems of important data that’s not purely temporal.

Solutions like RabbitMQ, Amazon’s Simple Message Service, and Google PubSub are among the many options available that can provide the ability to relay important information to the various components of your application.

Well, that’s about it for this article. Keep in mind that this is a conceptual introduction to message queues. We’ve barely scratched the surface of the different functionality or features some message queue technologies can offer. If you want to learn more, I recommend installing RabbitMQ and playing with it for yourself.