Down The RabbitMQ Hole

Working with most queuing systems like RabbitMQ is one of those infrastructure write-once tasks you just expect to perform and forget. However, with a few key points to bear in mind, you could manage this boilerplate coding task with minimal investment of time, unscathed.

In the following lines I’ll try to portray some tips and major pitfalls we gathered when implementing a simple-yet-robust dispatching mechanism.

Robustness is a must

The number one priority when implementing a message queue dispatcher should be maintaining a robust connection, and making sure that pushing and dispatching messages will survive even the most unexpected situations.
The most naive implementation can poll a process queue using channel’s basicGet method:

At first glance, mind the problematic busy-wait, this seems to perfectly serve our needs. But taking a closer look points out a code smell.

The rabbit channel might get closed for a myriad of reasons — for example, when a message gets published to the wrong exchange. We’ll know this is the case if our attempts to draw messages from the channel results in an AlreadyClosedException. When this happens we should not access channel directly, but should write a getChannel() method to make sure we’re not posting or listening to a broken channel:

While that is nice, it still doesn’t eliminate the need to perform busy-wait. Luckily, RabbitMQ allows us to bind the channel to a queue, and register a Consumer. This implements a handleDelivery callback to take care of incoming messages:

A simple “Hello World” example for implementing Consumers from the official RabbitMQ documentation

For the sake of robustness, this still does not fulfill our needs. Guess what’s going to happen if the connection or the channel gets abruptly terminated?
Yep, the consumer will no longer handle incoming messages…

But fear not, AutorecoveringConnection and AutoRecoveringChannel are here for the rescue! Using this mechanism will assure us that the channel gets recovered, and will allow us to register a RecoveryListener to handle our consumer registration again and resume work.

Always keep the model consistent under failures

Let’s assume for a moment that our system manages a state for a given object, which travels between several process “phases”. Sending the object back and forth between those phases is maintained using — you guessed right — Rabbit message queues.

Our model object has a status field to persist its current state, whether it’s on its way to get processed, currently in progress, or finished and sent to the next phase.

An object’s lifecycle is managed using a status and queues

A slightly careless management of those statuses can either lead to messages being processed twice, or to messages getting lost in the oblivion of the rabbit hole. Of course, a normal chain of events within every phase should look like this:

Let’s say we just finished “Phase X”. Now we want to change the status to “sentToY”, and send the message over to “Phase Y”.

Acknowledging the message before a successful queuing of the job to the next phase may result in a lost message in the event of a failure.
On the other hand, acknowledging the message after a successful job queuing will make sure we don’t leave a soldier behind: In that case the job gets requeued to “Phase X”, which should know not to perform its work again, but to reiterate the message queuing to the next phase.

We should also think about concurrent changes to the status across threads, so a potential lock should be maintained across the object’s status field.

Debugging message dispatching does not simulate real life events

One last thing to bear in mind is the configuration of heartbeat settings to the ConnectionFactory via the setRequestedHeartbeat method. This will determine the amount of time (in seconds) in which an idle channel will get closed.

Why is that so important? Because many debugging scenarios may hold back the heartbeat from being issued in time, thus closing down the channel.
When not working in the auto-recovery strategy we discussed before, this could be a real pain in debug scenarios, so you should keep that in mind.

The rabbit hole is deep, but we’ve got a map for you

So, a RabbitMQ dispatching and posting code can be a hassle if not maintained correctly, but I’ve tried to portray some pitfalls for your consideration. Hopefully this will help make that coding task a bit less complex and a whole lot faster for you to go through.

Looking for a chance to overcome challenges with creative solutions just like this one, with a talented and results-driven team of developers? Wibbitz is hiring in our Tel Aviv office! Click here to apply.