Message Expiration Pattern Explained
How can a sender indicate when a message should be considered stale and thus shouldn’t be processed?
A message might have a limited lifetime, and if it is not processed within this period, it is useless and should be discarded.
For example, a pizza order is useless when its maximum wait time is passed. The customer might have considered ordering from another shop. Also, in a highly dynamic environment like stock trading, a BUY order must be processed within seconds. If not, the order will be irrelevant as the prices can fluctuate.
In this post, let’s see how a sender can indicate when a message should be considered stale and shouldn’t be processed.
Why expire messages?
After accepting a message from the sender, the messaging system guarantees that a message will eventually be delivered to the receiver. What it cannot guarantee is how long the delivery may take. For example, if the network connecting the sender and receiver is down for a week, it could take a week to deliver a message.
By the time the receiver gets such a message, its business value might have been lost. Perhaps, processing such a message might cause inconsistencies.
In such a situation, the sender can set the Message Expiration to specify a time limit of how long the message is viable. That is usually done by setting the Time-to-Live (TTL) property of a message to an arbitrary value. In most cases, it is expressed in milliseconds.
Once the TTL period elapses and the message still has not been consumed, the message will expire. The messaging system’s consumers will never see an expired message.
Message expiring is beneficial when receiving a stale message could cause harm or waste valuable processing resources.
A Message Expiration is like the expiration date on a milk carton. After that date, you shouldn’t drink the milk. Likewise, when a message expires, the messaging system should no longer deliver it. If a receiver still receives a message but cannot process it before the expiration, the receiver should throw away the message.
— Gregor Hohpe, Bobby Woolf
What happens after messages get expired?
Most messaging systems move the expired messages into the Dead Letter Queue (DLQ). After that, they can be manually examined and discarded if needed.
The other option is that the messaging system discards the expired messages.
However, each option can be configurable. For example, Azure Service Bus discards expired messages by default. But you can configure it to re-route them to the DLQ.
How to set the TTL?
That can be controlled in two ways.
1. At the individual message level
The sender can control the expiration for any individual message by setting the time-to-live property, which specifies a relative duration.
For example, by setting the TTL to 1 hour, the sender instructs the messaging system to expire the message after 1 hour if it had not been consumed.
The following shows how RabbitMQ’s Java publisher sets the TTL of a message which can reside in the queue for at most 60 seconds:
byte[] messageBodyBytes = "Hello, world!".getBytes();
AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
.expiration("60000")
.build();
channel.basicPublish("my-exchange", "routing-key", properties, messageBodyBytes);The same example for Azure Service Bus would be:
ServiceBusSenderClient senderClient = new ServiceBusClientBuilder()
.connectionString(connectionString)
.sender()
.queueName(queueName)
.buildClient();
// Create a message to send.
ServiceBusMessage message = new ServiceBusMessage(BinaryData.fromString("Hello world!"));
message.setTimeToLive(Duration.ofMinutes(1));
// send one message to the queue
senderClient.sendMessage(message);2. Queue/topic level message expiration
All messages sent into a queue or topic are subject to a default expiration period. When creating a queue or a topic, the default TTL value can be specified and adjusted later.
The default expiration is used for all messages sent to the queue/topic where time-to-live isn’t explicitly set. The default expiration also functions as a ceiling for the time-to-live value. Messages with a longer time-to-live expiration than the default value are silently adjusted to the default message time-to-live value before being enqueued.
Tricky situations
If a receiver is locking a message for reading, the expiration doesn’t affect that message. The receiver can process the message as usual. If the lock timeout expires or the message is abandoned (returned to the queue), the expiration takes immediate effect.
While the message is under lock, the receiver might be reading an expired message. At that point, the receiver has to decide whether to process the message or return it to the queue.
It is more of an implementation choice.
Takeaways
The sender of a message can control how long a message should live by setting its Time to Live (TTL) property. If the TTL passes, the message is considered expired and indicates that it should not be processed.
The messaging system automatically discards expired messages. Optionally, it can be configured to move them to the Dead Letter Queue.
Setting the TTL can be done at the individual message level and the queue/topic level. A queue/topic will have a default TTL by the time it is created. If you set the message level TTL to a value greater than that, the messaging system silently adjusts the TTL to the default value.
Extremely low TTL in the order of milliseconds or seconds may cause messages to expire before receiver applications receive them. Consider the highest TTL that works for your application.
