Integrate Spring Boot with RabbitMQ easily
Heeey people!!
I would like to share this open-source library that makes it easier to integrate between a Spring Boot App and RabbitMQ. In addition, this library brings a new concept of retries which are more performative than the standard of the Spring AMQP.
Makes it easier? But how?
Auto-configuration
Let’s say, we want to create a custom communication (like one using SSL for example) with RabbitMQ using Spring AMQP. We would at least need to create a couple of beans like ConnectionFactory, RabbitAdmin, RabbitTemplate, and AbstractRabbitListenerContainerFactory.
Now imagine if we have more than 1 virtual host in your RabbitMQ. So, by default in Spring AMQP, we need to create this ecosystem for each virtual host and we need to manually manage the RabbitTemplate and RabbitListener beans.
As you can see, all these configurations take a long time to work well and could be a headache.
Thinking about this problem, this library auto-configures all these beans and makes the management for you. The only thing that you need to do is define the configuration in your application.properties and the magic happens!
Wow, but how this config at application.properties look like?
The pattern that we’ve created is very simple. First, we have a prefix for our properties that is spring.rabbitmq.custom.
After this, you need to inform us of the name of your event. This part is very important because all the management that this library does is based on this name.
After the name of the event, you can set the property and the value. (We have documentation for all properties available.
So, our pattern is:
spring.rabbitmq.custom.<YOUR_EVENT>.<PROPERTY>=<VALUE>Below we have an example of a configuration for 2 different connections.
spring.rabbitmq.custom.some-event.host=localhost
spring.rabbitmq.custom.some-event.port=5672
spring.rabbitmq.custom.some-event.ttlRetryMessage=5000
spring.rabbitmq.custom.some-event.maxRetriesAttempts=5
spring.rabbitmq.custom.some-event.ttlMultiply=2
spring.rabbitmq.custom.some-event.queueRoutingKey=ROUTING.KEY.TEST
spring.rabbitmq.custom.some-event.exchange=ex.custom.direct
spring.rabbitmq.custom.some-event.exchangeType=direct
spring.rabbitmq.custom.some-event.queue=queue.custom.test
spring.rabbitmq.custom.some-event.autoCreate=true
spring.rabbitmq.custom.some-event.concurrentConsumers=1
spring.rabbitmq.custom.some-event.maxConcurrentConsumers=1
spring.rabbitmq.custom.some-event.virtualHost=tradeshift
spring.rabbitmq.custom.some-event.primary=true
spring.rabbitmq.custom.some-event.sslConnection=true
spring.rabbitmq.custom.some-event.tlsKeystoreLocation=file:///etc/tradeshift/your-service/tls-keystore.pkcs12
spring.rabbitmq.custom.some-event.tlsKeystorePassword=${RABBITMQ_PASS_CERT}
spring.rabbitmq.custom.another-event.host=localhost
spring.rabbitmq.custom.another-event.port=5672
spring.rabbitmq.custom.another-event.ttlRetryMessage=5000
spring.rabbitmq.custom.another-event.maxRetriesAttempts=5
spring.rabbitmq.custom.another-event.queueRoutingKey=TEST.QUEUE
spring.rabbitmq.custom.another-event.exchange=ex_test_1
spring.rabbitmq.custom.another-event.exchangeType=direct
spring.rabbitmq.custom.another-event.queue=queue_test_1
spring.rabbitmq.custom.another-event.autoCreate=true
spring.rabbitmq.custom.another-event.concurrentConsumers=1
spring.rabbitmq.custom.another-event.maxConcurrentConsumers=1
spring.rabbitmq.custom.another-event.username=guest
spring.rabbitmq.custom.another-event.password=${RABBITMQ_PASS}As you can see, we don’t need to create a single line of code!!!
Nice, but you said something about new retry strategy, correct?
Yes, my friend, I did!
But before explaining this new strategy, let’s take a look at the Spring AMQP default behavior and RabbitMQ.
By default, RabbitMQ does not provide a retry strategy that allows us to control the whole life cycle of the message.
For example, until RabbitMQ 3.8, we did not have a property in the header to control the number of retries that Rabbit already did.
RabbitMQ default behavior:
- If you didn’t define the time-to-live argument, RabbitMQ will try to re-queue your message forever.
- If you defined a TTL but didn’t define a dlx, after TTL, RabbitMQ will remove your message to the queue and you will lose the message.
- If you defined a TTL and a dlx, after the TTL, RabbitMQ will send the message to the exchange defined in the dlx.
So, but what if we want to have a growing TTL (in case of instability, for example), and control the number of retries, how can we do this?
Excellent question!
Now, I’ll explain how Spring AQMP works and how Spring Rabbit Tuning lib works!
Retry Strategy
Default Spring AMQP
With Spring AMQP default, you can define retry configurations using the properties below, where simple is the default container bean.
But this strategy has a problem. By default, Spring AMQP will lock your queue when it tries to deliver the message in the retry process.
To resolve this use a workaround: Define the concurrency. But this way we are overloading the JVM, and it’s not the best approach. If you have 5 messages (following the example) with issues, you’ll have the bottleneck again, and we still need to define the beans manually in our @Configuration bean for each connection and container.
spring.rabbitmq.listener.simple.retry.enabled=true
spring.rabbitmq.listener.simple.retry.initial-interval=2000
spring.rabbitmq.listener.simple.retry.max-attempts=5
spring.rabbitmq.listener.simple.retry.multiplier=2
spring.rabbitmq.listener.simple.max-concurrency=5
spring.rabbitmq.listener.simple.concurrency=5Spring RabbitMQ Tuning
This library creates a strategy using another queue for retries. With this, we can control the TTL using the expiration message property and we can use the x-death message property to control the number of retries.
But how?
We are using the dlx concept in the retry queue to re-queue a message in the main queue. Doing this, we have access to the x-death property and we can define programmatically the message expiration.
Note: How this lib is an extension of Spring AMQP, if you want to use the default retry strategy, you can still use this lib only for auto configurations of beans.
So, is it possible to use this strategy without the autoconfigure? I already have a lot of beans and I don’t have time to change it now.
We know that the world is not perfect and thinking in situations like this, we create a feature flag that allows you to disable this autoconfiguration and use only the retry strategy and other benefits, like the management beans.
You can disable the autoconfiguration, using the properties below:
spring.rabbitmq.enable.custom.autoconfiguration=falseBut I have 2 different connections, how can I proceed in this case?
If you have more than 1 connection and you want to disable the autoconfiguration you need to provide the name of the RabbitTemplate bean for each connection. You can read more here.
spring.rabbitmq.custom.<YOUR_EVENT>.rabbitTemplateBeanName= <RABBITTEMPLATE_BEAN_NAME>Now you can still use our RabbitTemplateHandler bean to facilitate the experience with Spring AMQP.
Really nice! So, I want to use, how can I do this, do you have an example or documentation?
Yeees!
We have an example project in this repo where you can see how to use this lib for publishers and listeners.
But it is so simple that I’ll explain it here too!
Publisher
This lib has a class called RabbitTemplateHandler and it’s very simple to use. We just need to call the getRabbitTemplate method and pass the virtual host as a parameter for getting the correct bean of RabbitTemplate. Once you did this, you can call the convertAndSend method and pass the exchange and the routing key as parameters.
Note: You can get the exchange and the routing key using the @Value annotation.
Example:
@Value("${spring.rabbitmq.custom.some-event.exchange}")
private String exchangeSomeEvent;
@Value("${spring.rabbitmq.custom.some-event.queueRoutingKey}")
private String routingKeySomeEvent;
@Autowired
private final RabbitTemplateHandler rabbitTemplateHandler;
public void sendMessage(final String message) {
rabbitTemplateHandler.getRabbitTemplate("some-event").convertAndSend(exchangeSomeEvent, routingKeySomeEvent, message);
}Listener
Listeners are very simple too. The only thing that you need to do is annotate a method with the RabbitListener annotation (default annotation of Spring AMQP) and pass the name of containerFactory.
How do I know the name of the correct container factory for each virtual host?
You don’t need to know, just pass the name of your event and the library will do the magic for you!
This lib also has an option to enable the retry and dlq strategy, that is recommended to use.
It is very simple to enable this behavior. We just need to annotate our method with the EnableRabbitRetryAndDlq annotation and pass the name of the property as an argument.
Note: You also have an option to specify which exceptions you want to enable the retry and dlq strategy. By default, the value is Exception.class, which means that all exceptions will be handled.
Example:
@RabbitListener(containerFactory = "some-event", queues = "${spring.rabbitmq.custom.some-event.queue}")
@EnableRabbitRetryAndDlq(event = "some-event", exceptions = { IllegalArgumentException.class, RuntimeException.class })
public void onMessage(Message message) {
...
}And that is it! I hope you enjoy this library as such as we enjoyed it!
And, the most important thing: Feel free to contribute or to send feedback!
