Using spring-rabbit under high throughput
In rabbitmq, a connection is a physical TCP connection between the broker and the client. Whereas, a channel is a lightweight virtual connection over the actual TCP connection. Having multiple connections certainly comes with an overhead of consuming a lot of system resources, but is sometimes required under high throughput.
spring-rabbit is one of the libraries on top of rabbitmq’s AMQP client which can be used for communication with the broker. spring-rabbit provides a CachingConnetionFactory to publish to broker. You can configure this factory in either channel caching or connection caching mode.
With these few lines of code and a locally running rabbitmq server, I tried publishing messages at several rates in different caching modes to the broker. (GitHub Repo for the experiment)
Channel Cache Mode
Under low throughput (100–1000 messages), low thread count (5–10), all the components behave normally.
With a thread pool of 10 (thus maximum 10 channels) publishing 1000000 messages, threads start blocking each other. Each of the thread has its own channel instance (hence no channel sharing). But all these channels are trying to write over the same socket at the same time. To avoid this from happening, SocketFrameHandler puts a monitor lock on the data object which is currently being transferred to the broker. This results in thread blocking to get a monitor lock on the data object under high concurrency.
Read up this stack overflow discussion for more details.
“arpit-rmq-5” #1019 prio=5 os_prio=31 tid=0x00007fb90c5b8800 nid=0x83303 waiting for monitor entry [0x0000700047bca000]
java.lang.Thread.State: BLOCKED (on object monitor) at com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(SocketFrameHandler.java:170)
Connection Cache Mode
Running the same setup in connection cache mode results in no thread blocking. Though do note, this may end up having network or bandwidth bottlenecks in a large environment.
Conclusion
Under high throughput, you should opt for ConnectionCache mode, but tune your connections and their internal channels in order to overcome network and bandwidth bottlenecks. (Will be experimenting this for my next post)
PS: This post is inspired by a similar issue we recently faced in our production environment ONLY during high throughput on the system, followed by a set of experiments to reproduce and understand the same.
