BigData & Web3. RabbitMQ High Availability and High Load. Part 2
Large flow of messages
Suppose you need help with RabbitMQ performance with a large flow of messages. In that case, we use a similar scheme, only not combining all messages into one final RabbitMQ, but processing messages on each instance
Alternatively, you can install one balancing RabbitMQ, work on the simplest and fastest mechanisms (fanout/direct exchange), and shovel messages from this rabbit into separate RabbitMQs with more complex and less productive logic. Limiting the length of messages from these internal RabbitMQs allows you to flexibly manage the load. Here it is more logical to place shovel on the side of internal RabbitMQs. This will make it easier to add new RabbitMQ instances to the scheme.
It is clear that here we are limited by the performance of the first RabbitMQ, but if you do not have enough speeds up to 30,000 mps, perhaps RabbitMQ is not suitable for such a project at all. Well, or you can unbalance LB
Large number of connections/channels opened/closed
It is desirable, of course, not to allow this, but if there is no choice, you can install AMQProxy between the publisher and RabbitMQ. It will take over all channel re-creations. As a result, RabbitMQ will see only one channel within which the publication occurs.
Important: Consumers should not be connected via this service.
Of course, all these solutions can and should be combined if necessary
AMQProxy can only work with one RabbitMQ entry point, so it needs to be installed before each one.
Resume: for high load acceptance, our main tools are HAproxy as a load balancer across shards, AMQProxy (if there are multiple re-creations of connections for publishing), and shovel (or its self-written analog) for transferring messages between RabbitMQ instances and/or clusters.
You can say Hi to us at Pharos Production — a software development company
Follow our product Ludo — the reputational system of the Web3 world
