RABBIT MQ CLUSTERING GUIDE

This Tutorial Makes use of the Bitnami RabbitMQ AMI and is designed for aws ec2 instances; but can be modified to work elsewhere.

AWS specific setup
To cluster two or more nodes do the following.

• Deploy one instance using the Bitnami AMI. this should give you a single node rabbitmq server.
• Next, create and ami of the first node you have just created and use it to deploy two other nodes.
• Deploying the two ec2-instances and make sure all the instances or any more that you chose to add share the following:

1. security group
2. Region.
3. PEM file

• Give all the instance elastic ips.
• On each instance perform the following steps:

1. Log into the ec2-instances run the following command:

sudo /opt/bitnami/rabbitmq/sbin/rabbitmqctl stop_app
sudo /opt/bitnami/ctlscript.sh stop rabbitmq
sudo su
vim /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf

Make sure you rabbitmq-env.conf looks like below:

#Configuration variables of RabbitMQ
NODE_PORT=5672
NODENAME=rabbitmq@ip-*-*-*-*.<region>.compute.internal
HOME=/opt/bitnami/rabbitmq/rabbitmq_home
RABBITMQ_USE_LONGNAME=true

replace

ip-*-*-*-* with your instance's private ip eg: ip-172-15-78-40.eu-west-1.compute.internal

Then save the file and exit sudo and run the below commands:

sudo /opt/bitnami/ctlscript.sh start rabbitmq
sudo /opt/bitnami/rabbitmq/sbin/rabbitmqctl start_app

All should be well. At this point, you have now given your rabbitmq node an FQDN that is resolvable from another node.
Repeat this step for all available nodes.

CLUSTERING
From another node run the following command to cluster two nodes together.

sudo /opt/bitnami/rabbitmq/sbin/rabbitmqctl stop_app
sudo /opt/bitnami/rabbitmq/sbin/rabbitmqctl join_cluster rabbitmq@ip-*-*-*-*.<region>.compute.internal - (FQDN of one node)

The expected output should be :

Clustering node rabbitmq@<node-one>.compute.internal with rabbitmq@<node-two>.compute.internal

finally, run the following command to bring your node back up and initiate clustering.

sudo /opt/bitnami/rabbitmq/sbin/rabbitmqctl start_app

And that's IT!!! repeat the clustering commands on all other available nodes.

To set up an elb follow the following tutorial

GOTCHAS

ELB resets the connection with the application servers periodically. Thus, it is possible that your producers send a task to the ELB which gets lost because of this resetting of connection between the ELB and the nodes.
In such a situation, we will need to enable confirmation of the tasks published. The solution discussed here is specific to Celery-based integration with RMQ where the backend is built on Django. The following needs to be set in settings.py in such case:

BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True}

This way, your publisher will wait for the RMQ to send an ack before sending any further tasks to the same queue.

AOB:

Connections to rabbitmq are blocked, have a look at this tutorial.

REFERENCE:
- https://medium.com/squad-engineering/set-up-highly-available-rmq-cluster-on-aws-30f6882d06f1
- https://support.symantec.com/en_US/article.HOWTO100093.html
- http://blog.flux7.com/blogs/tutorials/how-to-creating-highly-available-message-queues-using-rabbitmq
- https://docs.bitnami.com/aws/infrastructure/rabbitmq/#how-to-connect-to-the-rabbitmq-management-panel