RabbitMQ is an open-source message-broker software that originally implemented the Advanced Message Queuing Protocol and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol, Message Queuing Telemetry Transport, and other protocols. It is lightweight and easy to deploy on premises and in the cloud. It supports multiple messaging protocols. RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements. Here we will see how to make RabbitMQ cluster with 2 nodes and make use of AWS Elastic Load Balancer and Auto Scaling Groups to handle to make more reliable and handle any load.
Create RabbitMQ cluster manually:
You can refer the ansible playbook which does all the installation and configuration (or) install and configure everything manually using the below steps.
Install RabbitMQ. For RabbitMQ to work we need to install erlang package first. Execute step 1 to 9 on both master and slave/mirror node.
- Import Erlang GPG Key.
$ wget -O- https://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc | sudo apt-key add -2. Add Erlang repository.
$ echo "deb https://packages.erlang-solutions.com/ubuntu bionic contrib" | sudo tee /etc/apt/sources.list.d/rabbitmq.list3. Now Install Erlang package
$ sudo apt update
$ sudo apt -y install erlang4. Let’s Install RabbitMQ now. Add RabbitMQ Keys.
$ wget -O- https://dl.bintray.com/rabbitmq/Keys/rabbitmq-release-signing-key.asc | sudo apt-key add -
$ wget -O- https://www.rabbitmq.com/rabbitmq-release-signing-key.asc | sudo apt-key add -5. Add RabbitMQ repository to ubuntu.
$ echo "deb https://dl.bintray.com/rabbitmq/debian $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/rabbitmq.list6. Now Install RabbitMQ server
$ sudo apt update
$ sudo apt -y install rabbitmq-server --fix-missing7. Check RabbitMQ Installation.
$ sudo service rabbitmq-server status● rabbitmq-server.service - RabbitMQ brokerLoaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)Active: active (running) since Wed 2020-01-29 19:01:02 UTC; 1 weeks 0 days ago
8. Create RabbitMQ user and provide full permissions.
$ /usr/sbin/rabbitmqctl add_user test test
$ /usr/sbin/rabbitmqctl set_user_tags test administrator
$ /usr/sbin/rabbitmqctl set_permissions -p / test ".*" ".*" ".*"9. Enable RabbitMQ plugins. Below are the plugins enabled.
$ sudo rabbitmq-plugins enable rabbitmq_management$ sudo rabbitmq-plugins listListing plugins with pattern ".*" ...Configured: E = explicitly enabled; e = implicitly enabled| Status: * = running on rabbit@ip-172-31-42-44|/[ ] rabbitmq_amqp1_0 3.8.2[ ] rabbitmq_auth_backend_cache 3.8.2[ ] rabbitmq_auth_backend_http 3.8.2[ ] rabbitmq_auth_backend_ldap 3.8.2[ ] rabbitmq_auth_backend_oauth2 3.8.2[ ] rabbitmq_auth_mechanism_ssl 3.8.2[ ] rabbitmq_consistent_hash_exchange 3.8.2[E*] rabbitmq_event_exchange 3.8.2[e*] rabbitmq_federation 3.8.2[E*] rabbitmq_federation_management 3.8.2[ ] rabbitmq_jms_topic_exchange 3.8.2[e*] rabbitmq_management 3.8.2[e*] rabbitmq_management_agent 3.8.2[ ] rabbitmq_mqtt 3.8.2[ ] rabbitmq_peer_discovery_aws 3.8.2[ ] rabbitmq_peer_discovery_common 3.8.2[ ] rabbitmq_peer_discovery_consul 3.8.2[ ] rabbitmq_peer_discovery_etcd 3.8.2[ ] rabbitmq_peer_discovery_k8s 3.8.2[ ] rabbitmq_prometheus 3.8.2[ ] rabbitmq_random_exchange 3.8.2[ ] rabbitmq_recent_history_exchange 3.8.2[ ] rabbitmq_sharding 3.8.2[E*] rabbitmq_shovel 3.8.2[E*] rabbitmq_shovel_management 3.8.2[ ] rabbitmq_stomp 3.8.2[ ] rabbitmq_top 3.8.2[ ] rabbitmq_tracing 3.8.2[ ] rabbitmq_trust_store 3.8.2[e*] rabbitmq_web_dispatch 3.8.2[ ] rabbitmq_web_mqtt 3.8.2[ ] rabbitmq_web_mqtt_examples 3.8.2[ ] rabbitmq_web_stomp 3.8.2[ ] rabbitmq_web_stomp_examples 3.8.2
10. Now take the AMI of the image. This will be our Base Image of RabbitMQ. Let’s make them as a cluster now. For that, .erlang.cookie file must be same for both the servers. So, copy the .erlang.cookie file from your master server to mirror server. (Execute this on master node).
$ sudo scp /var/lib/rabbitmq/.erlang.cookie user@server-ip2:/var/lib/rabbitmq/.erlang.cookieProvide only read permission to .erlang.cookie file for system user rabbitmq.
11. Now join the cluster with master node. Before that, stop RabbitMQ application on slave node. (Execute this on slave node).
$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl reset
$ sudo rabbitmqctl join_cluster rabbit@ip-172-12-12-12 $ sudo rabbitmqctl start_app
$ sudo rabbitmqctl cluster_status12. Now we have installed RabbitMQ servers on both the nodes and created a cluster with 2 nodes.
13. (Optional) Install RabbitMQAdmin to create exchanges, queues, bindings and policies.
Install RabbitMQ Admin
# Get the cli and make it available to use.
$ wget http://localhost:15672/cli/rabbitmqadmin
$ chmod +x rabbitmqadmin
$ mv rabbitmqadmin /usr/local/bin/14. (Optional) Create exchanges, queues, bindings and policies.
$ rabbitmqadmin declare exchange --vhost=/ name=userrequest type=direct$ rabbitmqadmin declare queue --vhost=/ name=ha_monitor_queue durable=true$ rabbitmqadmin --vhost=/ declare binding source=userrequest destination_type=queue destination=ha_monitor_queue routing_key=request$ rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic","ha-promote-on-shutdown":"always"}'
15. Check if they were created properly.
$ sudo rabbitmqctl list_queuesTimeout: 60.0 seconds ...Listing queues for vhost / ...name messagesha_monitor_queue 0$ sudo rabbitmqctl list_policiesListing policies for vhost "/" ...vhost name pattern apply-to definition priority/ ha-all .* all {"ha-mode":"all","ha-promote-on-shutdown":"always","ha-sync-mode":"automatic"} 0
Now let’s configure ELB and ASG for our 2 node RabbitMQ cluster.
Create ELB
- Create classic load balancer on port 15672 and 5672.
2. Assign proper security groups. Here port 15672 and 5672 is opened.
3. We are not using HTTS now, so no security configurations needed.
4. Configure health check for TCP protocol at port 5672.
5. Add our existing RabbitMQ nodes to the load balancer.
You will see the instances now coming up InService once health check is passed.
Let’s create the Launch Configurations.
- Select the RMQ Base Image AMI.
2. Choose the Instance type based on your requirement. In configure details, provide python script to automatically find the nodes in your ELB and add the new machines into the existing RabbitMQ cluster.
3. Go all the way to the review page and create the launch configuration.
Let’s create the Auto Scaling Groups using the launch configuration which we created above.
- Select create auto scaling group from the existing launch configuration and select your launch configuration.
2. Select Appropriate VPC and Subnet. Select receive traffic from one or more load balancers and choose your ELB.
3. Create Auto Scaling policies for adding or removing the nodes automatically when there is a CPU Spike.
4. Optionally configure notifications and tags. Finally review and create the Auto Scaling Groups.
Conclusion
We have created a 2 Node RabbitMQ cluster manually or using ansible playbook and configured it with ELB (AWS Classic Load balancer) and with Auto Scaling Group to automatically scale up or down based on the traffic. It is best practice to spread the 2 RabbitMQ nodes in different availability zone and also have a region failover of the complete cluster with another new RabbitMQ cluster as a secondary for this cluster (if your infrastructure budget allows).
