RabbitMQ Cluster using Terraform and Ansible in AWS
In this blog, we will configure rabbitmq cluster using Terraform and ansible and also configure a load balancer in front of the cluster. In AWS, we’ll be working on Frankfurt(eu-central-1) Region.
Decoupling of software components is one of the most important parts of software design. One way of achieving this is using messaging systems, which provide an asynchronous way of communication between components. RabbitMQ is a message broker that implements Advanced Message Queuing Protocol (AMQP). It provides client libraries for major programming languages.
You can find the code in this repository.
Prerequisites:
1. AWS Account
2. IAM User with Console and Programmatic Access
3. Terraform v0.12.10
4. Ansible Latest Version
The directory structure should be look like this
In variables.tf, change your AWS ACCESS KEY and SECRET ACCESS KEY and also the PEM file location of yours. You can update your Region and also AMI ID associated to the region. Now, run terraform:
node a
$ terraform init
$ terraform apply -auto-approve
After applying the script you should be able to see the output.
You should get the ips of your newly created instances. Now, copy and paste these ip addresses in Ansible’s inventory hosts file. Also change the ansible_private_key_file value. RabbitMQ admin password is defined in install_rabbitmq/defaults/main.yml file. You can update if you want. Now, run ansible playbook.yml
$ ansible-playbook -i hosts playbook.yml
After applying the script you should be able to see the output.
Your RabbitMQ Cluster is up and running.
Now configure a Classic Load Balancer under your VPC and specify the security group, add the instances and in listeners add TCP port
80 => 5672
15672 => 15672
In Health Check, Ping Target should be: HTTP:15672/
Copy and paste the load balancer url:15672 in your browser, you can see the management portal. login using admin,qweqwe
In this RabbitMQ cluster I’ve configured mirroring. By default, contents of a queue within a RabbitMQ cluster are located on a single node (the node on which the queue was declared). This is in contrast to exchanges and bindings, which can always be considered to be on all nodes. Queues can optionally be made mirrored across multiple nodes.
Each mirrored queue consists of one master and one or more mirrors. The master is hosted on one node commonly referred as the master node. Each queue has its own master node. All operations for a given queue are first applied on the queue’s master node and then propagated to mirrors.
Also you can enable sharding queues by enabling rabbitmq_sharding plugin. One of the main properties of this plugin is that when a new node is added to the RabbitMQ cluster, then the plugin will automatically create more shards on the new node. Say there is a shard with 4 queues on node a and node b just joined the cluster. The plugin will automatically create 4 queues on node b and "join" them to the shard partition. Already delivered messages will not be rebalanced but newly arriving messages will be partitioned to the new queues.
