Scaling EC2 Instances Subject to Job Queue Size
How to scale the EC2 Instances based on the background job queues’ length
EC2 Autoscaling Groups are widely used by the developers to scale their application using various parameters like CPU, Memory, and many other CloudWatch metrics. Many times it is required to launch/remove instances based on the length of a job queue.
We were using Sidekiq for our background jobs. Redis was used as job queue(s). Ours was a background job-intensive system, so we wanted to scale based on various queues primarily to save cost.
The solution is quite straightforward. It took us some effort to make it production-ready. The solution looks like this in simple steps.
Prerequisites
You should be familiar with AWS CloudWatch, CloudWatch Metrics, and Auto-scaling groups.
Step 1 — Push data to CloudWatch Metrics
The first step is to push data to a CloudWatch Metric periodically. This data need to be sent to the metric quite often. We do it every 2–3 minutes. The pseudo-code looks like this:
client = Aws::CloudWatch::Client.newmetric_params = {namespace: "<<aws_metric_namespace_prefix>>/Backgroud", metric_data: []}metric_params[:metric_data] = {metric_name: <<queue_name>>, value: <<queue_size_in_string>>, timestamp: Time.now}client.put_metric_data(metric_params)
The Metric on the AWS console looks like this:
Step 2 — Create Alarms
Now, set up two CloudWatch alarms, to scale up and to scale down. Here is an example of an alarm configured to scale up the system.
This alarm will be triggered once the size of the default queue goes beyond 50 for 5 minutes. These parameters are configurable. Similarly create an alarm to scale down the system, say when the queue size is zero.
Step 3 — Autoscale based on the alarms
Finally, the Auto-scaling Scaling policies need to be defined as shown in this screenshot. Two scaling policies are required, one to increase the group size and another to decrease group size. Choose the Alarms we created in the previous step to create the policies.
Just three simple steps and you have an Autoscaling group configured based on a background job queue. This helped us save a lot of money and time.
Cheers.
