Elastic Beanstalk Worker Tier Auto Scaling based on SQS message depth

Image via Wikimedia

AWS Elastic Beanstalk Worker tier is an amazing tool, especially for managing scheduled tasks. AWS Elastic Beanstalk manages it using Amazon SQS queue and running a daemon process that reads messages from the SQS and performs the desired task.

Why do we need SQS message depth based auto scaling?

One problem for scheduled task is, if the tasks are long running or if it’s difficult to calculate the processing time required for each task, then we cannot accurately predict and allocate instances to the Beanstalk Application. Also, after deployment, at any time, the messages in SQS may start stacking up if the allocated instances are unable to process all of them in their capacity. To tackle this problem, we need to trigger auto scaling based on the SQS message depth.

Key points to consider

This solution is based on the following key points.

• It will be triggered by the application code. For this, we will use “. ebextensions” to create a configuration i.e. “.ebextensions/01_sqs_based_autoscaling_policy.config” file.
• After we deploy this, the configuration will add a scaling policy to the auto scaling group associated with the EC2 instances of that Beanstalk application.
• For this configuration to take effect, we need to define the maximum and the minimum number of instances of Beanstalk application. So that, after scaling up or down it never goes beyond the maximum or the minimum number of instances respectively.
• If any change is required in the configuration like the evaluation period or threshold, we just need to update the file and deploy. After deployment, the auto scaling group policy will be updated accordingly.

Alarm creation based on ApproximateNumberOfMessagesVisible metric

Firstly, we need to create an Alarm to detect the number of messages being stacked up in the SQS.

For that, we need the SQS queue name first. If we are using the auto generated SQS queue defined by AWS, then we can get it as follows.

{ "Fn::GetAtt": ["AWSEBWorkerQueue", "QueueName"] }

If the SQS queue was manually defined, then we can get the name in the following way,

"Fn::Select":
  - 4
  - "Fn::Split":
    - '/'
    - "Fn::GetOptionSetting":
        Namespace: "aws:elasticbeanstalk:sqsd"
        OptionName: "WorkerQueueURL"

Then, we will define the threshold for scaling up or down. For this, we will use the default “ApproximateNumberOfMessagesVisible” metric provided by AWS. Then, we need to define the evaluation period and threshold. For example, we would create Alarm if the “ApproximateNumberOfMessagesVisible” metric is greater than 20 for 1 datapoint within 5 minutes. So, the complete configuration will be as follows.

Resources:
ApproximateNumberOfMessagesVisibleAutoScalingAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: "Alarm if queue depth grows beyond threshold"
      Namespace: "AWS/SQS"
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        — Name: QueueName
          Value:
            "Fn::Select":
            — 4
            — "Fn::Split":
              — '/'
              — "Fn::GetOptionSetting":
                  Namespace: "aws:elasticbeanstalk:sqsd"
                  OptionName: "WorkerQueueURL"
      Statistic: Average
      Period: 300
      EvaluationPeriods: 1
      Threshold: 20
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        — Ref: ScaleUpPolicy
      OKActions:
        — Ref: ScaleDownPolicy

Alarm Actions

Now, we will create 2 policies namely, “ScaleUpPolicy” and “ScaleDownPolicy” to increase and decrease instance size by 1 respectively. So, when Alarm is raised, “ScaleUpPolicy” will be triggered and when Alarm is in OK state, “ScaleDownPolicy” will be triggered.

Please note that Cloudwatch Alarms associated with auto scaling is different from other Cloudwatch Alarm. A Cloudwatch Alarm keeps triggering Auto Scaling actions when that Alarm is in a specified state, even if there are no state changes and the Alarm remains in that state. So, it will keep on adding instances if the number of SQS messages go beyond our desired threshold up to the maximum number of instances defined. Conversely, it will keep on removing instances if the number of SQS messages go below our desired threshold up to the minimum number of instances defined.

Make Sure That CloudWatch Alarms Trigger Actions

So, now it’s time to define the policies.

ScaleUpPolicy:
  Type: AWS::AutoScaling::ScalingPolicy
  Properties:
    AdjustmentType: ChangeInCapacity
    AutoScalingGroupName: {Ref: AWSEBAutoScalingGroup}
    ScalingAdjustment: 1
 
ScaleDownPolicy:
  Type: AWS::AutoScaling::ScalingPolicy
  Properties:
    AdjustmentType: ChangeInCapacity
    AutoScalingGroupName: {Ref: AWSEBAutoScalingGroup}
    ScalingAdjustment: -1

The complete configuration is given here for your convenience.

If you are facing this kind of problem, hopefully, this will make your life a little bit easier. Thanks for your patience. Feel free to connect.