How to create a scheduled AWS CloudWatch event that sends a message to AWS Simple Queue Service (SQS)
Recently our team had to deal with a situation when there were three servers (EC2 instances) in AWS deployment stack that shared a common database and there was a scheduled job that was launched once in 2 hours. The interesting part was that only one of these servers needed to perform this job, so the other instances should not be aware of it at all and instead could be busy with processing the incoming user requests.
In order to achieve this goal we had to abandon the task schedulers present in the code of our web services (we used something similar to Quartz.NET at that time), as in that case each of instances receives its own event when the scheduled time comes.

Instead we had to rely on common AWS infrastructure that was expressed in our Cloud Formation template and served all of our servers at the same time. It was decided to create a scheduler that sends a message (that tells a server to start processing the scheduled job) to Amazon Simple Queue Services (SQS) that all EC2 instances are subscribed to. Unlike Amazon Simple Notification Service (SNS) the main principle of Amazon queue is that the message is processed only by the first receiver. In our case all instances compete for receiving the message and only the winner (an instance that receives this event first) is performing this scheduled job. The losers (other servers) can return to their common routines.

It’s quite easy to achieve this requirement by utilizing Amazon scheduled CloudWatch events that trigger some action once in a set time. The list of possible targets (handlers) for such events can be found here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html
Recently Amazon has added an SQS as a target for CloudWatch events (https://aws.amazon.com/en/about-aws/whats-new/2016/03/cloudwatch-events-now-supports-amazon-sqs-queue-targets/), so we were able to utilize this functionality on the fly (and without any additional tweaks).
The scheduled CloudWatch event can be created manually by means of AWS Console (https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html), though we had to utilize CloudFormation templates for this matter, in order to automate the deployment process.
Here is the example of such CloudFormation template:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "stack 1",
"Parameters": {},
"Resources": {
"MyPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyDocument": {
"Statement": [{
"Action": "sqs:*",
"Effect": "Allow",
"Resource": {
"Fn::GetAtt": ["MyQueue",
"Arn"]
}
}],
"Version": "2012-10-17"
},
"PolicyName": "MyPolicyName",
"Roles": [{
"Ref": "MyRole"
}]
}
},
"MyRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": ["events.amazonaws.com",
"sqs.amazonaws.com"]
}
}],
"Version": "2012-10-17"
}
}
},
"MyQueue": {
"Type": "AWS::SQS::Queue",
"Properties": {
"QueueName": "MyQueue"
}
},
"MyRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Description": "A rule to schedule data update",
"Name": "MyRule",
"ScheduleExpression": "rate(1 minute)",
"State": "ENABLED",
"RoleArn": {
"Fn::GetAtt": ["MyRole",
"Arn"]
},
"Targets": [{
"Arn": {
"Fn::GetAtt": ["MyQueue",
"Arn"]
},
"Id": "MyRule1",
"Input": "{\"property\":\"value\"}"
}]
}
},
"MyQueuePolicy": {
"DependsOn": ["MyQueue", "MyRule"],
"Type": "AWS::SQS::QueuePolicy",
"Properties": {
"PolicyDocument": {
"Version": "2012-10-17",
"Id": "MyQueuePolicy",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": ["events.amazonaws.com",
"sqs.amazonaws.com"]
},
"Action": "sqs:SendMessage",
"Resource": {
"Fn::GetAtt": ["MyQueue",
"Arn"]
}
}]
},
"Queues": [{
"Ref": "MyQueue"
}]
}
}
},
"Outputs": {
}
}The resources section consists of the follwoing parts:
- MyPolicy — an AWS IAM policy describing what the resources that refer to this policy are allowed to do. In our case all actions related to SQS are alowed (described via
sqs:*syntax) - MyRole — applies MyPolicy to SQS and CloudWatch event services
- MyQueue — describes the SQS queue resource (remember that the queue name should be unique throughot one AWS account!)
- MyRule — describes the AWS CloudWatch event. It is triggered each 1 minute in the example; two syntaxes (cron and rate) can be utilized in order to set the event schedule. The Target of this event points to our SQS, the Input property defines the JSON message that will be sent to queue
- MyQueuePolicy — the policy used to allow sending SQS messages (described via
sqs:SendMessagesyntax)