AWS Lambda with SAM Template to subscribe an SQS to an SNS Topic.

This block diagram describes an SNS Topic that has an SQS Subscription and a Lambda that fires off when a message is received by the SQS Queue. If you are like me, you created a simple SAM Template that defined the Lambda AWS::Serverless::Function definition that has an SQS Queue as its event source. While that is most of the solution you would find that messages never flow into your SQS Queue. You refresh the console and there are always zero messages.

Basic Lambda SQS Event handler with a SQS Queue buffering SNS Messages
But wait.. I can fire a Lambda straight off of the SNS Messages. True, you can, but in some cases your Lambda may be doing operations against an external API that could be down or have issues. In that case you may want to allow the SQS Message to be automatically retried. More on that in another post.

You have to add an AWS::SQS::QueuePolicy to the SQS Queue to allow the SNS Topic subscription to flow messages into your SQS Queue. I found that the documentation surrounding this process is a bit vague. So here is a full working example.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
MyLambdaFunction:
Type: AWS::Serverless::Function
Properties:
Handler: my-lambda.handler
Runtime: nodejs8.10
Role: arn:aws:iam::[your-account-number]:role/acct-managed/[an-existing-lambda-role]
CodeUri: ./
Environment:
Variables:
MY_SNS_TOPIC_ARN: !Ref myTopic
MY_SQS_QUEUE_URL: !Ref myQueue
Events:
SqsJobQueue:
Type: SQS
Properties:
Queue: !GetAtt paymentPricingJobQueue.Arn
BatchSize: 1
# sns topics that the lambda depends on
myTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: "my-topic"
# sqs queues that the lambda uses
myQueue:
Type: AWS::SQS::Queue
myQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref myQueue
PolicyDocument:
Statement:
Effect: Allow
Principal: "*"
Action: "sqs:*"
Resource: "*"
myQueueToSnsSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: !GetAtt myQueue.Arn
Protocol: sqs
RawMessageDelivery: true
TopicArn: !Ref myTopic

The SAM template above builds out all of the infrastructure needed for the SNS Topic, the SQS Queue, the Subscription of the two, The security policy for the Queue to allow the SNS Topic to send messages to the Queue and the Lambda that is invoked when a message arrives in the queue. Whew!


The code for this is available here: https://github.com/dsandor/example-sns-sqs-lambda

To use this code edit the template.yaml file and add an actual Lambda execution role at line #9. Next, you can package and deploy this code to your AWS (be sure your AWS Environment Variables are all set up) by using this command:

$ yarn package-deploy

The output of that command should look something like the following:

yarn run v1.12.3
$ yarn package ; yarn deploy
$ sam package --template-file template.yaml --s3-bucket dsandor-blog-examples --output-template-file packaged.yaml
Uploading to 96569489267d4394984ad09ce2fbe6e0  1916 / 1916.0  (100.00%)
Successfully packaged artifacts and wrote output template to file packaged.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /Users/dsandor/Projects/blog-posts/sns-sqs-lambda/packaged.yaml --stack-name <YOUR STACK NAME>
$ sam deploy --template-file ./packaged.yaml --stack-name my-sns-sqs-lambda-example --capabilities CAPABILITY_IAM
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - my-sns-sqs-lambda-example

When that command completes you can view the CloudFormation console in AWS and see all the changes that have been applied.