Amazon SQS Queue Polling From AWS Lambda with Event Source Mapping Invocations

Published in

AWS Lambda & Serverless — Developer Guide with Hands-on Labs

7 min readOct 6, 2022

In this article, we are going to learn Amazon SQS Queue Polling From AWS Lambda with Event Source Mapping Invocations.

We have seen Add SQS Destination to Lambda function, but this time we will see how Amazon SQS queue polls from AWS Lambda functions and process event. By the end of the article, we will do Hands-on Lab : Amazon SQS Queue Polling From AWS Lambda.

I have just published a new course — AWS Lambda & Serverless — Developer Guide with Hands-on Labs.

AWS Lambda Event Source Mapping (Polling) Invocations

But before that I would like to remember AWS Lambda Invocation Types and Event Source Mapping (Polling) Invocations.

AWS Lambda integrates with other AWS services to invoke functions or take other actions. These are some common use cases:

Invoke a function in response to resource lifecycle events, such as with Amazon S3. You can see Using AWS Lambda with Amazon S3.
Respond to incoming HTTP requests. You can see Tutorial: Using Lambda with API Gateway.
Consume events from a queue. For more information, We will see using Lambda with Amazon SQS in this article.
Run a function on a schedule. We can see Using AWS Lambda with Amazon EventBridge (CloudWatch Events).

Depending on which service you’re using with AWS Lambda, the invocation generally works in one of two ways. An event drives the invocation or Lambda polls a queue or data stream and invokes the function in response to activity in the queue or data stream.

So far we have done with;

Now we will focus on Event Source Mapping invocation.

Lambda polling

AWS has several services that generate a queue or data stream, we set up an event source mapping in Lambda to have Lambda poll the queue or a data stream. When we implement a Lambda polling architecture, we have to grant Lambda permission to access the other service in the function’s execution role. Lambda reads data from the other service, creates an event, and invokes your function.

Basically, an event source mapping is a Lambda resource that reads from an event source and invokes a Lambda function. We can use event source mappings to process items from a stream or queue in services that don’t invoke Lambda functions directly. Lambda provides event source mappings for the following services.

See list of services that Lambda reads events from

Amazon DynamoDB
Amazon Kinesis
Amazon MQ
Amazon Managed Streaming for Apache Kafka (Amazon MSK)
Amazon Simple Queue Service (Amazon SQS)

Using Lambda with Amazon SQS

We can use a Lambda function to process messages in an Amazon Simple Queue Service (Amazon SQS) queue.

Basically Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages. Lambda reads messages in batches and invokes your function once for each batch. When your function successfully processes a batch, Lambda deletes its messages from the queue.

See example Amazon SQS message event json file polls from Lambda:

{
 “Records”: [
 {
 “messageId”: “059f36b4–87a3–44ab-83d2–661975830a7d”,
 “receiptHandle”: “AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a…”,
 “body”: “Test message.”,
 “attributes”: {
 “ApproximateReceiveCount”: “1”,
 “SentTimestamp”: “1545082649183”,
 “SenderId”: “AIDAIENQZJOLO23YVJ4VO”,
 “ApproximateFirstReceiveTimestamp”: “1545082649185”
 },
 “messageAttributes”: {},
 “md5OfBody”: “e4e68fb7bd0e697a0ae8f1bb342846b3”,
 “eventSource”: “aws:sqs”,
 “eventSourceARN”: “arn:aws:sqs:us-east-2:123456789012:my-queue”,
 “awsRegion”: “us-east-2”
 },
 {
 “messageId”: “2e1424d4-f796–459a-8184–9c92662be6da”,
 “receiptHandle”: “AQEBzWwaftRI0KuVm4tP+/7q1rGgNqicHq…”,
 “body”: “Test message.”,
 “attributes”: {
 “ApproximateReceiveCount”: “1”,
 “SentTimestamp”: “1545082650636”,
 “SenderId”: “AIDAIENQZJOLO23YVJ4VO”,
 “ApproximateFirstReceiveTimestamp”: “1545082650649”
 },
 “messageAttributes”: {},
 “md5OfBody”: “e4e68fb7bd0e697a0ae8f1bb342846b3”,
 “eventSource”: “aws:sqs”,
 “eventSourceARN”: “arn:aws:sqs:us-east-2:123456789012:my-queue”,
 “awsRegion”: “us-east-2”
 }
 ]
}

The payload includes Records array, and every record item represent SQS queue item. As you can see that we have understood Lambda invocation types and understood event-source mapping invocation. So we will start our hands on lab with Lambda and SQS queues.

Hands-on Lab: Amazon SQS Queue Polling From AWS Lambda

We are going to start our lab which is Hands-on Lab: Amazon SQS Queue Polling From AWS Lambda. So we will Using Amazon SQS Queue Polling From AWS Lambda. Here you can find the architecture that we will implement in this section:

There are several steps that we will follow:

Create an Amazon SQS queue
Create a Lambda function
Create a Lambda trigger to Amazon SQS queue
Develop Lambda function for incoming event from Amazon SQS queue
Send Message from Amazon SQS

As you know that, when we implement any architecture on AWS, we have 2 main steps;

Create infrastructure on AWS Cloud
Develop Lambda code for interacting SQS

So we will start with the first step, Lets Create this architecture infrastructure on AWS Cloud. Goto AWS Management Console and follow the steps below;

Create Amazon SQS queue — defaults — myQueue — Create
Create AWS Lambda function — defaults — myFunc — Create
Add Trigger: To perform event-source mapping invocations, we are going to add triger to lambda fucntion which trigger is SQS.
Add SQS — default

In order to read from the SQS trigger, our Lambda execution role must have proper permissions. This will automatically add policies into lambda execution role when we add SQS trigger into Lambda function. If it is not added, you can do it manually,

Lambda — Permission — Role — IAM Page — Add Permission — Attach Policies — Attach AWSLambdaSQSQueueExecutionRole policy

So as you can see that we have configured Lambda and SQS. Now if we send queue message to SQS, this will poll from Lambda function with event-source mapping invocation and execute with incoming message.

Develop Lambda Function for Polling Queue Message with event source mapping invocation

We are going to Develop Lambda Function for Polling Queue Message with event source mapping invocation. Lets Develop initial Lambda Function:

exports.handler = async (event) => {
 console.log(“event:”, JSON.stringify(event, undefined, 2));
 console.log(‘function executed !’);
 
 event.Records.forEach(record => {
 const { body } = record;
 console.log(body);
 });
 return {};
};

Basically with this lambda function code, we will iterate incoming batch queue records from Amazon SQS and perform log operation. This time we don’t have any business logic, but you can apply any business logic with following these code block.

Verify and Test Queue Message Polling From AWS Lambda

We are going to start our lab which is Verify and Test Queue Message Polling From AWS Lambda. Now I am going to test our architecture with sending message to SQS.

goto SQS — send message — test

See lambda executions, See and save incomming event from SQS:

2022–06–03T13:09:36.473Z 82e19156-eafb-5e3f-81e5-ca42e0bf1d28 INFO event: {
 “Records”: [
 {
 “messageId”: “035562fe-c04a-4040–82b1–26dcc74c8978”,
 “receiptHandle”: “AQcxxxxUem4fxQHdQISgQ”,
 “body”: “testMessage”,
 “attributes”: {
 “ApproximateReceiveCount”: “1”,
 “SentTimestamp”: “1654261775846”,
 “SenderId”: “AIDAUPS5WIU7ALSBTGGIJ”,
 “ApproximateFirstReceiveTimestamp”: “1654261775850”
 },
 “messageAttributes”: {},
 “md5OfBody”: “b11719be470ca68bc40a36270e1ebdca”,
 “eventSource”: “aws:sqs”,
 “eventSourceARN”: “arn:aws:sqs:us-east-2:xxx:myQueue”,
 “awsRegion”: “us-east-2”
 }
 ]
}

We have successfully get testMessage into our lambda function code.

AWS Lambda Event Filtering from SQS data

For Amazon SQS event sources, we can use event filtering to control which events Lambda sends to your function for processing. We can define up to five different filters for a single event source. If an event satisfies any one of these five filters, Lambda sends the event to our function. Otherwise, Lambda discards the event.

As you can see that we have successfully developed Hands-on Lab : Amazon SQS Queue Polling From AWS Lambda. To see full developments of this hands-on lab, you can check below course on Udemy.

Step by Step Design AWS Architectures w/ Course

I have just published a new course — AWS Lambda & Serverless — Developer Guide with Hands-on Labs.

In this course, we will learn almost all the AWS Serverless Services with all aspects. We are going to build serverless applications with using AWS Lambda, Amazon API Gateway, Amazon DynamoDB, Amazon Cognito, Amazon S3, Amazon SNS, Amazon SQS, Amazon EventBridge, AWS Step Functions, DynamoDB and Kinesis Streams. This course will be 100% hands-on, and you will be developing a real-world application with hands-on labs together and step by step.

Source Code

Get the Source Code from Serverless Microservices GitHub — Clone or fork this repository, if you like don’t forget the star. If you find or ask anything you can directly open issue on repository.

References:

Using AWS Lambda with other services

AWS Lambda integrates with other AWS services to invoke functions or take other actions. These are some common use…

docs.aws.amazon.com