AWS Lambda Destination to SQS for Asynchronous Invocations — DLQ Example
In this article, we are going to learn AWS Lambda Asynchronous Invocations and Lambda Destinations.
So far we have seen Add Trigger to Lambda function, but also we will see right hand side of Lambda function which is Lambda Destinations. By the end of the article, we will do Hands-on Lab : AWS Lambda Destination to SQS — DLQ Case.
I have just published a new course — AWS Lambda & Serverless — Developer Guide with Hands-on Labs.
AWS Lambda Destinations for Async Invocations
Lambda can define Destination when invoke async way. By this feature, Lambda simplifying event-driven applications and reducing code complexity. We can set destination onSuccess or onFail events after invocation done.
We can also configure Lambda to send an invocation record to another service. Lambda supports the following destinations for asynchronous invocation.
- Amazon SQS — A standard SQS queue.
- Amazon SNS — An SNS topic.
- AWS Lambda — A Lambda function.
- Amazon EventBridge — An EventBridge event bus.
The invocation record contains details about the request and response in JSON format. We can configure separate destinations for events that are processed successfully, and events that fail all processing attempts. Alternatively, we can configure an Amazon SQS queue or Amazon SNS topic as a dead-letter queue for discarded events. For dead-letter queues, Lambda only sends the content of the event, without details about the response.
Configuring destinations for asynchronous invocation
In order to send records of asynchronous invocations to another service, add a destination to your function. We can configure separate destinations for events that fail processing and events that are successfully processed.
In error handling settings, you can configure destinations on a function.
You can see example on image that shows a function that is processing asynchronous invocations. When the function returns a success response or exits without throwing an error, Lambda sends a record of the invocation to an EventBridge event bus. When an event fails all processing attempts, Lambda sends an invocation record to an Amazon SQS queue.
The idea of Destinations is that route asynchronous function results as an execution record to a destination resource. An execution record contains details about the request and response in JSON format. For each execution status such as Success or Failure you can choose one of four destinations: another Lambda function, SNS, SQS, or EventBridge. Lambda can also be configured to route different execution results to different destinations.
Required Permissions when Destination to Other Services
In order to send events to a destination, lambda function needs additional permissions. We should add a policy with the required permissions to your function’s execution role. Each destination service requires a different permission, as follows:
- Amazon SQS — sqs:SendMessage
- Amazon SNS — sns:Publish
- Lambda — InvokeFunction
- EventBridge — events:PutEvents
Hands-on Lab : AWS Lambda Destination to SQS — DLQ Case
We are going to do Hands-on Lab : AWS Lambda Destination to SQS — DLQ Case. In this hands-on lab, we will configure AWS Lambda onFailure event and redirect event.json to Amazon SQS as a Dead letter Queue (DLQ).
Here you can find steps that we are going to follow:
- Throw an error in lambda invocation
- Retry async invocation
- Send Destination to add SQS queue record
Create the Lambda function
Open the Functions page of the Lambda console. Choose Create function. On the Create function page, choose — Create from Scratch
- Name — lambda-destination
- Permissions: Create a new role from AWS policy templates. Role Name — swnNewRole
Create Function. Ready to develop our codes on Lambda Function. Add throw exception into function code:
exports.handler = async (event) => {
throw new Error('lambda error');
const response = {
statusCode: 200,
body: JSON.stringify(‘Hello from Lambda!’),
};
return response;
};Test with AWS Management Console and see exception throwed. When we invoke from console, this will invoke request-response that means synchronous invocation. But this section, we will focus on asynchronous invocation.
Lambda Asynchronous Configurations
Goto Lambda Configuration — Asynchronous invocation. See we have
Retry attempts and The maximum number of times to retry when the function returns an error. See Dead-letter queue, we can also set DLQ here but I will follow the destination path.
So far we have seen Add Trigger to Lambda function now we will also see right hand side of lambda function which is Lambda Destinations. You can see these Trigger and Destination when create a lambda function.
- Click — Add destination
- Source — Asynchronous invocation
- Condition — On Failure
- Destination type
- SNS — SQS
Create SQS Queue to catch lambda errors in the queue
Goto SQS and Create default SQS — lambdaQueue. This queue will use in lambda destinations as a DLQ.
If you back to lambda function, Add destinations to our function in the Lambda console’s function visualization.
Add destinations to our function
To configure a destination for asynchronous invocation records, Open the Functions page of the Lambda console. Choose a function. Under Function overview, choose Add destination.
- For Source, choose Asynchronous invocation.
- Condition — On Failure
- Destination type
- SQS — that we create above.
- Save Destination.
When an invocation matches the condition, Lambda sends a JSON document with details about the invocation to the destination.
Don’t forget to update our execution-role to have SQS:read permission
Test Lambda Function
Now its time to test this Destination path. But we should invoke lambda asynchronous. Use AWS CLI invoke command :
aws lambda invoke \
— function-name my-function \
— invocation-type Event \
— cli-binary-format raw-in-base64-out \
— payload ‘{ “key”: “value” }’ response.jsonResponse:
{
"StatusCode": 202
}
StatusCode-202 means the lambda async execution happens successfully. It doesnt return any response. We should check status from console logs.
AWS Lambda CloudWatch Logs
See that there are 3 invocation logs. Because we invoke async way and it has 2 retry configuration. You can check from Lambda Configurations -Async Configuration and see the Retry count. Thats why it tries 2 times to execute lambda but still get exception so it redirect to write to SQS queue this on-failure case.
Goto SQS — check send-receive message — get message — see exception comes here.
As you can see that we have successfully developed Hands-on Lab : AWS Lambda Destination to SQS — DLQ Case. To see full developments of this hands-on lab, you can check below course on Udemy.
Step by Step Design AWS Architectures w/ Course
I have just published a new course — AWS Lambda & Serverless — Developer Guide with Hands-on Labs.
In this course, we will learn almost all the AWS Serverless Services with all aspects. We are going to build serverless applications with using AWS Lambda, Amazon API Gateway, Amazon DynamoDB, Amazon Cognito, Amazon S3, Amazon SNS, Amazon SQS, Amazon EventBridge, AWS Step Functions, DynamoDB and Kinesis Streams. This course will be 100% hands-on, and you will be developing a real-world application with hands-on labs together and step by step.
Source Code
Get the Source Code from Serverless Microservices GitHub — Clone or fork this repository, if you like don’t forget the star. If you find or ask anything you can directly open issue on repository.
