Advanced Serverless Techniques VI: Building Resilient and Efficient Cloud Architectures With AWS SNS, Lambda, and DynamoDB Streams

When you shouldn’t use DynamoDB Streams to AWS Lambdas Directly (use SNS in the middle)

Certain business cases or requirements may necessitate using an intermediary service like Amazon Simple Notification Service (SNS) between Amazon DynamoDB Streams and AWS Lambda, instead of directly triggering Lambda functions from DynamoDB Streams. Here are three examples where the second approach (DynamoDB Streams to SNS to AWS Lambda) offers advantages over the direct integration:

1. Complex Event Processing with Multiple Consumers

Scenario: An e-commerce platform needs to process inventory updates in real-time. The updates need to trigger multiple independent processes, such as updating the search index, notifying users of restocks, and adjusting related product recommendations.

Why SNS Over Direct Integration:

• Decoupling: SNS allows for better decoupling of the event producers (DynamoDB Streams) from consumers (various Lambda functions). This architecture simplifies managing and updating processing logic for different consumers without impacting the source table or other consumers.
• Fan-out: SNS effortlessly supports the fan-out scenario where a single message needs to be delivered to multiple consumers. While DynamoDB Streams directly triggering a Lambda function is efficient for one-to-one interactions, SNS facilitates one-to-many message dissemination without additional infrastructure or management overhead.

2. Need for Flexible Message Filtering and Routing

Scenario: A financial application processes transactions and needs to route messages based on transaction types, such as deposits, withdrawals, and transfers, to different processing systems for fraud analysis, notifications, and auditing.

Why SNS Over Direct Integration:

• Message Filtering: SNS supports message attributes and filtering policies, allowing messages to be routed to different Lambda functions based on content. This capability is crucial for scenarios requiring intelligent routing decisions without processing each message to determine its destination.
• Adaptability: The use of SNS as an intermediary layer adds flexibility to change routing logic or add new processing functions without modifying the existing DynamoDB or Lambda setup, accommodating evolving business requirements with minimal friction.

3. Enhanced Reliability and Error Handling for Critical Operations

Scenario: A healthcare application collects patient data in DynamoDB. This data triggers notifications for critical lab results that must be processed reliably. It’s essential that no messages are lost, even if the processing function is temporarily unavailable or encounters errors.

Why SNS Over Direct Integration:

• Improved Error Handling: By integrating SNS, you can leverage features like Dead Letter Queues (DLQs) for messages that cannot be processed successfully, ensuring that critical notifications are not lost and can be retried or investigated.
• Resilience: The SNS to Lambda integration allows for more sophisticated error handling and retry policies compared to direct DynamoDB Streams to Lambda integration. SNS can store messages until the Lambda function is available again, enhancing the overall resilience of the system in handling temporary outages or throttling issues.

AWS SNS

Amazon Simple Notification Service (SNS) is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications. It provides a powerful way to simplify and scale communication between components of your application or with end-users, directly through push notifications.

What is AWS SNS?

AWS SNS allows you to send messages or notifications to subscribing endpoints or clients. It supports various types of subscribers, including web servers, email addresses, AWS Lambda functions, SMS text messages, and more. SNS enables you to group multiple recipients using topics. A topic is an access point for allowing recipients to dynamically subscribe for identical copies of the same notification.

Types of SNS

AWS SNS supports two types of messaging:

1. Standard Topics: These provide a high-throughput, at-least-once delivery system, which means messages are delivered at least once but may be delivered more than once in rare circumstances. Standard topics are suitable for applications where the occasional duplicate message is acceptable and where you need high throughput.
2. FIFO (First-In, First-Out) Topics: Introduced to provide ordering and deduplication of messages. FIFO topics ensure that messages are delivered exactly once and in the exact order they are sent. They are ideal for applications where the order and exact delivery counts, such as banking transactions or order processing systems.

Limits

AWS SNS has several limits, some of which are adjustable upon request. Key limits include:

• Message Size: For both Standard and FIFO topics, the maximum message size is 256 KB (262,144 bytes).
• Subscriptions: No limit on the number of subscriptions per topic, but there’s a limit on the number of topics and subscriptions you can create in your AWS account, which can be increased by contacting AWS support.
• Throughput: For Standard topics, there’s no limit on the number of messages per second. For FIFO topics, AWS guarantees 300 transactions per second (TPS) with batching, or 10 TPS without batching.

Costs

AWS SNS pricing is pay-as-you-go and varies based on usage, message format, and destination. Key aspects include:

• Publishing: Charges are based on the number of messages published to SNS topics.
• Delivery: Costs vary by delivery protocol (e.g., SMS, email, HTTP, Lambda). For example, delivering messages to HTTP endpoints is cheaper than sending SMS messages.
• Data Transfer: Data transfer fees apply for messages sent to the internet but not for messages sent within the same AWS region.
• FIFO Topics: May incur additional costs compared to Standard topics due to the ordering and deduplication features.

SNS offers a free tier, which includes 1 million publishes, 100,000 HTTP/S deliveries, 1,000 email deliveries, and other benefits per month. Beyond the free tier, prices are based on the number of requests, the data transfer out, and specific features used like SMS or mobile push notifications. Pricing details can vary by region and over time, so it’s always a good idea to check the AWS website for the most current pricing information.

AWS Lambda + SNS in Serverless

To integrate AWS SNS with AWS Lambda using the Serverless Framework, you will need to define the SNS topic in your serverless.yml file, subscribe a Lambda function to this topic, and then implement the Lambda function to process the messages it receives from the SNS topic. Here's how you can set this up, including an example Lambda function that uses the AWS SDK for JavaScript v3 to log the details of the SNS message it processes.

Define SNS Topic and Lambda Subscription in serverless.yml

Below is an example configuration in serverless.yml that creates an SNS topic and defines a Lambda function which is subscribed to this topic.

service: sns-lambda-example

provider:
  name: aws
  runtime: nodejs14.x
  region: us-east-1

functions:
  snsMessageProcessor:
    handler: handler.processSNSMessage
    events:
      - sns: mySNSTopic

resources:
  Resources:
    mySNSTopic:
      Type: AWS::SNS::Topic
      Properties:
        DisplayName: "My SNS Topic"

In this example, the snsMessageProcessor Lambda function is triggered by messages published to mySNSTopic. The events section under the function definition in the serverless.yml links the Lambda function to the SNS topic.

Implement the Lambda Function

Next, you’ll implement the Lambda function in Node.js. This function will be triggered by an SNS event, and it will use the AWS SDK for JavaScript v3 to log the message details. In this example, the function simply logs the message; in a real-world scenario, you could extend this to process the message as needed.

handler.js:

// Import the required AWS SDK client for Node.js v3
const { SNSClient, PublishCommand } = require("@aws-sdk/client-sns");

// Create an SNS client object
const snsClient = new SNSClient({ region: "us-east-1" });

exports.processSNSMessage = async (event) => {
  console.log("Received SNS event:", JSON.stringify(event, null, 2));

  // Process each SNS message
  event.Records.forEach(record => {
    const { Sns } = record;
    console.log(`SNS Message ID: ${Sns.MessageId}`);
    console.log(`SNS Message: ${Sns.Message}`);

    // Here you could add more logic to process the message
    // For example, publishing a new message, interacting with another AWS service, etc.
  });

  // Example of publishing a new message (optional and context-dependent)
  // Replace 'YOUR_TOPIC_ARN' with the actual ARN of the SNS topic
  const publishParams = {
    Message: 'Processed a message successfully',
    TopicArn: 'YOUR_TOPIC_ARN',
  };

  try {
    const data = await snsClient.send(new PublishCommand(publishParams));
    console.log("Success. Message published. MessageID:", data.MessageId);
  } catch (err) {
    console.error("Error", err.stack);
  }
};

Notes

• This Lambda function logs the details of each SNS message it receives to CloudWatch Logs.
• The optional section included at the end of the function demonstrates how to publish a new message to an SNS topic, showcasing the use of the AWS SDK for JavaScript v3’s PublishCommand. You would need to replace 'YOUR_TOPIC_ARN' with the actual ARN of your SNS topic if you choose to use this part of the code.

Remember to install the AWS SDK v3 for JavaScript in your project by running npm install @aws-sdk/client-sns if you're using this SDK in your function code.

When integrating AWS SNS with AWS Lambda, there are several special considerations to keep in mind, especially regarding message delivery failures and error handling. Understanding these aspects is crucial for designing robust and reliable applications.

Delivery Failures and Error Handling

1. Retries and Back-off: AWS Lambda automatically retries the execution of the function in response to SNS notifications. The retry strategy includes a back-off mechanism where the interval between retries increases gradually. This behavior helps manage temporary issues that might prevent successful execution.
2. Dead Letter Queues (DLQs): SNS supports Dead Letter Queues (DLQs) to help you manage messages that can’t be delivered successfully to subscribing endpoints, including Lambda functions. DLQs are not automatically configured for SNS subscriptions; you must explicitly set them up.

• For Lambda functions triggered by SNS, you can configure a DLQ by attaching an SQS queue or an SNS topic as the DLQ for the SNS subscription. This way, messages that cannot be delivered successfully after all retry attempts are made can be captured in the DLQ for further analysis and processing.
• Messages might end up in a DLQ for various reasons, such as throttling by Lambda (if it exceeds concurrency limits) or execution errors within the Lambda function that are not handled properly.

Configuring DLQ for SNS Subscriptions

To configure a DLQ for an SNS subscription in the Serverless Framework, you would typically use an SQS queue as the DLQ and reference it in your serverless.yml configuration. Here's how you might define an SQS queue for DLQ and attach it to an SNS subscription:

resources:
  Resources:
    MyDLQ:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "MySNSDLQ"
    
    SNSSubscription:
      Type: "AWS::SNS::Subscription"
      Properties:
        Protocol: "lambda"
        Endpoint: { "Fn::GetAtt": ["MyLambdaFunction", "Arn"] }
        TopicArn: { "Ref": "MySNSTopic" }
        RedrivePolicy:
          deadLetterTargetArn: { "Fn::GetAtt": ["MyDLQ", "Arn"] }

This example configures an SQS queue (MyDLQ) to serve as the DLQ for messages that cannot be delivered to the Lambda function (MyLambdaFunction) subscribed to the SNS topic (MySNSTopic).

Special Considerations

• Monitoring and Alerting: Implement monitoring and alerting for both the primary message processing flow and the DLQ to quickly identify and respond to issues.
• Message Processing Idempotency: Ensure that your Lambda function’s processing logic is idempotent, as SNS may deliver messages more than once in rare cases, especially for standard topics.
• Security and Access Control: Properly configure IAM roles and policies for Lambda functions and SNS topics to adhere to the principle of least privilege, ensuring that each component has only the permissions it needs to operate.

By considering these aspects and implementing DLQs where appropriate, you can build more resilient and fault-tolerant applications using AWS SNS and Lambda.

Conclusion

Integrating Amazon SNS between Amazon DynamoDB Streams and AWS Lambda offers unique benefits, especially in complex application architectures. Understanding these advantages, the flexibility SNS provides, how it compares to direct DynamoDB Streams integration, and its differences from using Amazon SQS, is crucial for making informed architectural decisions.

Benefits of Using SNS in the Middle of DynamoDB Streams and Lambda

1. Decoupling of Producers and Consumers: SNS serves as a pub/sub messaging hub, allowing event producers (DynamoDB Streams) to operate independently from event consumers (Lambda functions). This separation simplifies system components’ management, scaling, and updating.
2. Fan-out Scenario Support: SNS natively supports the fan-out pattern, efficiently distributing a single message to multiple subscribers. This feature is particularly useful for broadcasting updates to various services or processing pipelines without introducing additional complexity.
3. Flexible Message Routing and Filtering: With SNS, messages can be filtered and routed to different subscribers based on attributes. This capability allows for sophisticated processing logic where different Lambda functions can be targeted based on the message content, without each consumer having to filter irrelevant messages.
4. Enhanced Durability and Availability: SNS is designed for high availability and durability, ensuring that messages are reliably delivered to subscribers even in the face of component failures or network issues.
5. Improved Error Handling: Leveraging SNS topics with subscriptions to Lambda allows for the integration of Dead Letter Queues (DLQs) more seamlessly, providing a mechanism to capture and reprocess or analyze messages that fail to be processed successfully.

Flexibility Offered by SNS

• Scalability: Easily scale your workload by adding more subscribers to the SNS topic without modifying the existing infrastructure setup.
• Subscription Management: SNS provides an easy-to-use interface for managing subscriptions, including adding or removing subscribers dynamically, which aids in maintaining system adaptability over time.
• Multiple Protocol Support: SNS supports various protocols (e.g., HTTP, email, SMS, Lambda), enabling the same message to be consumed by different types of applications or notification systems.

Main Differences from Direct DynamoDB Streams to Lambda Integration

• Throughput and Performance: Direct integration with DynamoDB Streams may offer lower latency for processing stream records as they are generated. However, this approach lacks the flexibility and features provided by SNS, such as message filtering and fan-out distribution.
• Error Handling: Direct integration has limited error handling capabilities compared to the sophisticated mechanisms available when using SNS, such as message retries with back-off strategies and DLQs for undeliverable messages.

Comparison with SQS

• Delivery Semantics: SQS offers both standard and FIFO (First-In-First-Out) queues, providing at-least-once and exactly-once delivery semantics, respectively. SNS provides at-least-once delivery, similar to SQS standard queues, but lacks the inherent ordering guarantee of SQS FIFO queues.
• Polling vs. Push: SQS is primarily a polling service, where consumers actively poll the queue for messages. In contrast, SNS pushes messages to subscribers, which can reduce latency and resource consumption for event-driven architectures.
• Use Cases: SQS is ideal for decoupled message processing where individual messages are processed independently. SNS shines in scenarios requiring message broadcasting to multiple consumers or where different types of notifications (e.g., email, SMS) might be needed.

Empower Your Tech Journey:

Explore a wealth of knowledge designed to elevate your tech projects and understanding. From safeguarding your applications to mastering serverless architecture, discover articles that resonate with your ambition.

New Projects or Consultancy

For new project collaborations or bespoke consultancy services, reach out directly and let’s transform your ideas into reality. Ready to take your project to the next level?

• Email me at one@upskyrocket.com
• Visit My Partner In Tech for custom solutions

Protecting Routes

• How to Create Protected Routes Using React, Next.js, and AWS Amplify
• How to Protect Routes for Admins in React Next.js Using HOC
• Secure Your Next.js App: Advanced User Management with AWS Cognito Groups

Advanced Serverless Techniques

• Advanced Serverless Techinques I: Do Not Repeat Yourself
• Advanced Serverless Techniques II: Streamlining Data Access with DAL
• Advanced Serverless Techniques III: Simplifying Lambda Functions with Custom DynamoDB Middleware
• Advanced Serverless Techniques IV: AWS Athena for Serverless Data Analysis
• Advanced Serverless Techniques V: DynamoDB Streams vs. SQS/SNS to Lambda

Mastering Serverless Series

• Mastering Serverless (Part I): Enhancing DynamoDB Interactions with Document Client
• Mastering Serverless (Part II): Mastering AWS DynamoDB Batch Write Failures for a Smoother Experience.
• Mastering Serverless (Part III): Enhancing AWS Lambda and DynamoDB Interactions with Dependency Injection
• Mastering Serverless IV: Unit Testing DynamoDB Dependency Injection With Jest
• Mastering Serverless (Part V): Advanced Logging Techniques for AWS Lambda