Using the Serverless framework and AWS services to build robust and scalable event-based systems

Client requirements for capturing user/system interactions for processing efficacy and efficiency are paramount in today's systems.

Being able to capture and retain client requests (sometimes in the order in which they are received) whilst a provisioning service may be unavailable, has become a key requirement. Being able to build robust and scalable systems is expected.

With the predominance of Cloud providers providing key infrastructure services, Cloud Engineers have at their disposal a raft of cloud native tools to add to their solutioning kit.

In this article, we will cover the following AWS services that can be combined to provide a robust and scalable solution to meet some of these requirements.

• AWS SNS (Simple Notification Service) combined with AWS SQS (Simple Queue Service) will be a key enabler. Messaging and the ability to retain messages is a built-in feature.
• Whilst the SNS and SQS can act as carriers and routers/filterers of the messages to their intended targets, actioning on the message contents will be handled by AWS Lamdba functions.
• For the orchestration and deployment of these services we will use the Serverless Framework.

So why these services ?

In the opening statements of this article, I mentioned an architecture that would enable the capture and retention of client interactions and provide a process which would be enable efficacy and efficiency of process.

The below outlines what feature sets (not all) we could rely upon to provide these.

AWS Simple Notification Service

Features

• FIFO SNS Topic — Setting the First In First Out property of the Topic ensures messages are received and processed in the order in which they received.
• Message durability — Messages are stored redundantly across multiple availability zones and provide dead letter queues to handle exceptions occurring during any processing. For any type of error, SNS can send messages to SQS dead letter queues so that data lost is prevented.
• Message filtering — Simplifies the architecture by offloading the message routing logic from publisher systems and message filtering logic from subscriber systems.
• Message deduplication — Provides exactly once message delivery for when message filtering is not enabled in the Topic, or at most once, for when filtering is enabled.

AWS Simple Queue Service

Features

• FIFO SQS Queue — A queue can subscribe to a Topic. Supports FIFO at the queue level as well. Exactly once processing.
• Dead Letter Queue (DLQ) — A queue which can be sent messages which did not arrive at first to their intended target.
• Message retention — Message Retention Can be retained up to 14 days, 4 being the default. In terms of durability, it shares the same characteristics as SNS.

Our Booking Scenario

Utilising the above components, we have the example of the BookingTopic which is a FIFO SNS based Topic where clients can publish their messages to. Subscribing to this BookingTopic is the BookingQueue. Associated with this BookingQueue is the BookingDLQ where messages not reaching the intended target will be go for exception processing.

The intended target is a Lambda which represents the intended business logic to be executed on receipt of the message.

First in first out SNS

Defining a SNS FIFO Topic where clients would publish messages to could be defined as follows:

resources:
  Resources:
    BookingTopic:
      Type: "AWS::SNS::Topic"
      Properties:
        TopicName: "BookingTopic.fifo"
        FifoTopic: true
        ContentBasedDeduplication: true

In the above we define our BookingTopic.fifo which is identified by its type AWS::SNS::Topic with the property of FifoTopic set to true. In addition we have set ContentBasedDeduplication to true as well to provide for our single message send use case.

First in first out SQS

As we mentioned, this architecture calls for the use of a FIFO SQS queue. The below fragment would be defined under the resources section of our serverless.yml file.

BookingQueue:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "BookingQueue.fifo"
        FifoQueue: true
        ContentBasedDeduplication: true
        RedrivePolicy:
          deadLetterTargetArn:
            Fn::GetAtt:
              - BookingQueueDLQ
              - Arn
          maxReceiveCount: 3

In the above we define our BookingQueue.fifo setting its FifoQueue to be true and ContentBasedDeduplication also to be true. When both Topic and Queue are defined in this way, it ensures that we can meet the at most single message use case.

Associating a dead letter queue BookingQueueDLQ to this queue is achieved via the RedrivePolicy where the arn of the BookingDLQ is provided.

The BookingQueueDLQ is defined as follows:

BookingQueueDLQ:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "BookingQueueDLQ.fifo"
        FifoQueue: true
        ContentBasedDeduplication: true

Ensuring that our queue is able to be notified of messages being published to the topic, we define the following fragment

BookingQueueSubscription:
      Type: 'AWS::SNS::Subscription'
      Properties:
        TopicArn: 
          Fn::Join:
              - ':'
              - - 'arn:aws:sns'
                - Ref: 'AWS::Region'
                - Ref: 'AWS::AccountId'
                - 'BookingTopic.fifo'
        Endpoint: 
          Fn::GetAtt:
          - BookingQueue
          - Arn
        Protocol: sqs
        RawMessageDelivery: 'true'

RawMessageDelivery setting this to true avoids having Amazon SQS and HTTP/S endpoints process the JSON formatting of messages.

Ensuring that we have the correct permissions which allows for the BookingTopic to send messages to the BookingQueue, we need the following policy:

snsToBookingQueueSQSPolicy:
      Type: AWS::SQS::QueuePolicy
      Properties:
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Sid: "allow-sns-messages"
              Effect: "Allow"
              Principal:
                Service:
                  - "sns.amazonaws.com"
              Resource: 
                Fn::GetAtt:
                  - BookingQueue
                  - Arn
              Action: "SQS:SendMessage"
              Condition:
                ArnEquals:
                  "aws:SourceArn": 
                    Ref: BookingTopic
        Queues:
          - Ref: BookingQueue

In regards to other permissions, defined at the global level of the serverless.yaml file, where all defined lambdas would inherit, is as follows:

iamRoleStatements:
    - Effect: "Allow"
      Resource: "*"
      Action:
        - "sns:*"

In addition to this role statement, it’s also important to note that by using the Serverless framework, it will automatically create additional policy statements to permit access to log streams, log events, receive and delete messages.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogStream"
            ],
            "Resource": [
                "arn:aws:logs:ap-southeast-2:xxxx:log-group:/aws/lambda/benchproject-dev-register:*",
                "arn:aws:logs:ap-southeast-2:xxxx:log-group:/aws/lambda/benchproject-dev-handleDLQ:*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:ap-southeast-2:xxxx:log-group:/aws/lambda/benchproject-dev-register:*:*",
                "arn:aws:logs:ap-southeast-2:xxxx:log-group:/aws/lambda/benchproject-dev-handleDLQ:*:*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "sns:*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes"
            ],
            "Resource": [
                "arn:aws:sqs:ap-southeast-2:xxxx:BookingQueue.fifo"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes"
            ],
            "Resource": [
                "arn:aws:sqs:ap-southeast-2:xxxx:BookingQueueDLQ.fifo"
            ],
            "Effect": "Allow"
        }
    ]
}

Lambda Configuration

On receiving notification of a message in the Topic, the subscribing Queue will trigger the register Lambda. In this example, the lambda will be triggered on receiving an event from the BookingQueue. The following extract configures this :

functions:
  register:
    handler: bin/register
    events:
      - sqs:
          arn: 
            Fn::GetAtt:
              - BookingQueue
              - Arn
          batchSize: 1

sqs The event source is noted as reading from the BookingQueue

batchSize indicates the number of records to send to the function in each batch. For a FIFO queue the maximum is 10.

Finally, we define a dlq lambda handler to deal with messages which end up in the BookingDLQ as follows:

handleDLQ:
    handler: bin/dlq
    description: handle DLQ
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - BookingQueueDLQ
              - Arn
          batchSize: 1

The complete extract of this serverless.yaml file can be found in the below Appendix.

In the next series of articles we’ll continue the code and show some example lambda handler code.

Appendix — serverless.yaml

service: benchproject

provider:
  name: aws
  runtime: go1.x
  versionFunctions: false
  vpc:
    securityGroupIds:
      - sg-xxx
    subnetIds:
      - subnet-xxx
      - subnet-yyy
      - subnet-zzz
  iamRoleStatements:
    - Effect: "Allow"
      Resource: "*"
      Action:
        - "sns:*"

package:
 exclude:
   - ./**
 include:
   - ./bin/**

functions:
  register:
    handler: bin/register
    environment:
      db_user: ${env:db_user}
      db_pass: ${env:db_pass}
      db_host: ${env:db_host}
    description: inserts into database
    reservedConcurrency: 3
    events:
      - sqs:
          arn: 
            Fn::GetAtt:
              - BookingQueue
              - Arn
          batchSize: 1
  handleDLQ:
    handler: bin/dlq
    environment:
      db_user: ${env:db_user}
      db_pass: ${env:db_pass}
      db_host: ${env:db_host}
    description: handle DLQ
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - BookingQueueDLQ
              - Arn
          batchSize: 1
resources:
  Resources:
    BookingTopic:
      Type: "AWS::SNS::Topic"
      Properties:
        TopicName: "BookingTopic.fifo"
        FifoTopic: true
        ContentBasedDeduplication: true

    BookingQueueDLQ:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "BookingQueueDLQ.fifo"
        FifoQueue: true
        ContentBasedDeduplication: true

    BookingQueue:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "BookingQueue.fifo"
        FifoQueue: true
        ContentBasedDeduplication: true
        RedrivePolicy:
          deadLetterTargetArn:
            Fn::GetAtt:
              - BookingQueueDLQ
              - Arn
          maxReceiveCount: 3

    snsToBookingQueueSQSPolicy:
      Type: AWS::SQS::QueuePolicy
      Properties:
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Sid: "allow-sns-messages"
              Effect: "Allow"
              Principal:
                Service:
                  - "sns.amazonaws.com"
              Resource: 
                Fn::GetAtt:
                  - BookingQueue
                  - Arn
              Action: "SQS:SendMessage"
              Condition:
                ArnEquals:
                  "aws:SourceArn": 
                    Ref: BookingTopic
        Queues:
          - Ref: BookingQueue

    BookingQueueSubscription:
      Type: 'AWS::SNS::Subscription'
      Properties:
        TopicArn: 
          Fn::Join:
              - ':'
              - - 'arn:aws:sns'
                - Ref: 'AWS::Region'
                - Ref: 'AWS::AccountId'
                - 'BookingTopic.fifo'
        Endpoint: 
          Fn::GetAtt:
          - BookingQueue
          - Arn
        Protocol: sqs
        RawMessageDelivery: 'true'