Serverless queues and workers — Designing Lift

Matthieu Napoli
Apr 30 · 4 min read

This article is part of a series on Lift, an open-source project that simplifies deploying serverless applications.

As introduced in previous articles, Lift is a Serverless Framework plugin that simplifies deploying serverless applications.

As we approach the first beta planned for May, let’s talk about queues and workers:

  • How to deploy production-ready queues and workers on AWS today.
  • How we plan to simplify that with Lift.

The naive approach

When going serverless, SQS + AWS Lambda is an iconic duo:

  • SQS is a queue service that is entirely managed by AWS, scales automatically, and bills by the usage.
  • AWS Lambda runs our code and is entirely managed by AWS, scales automatically, and bills by the usage.

The best part: Lambda integrates with SQS natively. We can write “queue workers” on Lambda without having to poll messages from SQS: instead, our code (the worker) is automatically invoked when a new message (aka job) is pushed into SQS.

To deploy queues and workers with the Serverless Framework, we need a bit of CloudFormation:

# serverless.yml
# ...

functions:
worker:
handler: worker.handler
events:
# Subscribe our function to the SQS queue
- sqs:
arn: !GetAtt ReportGenerationQueue.Arn
resources:
Resources:
# This creates an SQS queue
ReportGenerationQueue:
Type: AWS::SQS::Queue

So far so good! Or at least, it’s doable.

Problems with the naive approach

The biggest missing piece here is error handling.

By default, SQS retries failed jobs indefinitely. Imagine your code has a bug (let’s pretend that can happen): the job will be retried over and over for days, possibly costing a lot of money and wasting resources.

To deal with errors, we need to set up a “dead letter queue” (a queue that stores failed messages) and limit the number of retries.

On top of that, there are many details that needs to be configured, for example:

  • We should configure the dead letter queue to store failed messages for 14 days (the maximum).
  • We should set up an alarm that sends our team an email whenever there are failed messages in the dead letter queue.
  • Many other settings need to be fine-tuned: the SQS “visibility timeout”, the Lambda max concurrency, message batching, etc.

A production-ready approach

Here is a preview of a serverless.yml configuration that includes those best practices:

# serverless.yml
# ...

functions:
worker:
handler: worker.handler
timeout: 10 # seconds
# Scale to maximum 10 worker instances
reservedConcurrency: 10
events:
- sqs:
arn: !GetAtt ReportGenerationQueue.Arn
# Process messages without batching
batchSize: 1
resources:
Resources:
# Create an SQS queue
ReportGenerationQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub '${AWS::StackName}-myqueue'
# 6 times the lambda function's timeout
VisibilityTimeout: 60
# Messages will be stored (and retried) up to 2 days
MessageRetentionPeriod: 172800
RedrivePolicy:
# Jobs will be retried 5 times
maxReceiveCount: 5
# Send failed messages to the dead letter queue
deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn

# Failed messages will be sent to this queue
DeadLetterQueue:
Type: AWS::SQS::Queue
Properties:
# Store messages up to 14 days (the max)
# we want time to debug/analyse
MessageRetentionPeriod: 1209600

If we wanted to add an alarm for failed messages, we would need 30 more lines of YAML.

Needless to say:

  • this is hard to figure out,
  • this is tedious to implement.

We want to make that simpler with Lift.

Serverless queues and workers with Lift

We are currently working on a “Queues” component that can be deployed via serverless.yml:

# serverless.yml
# ...
queues: report-generation: # Our Lambda worker function is defined in the "queue" component:
worker:
handler: worker.handler
# Scale to maximum 10 worker instances
reservedConcurrency: 10

As we can see in the example above, there is a new queues section that lets us define queues (and their workers). In that instance, we define a report-generation queue and its worker function.

On serverless deploy , Lift will create:

  • A report-generation SQS queue configured following best practices.
  • A worker Lambda function, that will be automatically subscribed to the SQS queue.
  • A report-generation-dlq SQS “dead letter queue” that will receive failed messages.

By default, messages are configured to be retried up to 3 times.

We are considering several settings on the component, including setting an email alert on failed messages:

# serverless.yml
# ...
queues:
report-generation:
alarm: alerting@my-company.com
worker:
...

On top of that, Lift also simplifies pushing messages into the SQS queue:

  • Lambda functions defined in serverless.yml will automatically get IAM permissions to send messages to the SQS queue.
  • To simplify referencing the SQS queue, Lift introduces a new variable syntax: ${queues:my-queue.queueUrl}

Here is a complete example with a publisher Lambda function:

# serverless.yml
# ...
functions:
publisher:
handler: publisher.handler
environment:
QUEUE_URL: ${queues:report-generation.queueUrl}
queues:
report-generation:
worker:
handler: worker.handler

Deploying queues, static websites and file storage is a small part of what we are working on with Lift.

To follow Lift’s development, star or watch Lift on GitHub.

We are also looking for feedback on the Queues feature: get involved in the GitHub discussion.

Serverless Transformation

Serverless Tools, Techniques, and Case Studies

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store