Deduplicating Amazon SQS Messages

Processing exactly-once between multiple Lambda consumers

Every day I find myself facing interesting problems to solve working with Amazon Web Services (AWS). A recent issue of particular interest to me took the form of message deduplication in Amazon’s Simple Queue Service (SQS).

We have a standard SQS queue, automatically triggering a Lambda function to consume messages. This function scales automatically— increasing and decreasing its number of concurrent executions — to handle the number of messages…