Deduplicating SNS and SQS messages using AWS’s new FIFO SNS Topics and a FIFO SQS Queue

I recently needed to build a process that fans out a bunch of work across multiple AWS Lambda functions. This allowed me to perform a bunch of tasks concurrently and reduce a 45 minute process down to just a few seconds. In doing so however I found myself troubleshooting a race condition when attempting to detect the completion of all the jobs. I fixed this by leveraging the new AWS FIFO SNS Topics and sending the SNS Message with a MessageGroupId and a MessageDeduplicationId.

As always, here is the example code and SAM Template.

AWS has documentation on the new FIFO Topics and a section about Message Deduplication. This functionality is really quite amazing. If you think about all the code and infrastructure you would need to stand up in order to do this yourself you will appreciate the simplicity in the new FIFO AWS SNS Topic! AWS will watch the flow of messages that are published to your topic. If SNS sees a second or third message that has the same DeduplicationId as the first (and in the same Message Group) SNS will drop the duplicated messages for you.

Message deduplication works when you have a FIFO SNS topic defined. I hope the reasons for this are obvious but if not, it is required to sequence the messages so the system can be aware of…