Infini-loop with the AWS SDK and the importance of Lambda idempotency

A short story today.

Did you know you can easily create an unknown loop when using the AWS SDK?

It all comes back to some of the defaults hardcoded into the AWS SDK.

By default any socket connection to another AWS service will time out after 120 seconds when using the Node.JS SDK or 60 seconds with Boto.

So if you invoke any service which takes longer than 120 seconds AWS will disconnect and reinvoke the service with the same parameters.

Here’s a slightly simplified version of an issue I witnessed play out in production recently.

Okay so I tricked you a little with the title. It’s not an infinite loop as AWS only retries 3 times before tripping a failure but it can lead to three successful calls downstream where a single failure is reflected upstream.

It’s an interesting edge case and if you are not aware of this little quirk of the SDK you can accidentally cause yourself some problems and lose some time wondering what the hell just happened.

It highlights how important it is that your lambda functions are designed to be idempotent. This is just one angle your lambda can be executed multiple times with the same payload.

Other scenarios include a service disruption causing your lambda to be re-invoked in another AZ.

Serverless can be a little more complex than advertised eh!?

There’s plenty of ways to design around this such which I’ll go into a further post.

About Me

A creative engineer and AWS Pro building products and freelancing with a love for AWS.

If you want to work together contact me over on brianfoody.com, on LinkedIn or Twitter for a chat.

And don’t forget to follow the Lambda Lego publication.