Files survive Lambda timeouts, but your handler doesn’t

In my previous post, I wrote about how AWS Lambda pauses threads between executions. That strange behaviour isn’t very useful, but there are ways to take advantage of container reuse.

Your code remains loaded in memory between executions. Therefore, objects stored outside the handler function can be reused during many executions. This is great for things like API clients, log managers, metric publishers, template caches, and IoC containers.
You could also load reusable data, such as configuration, from a file or remote service like DynamoDB, S3, or Parameter Store.

Remember, each container only runs one execution at a time. You don’t need to worry about thread-safety unless you’re creating threads yourself.

The handler function can also write to variables outside its scope for reuse in later executions. You could do this for things such as maintaining an open database connection, caching the results of an external API call, or returning previous responses based on input.

Did you know your function can write to disk? Lambda gives you 512MB of storage in /tmp to work with files. This limit can’t be changed and excludes the contents of your function package (code, libraries, and other assets) which are stored elsewhere.

We’re now at the title of this post and it’s time to look at some code. Below is a C# Lambda function I’ve written to show a gotcha in container reuse.

The constructor generates a GUID and stores it in an instance variable named _variableGuid
The EntryPoint method, which Lambda will call, prints _variableGuid and does one of three things based on the /tmp/zac file.

On the first execution, the /tmp/zac file won’t exist so it’s created by writing _variableGuid to it.

On the second execution, /tmp/zac exists and contains the same GUID as _variableGuid. In this case, the function sleeps for 3000ms causing it to timeout and fail.

On the third execution, /tmp/zac still exists but a new instance of the Handler class has been created as a result of the timeout. The new instance has generated a new _variableGuid.

Below are the CloudWatch logs from three executions. The green entries show when an instance of Handler is instantiated and the red entry is the timeout on the second execution. Take a look at the blue entries and you can see the behaviour I described.

Lambda timeouts reset your handler but don’t delete files or terminate the container. This will bust your caches, close your connections, clear your config, and undo anything else you’ve done to take advantage of container reuse.

You’re still going to get warm starts, though. Lambda won’t need to download your function package from S3 or setup slower things like VPC networking.

However, to get the most out of warm starts, you should keep timeouts to a minimum. That means setting your own timeouts on downstream dependencies and tuning your function’s execution timeout.

The Context Object that Lambda passes to your handler contains the time at which the execution will timeout and/or a helper function to determine how long is remaining until the timeout deadline.

From Working with XYZ » Context in the AWS Lambda Developer Guide

If you’re enjoying these posts, please let me know and I’ll keep them coming!