Lean DevOps: Constant Memory AWS Lambda

Make your lambdas cheap, reliable, and fast. Example in Python.

Why you care

You want to run an AWS lambda without having it killed for exceeding memory limits, and without paying for the higher memory tiers (1536Mb will only run to 9c/hr but still you might do it a lot), and you want it to be fast. You need to stay in the memory and time limits of your lambda.

Why AWS Lambda

You need to run code, you want to run it on AWS Lambda because it’s cheap, because you don’t want to care about computers, or you need to automate something within the world of AWS.

The most significant limitation is that each request must complete in 5 minutes.

Solution: Always know how much memory your program uses

This approach should work in any of the runtimes that Lambda supports (python, node, and JVM at time of writing), with the appropriate library support.

The essence of the approach is to only ever be processing a known amount of data, so memory consumption never goes beyond a steady limit. Also don’t have any caches or queues in process that are unbounded in practice.

The simplest way to achieve this is to read a fixed batch size, process, output, and flush all references to the processed, output, and intermediate data. In python that looks something like this:

This is fine, but only makes sense if you can process the fixed size chunks, or you will need to figure out how to stitch together whole records across read boundaries.

In case it’s not clear from the above, this only makes sense if you’re processing data with multiple input records. If there’s no split boundary (whether it’s delimiters, files, or whatever), this won’t help. This also assumes that each practically sized batch averages out to a predictable amount of data. Both of these are likely for a lot of tasks that one might do in lambda.

The easy way

So, the very simplest way is not the easy way. The easier way is to use a form of lazy stream processing that can pull a record at a time up to a certain number of results, flush the batch, and continue.

In python, that means structuring your functions such that they consume iterables and return generators. This way, if you can put something on top of your stream that can read a record at a time and respect the iterator protocol (e.g. the csv.DictReader) you can simply read records from your final generator until you hit either a record limit, or other limit as necessary, then flush.

Time for an example:

You’ll notice that in this example, other than only operating on generators, all object allocations not used for the whole processing are segregated in helper functions, so that those objects can be garbage collected as soon as they’re not necessary. If you have any loop bodies that will live for significant time, or functions where you do allocate objects, consider deleting objects when you’re done with them, again to allow the garbage collector to do its work.

How big should my batch be?

That’s for you to decide! You’ll want to look at how much memory your objects consume, and the performance you want (sending batches out may be slow) and the performance implications down stream of batch size.

How do I profile my code?

That’s another post coming soon ;)