Storyboard Dev Blog: Serverless Compute with Dynamic Ephemeral Storage

We use serverless technologies for many tasks at Storyboard. RESTful APIs, WebSockets, and audio processing are great fits for serverless, and AWS Lambda is at the center of the serverless world.

The Situation

Lambda’s 512 MB of ephemeral storage has always been great for tasks that don’t require downloading large files to disc. It offers fast and simple access to a standard file system, is automatically cleaned up for you when the function ends (or you can wipe it yourself if you’re paranoid), and it’s free!

However, you may occasionally need access to far more than what ephemeral storage has offered up to this point. We frequently work with audio files in the single-digit gigabyte range. While streaming this data back and forth in memory is often suitable, there are plenty of occasions when it isn’t. In times like these, the solution is often to kick it out to an EFS volume and go from there. While an acceptable solution, it does involve more work, more overhead, more maintenance, and more cost.

In March of 2022, AWS announced a significant upgrade in what you can do with Lambda ephemeral storage. Previously, Lambda had one option in this regard: 512 MB in the /tmp directory.

Now, you can configure ephemeral storage for your Lambdas from anywhere between 512 MB and 10,240 MB (10 GB!).

Much has been written about this already, so I won’t rehash the details here, but it is a significant upgrade in terms of both the possibilities for Lambda and the ease of working with it. I wanted to take advantage of this for some of our serverless services as quickly as possible.

The Problem

The rub, though, is that you are charged for the amount of storage you configure for the duration of your function invocations. The default 512 MB of storage is still free, but once you start configuring more than that, your functions will begin to cost more.

Lambda’s updated pricing page shows that cost is still calculated based on how long your function takes to execute and how much RAM it uses, but now you’re also charged for any additional allocated storage. $0.0000000309 for every GB-second, to be exact, where a GB-second is number of seconds your function runs for * amount of storage allocated.

This led our lead engineer to pose an interesting question: Is there a way to configure a function’s ephemeral storage on a per-invocation basis?

Put another way: every time a function is invoked, can we dynamically set the amount of storage attached to it in order to minimize unused storage and save us money?

As I mentioned early in the post, we work primarily with audio files. These can range from small files in the single-digit megabyte range to large files in the multi-gigabyte range. The only way we can guarantee enough space to download these files and process them locally is to configure the maximum 10 GB of ephemeral storage. 99% of the time, though, we don’t need nearly that much storage space. That’s a lot of unused storage, and, more importantly, that’s a lot of unnecessary costs.

Sadly, the basic answer to our question was a resounding “no, you can’t do that.” You manually set the amount of storage on the function itself, and then you’re done. You can edit it after the fact, but you can’t change it dynamically on each invocation — and that makes sense.

AWS says “The /tmp directory is backed by an Amazon EC2 instance store and is encrypted at-rest.” I imagine configuring an EC2 instance store with varying amounts of storage before each invocation would, among other things, dramatically slow down start-up times and thereby contribute to longer cold starts.

So there I was — understanding of the situation, but disappointed that this feature didn’t exist. And that’s when some inspiration, a bit of creativity, and the power of the Serverless Framework combined to form the solution I’m working with today.

The Solution

What if, instead of one catch-all function that’s configured with the maximum 10 GB of storage and wastes money, we had many functions, each configured with different “tiers” of ephemeral storage.

We could then have a “Step 0” single-responsibility function that acts like a point guard in basketball — analyzing the file size from S3 using s3.headObject() and then dishing out the work to the specific function with exactly the right amount of storage via various SNS topics. Each of these functions runs the same code, they’re just configured with different levels of storage.

Now you’re thinking, “But that is so much duplicated code!”, and you’d be right — if you’re writing your Lambdas manually in the console. But, honestly, you should never do this. A piece of advice and probably the subject of a future blog: treat the AWS Console like a read-only view into your AWS account. But I’m getting ahead of myself.

I mentioned we use the Serverless Framework. This simplifies and automates serverless deployments in a myriad of ways, and it plays a key role here.

Instead of writing the same code again and again in the console and assigning each function different amounts of storage, I can write the code once and declare multiple functions that each reference that same code, while assigning different amounts of storage for each one. With the Serverless Framework, that can look as simple as this:

functions:
 512MBHandler:
   handler: audioProcessor.encode
   ephemeralStorageSize: 512
   events:
     — sns: ${param:512MBSnsTopic}
 
 1GBHandler:
   handler: audioProcessor.encode
   ephemeralStorageSize: 1024
   events:
     — sns: ${param:1GBSnsTopic}
 
 3GBHandler:
   handler: audioProcessor.encode
   ephemeralStorageSize: 3072
   events:
     — sns: ${param:3GBSnsTopic}
 
 5GBHandler:
   handler: audioProcessor.encode
   ephemeralStorageSize: 5120
   events:
     — sns: ${param:5GBSnsTopic}
 
 10GBHandler:
   handler: audioProcessor.encode
   ephemeralStorageSize: 10240
   events:
     — sns: ${param:10GBSnsTopic}

That yaml automatically configures five new Lambda functions. Each function references the same audioProcessor.encode code, but they are each configured with different amounts of storage and they are each triggered by different SNS topics.

Remember: with Lambda, simply creating these functions doesn’t cost you anything. The costs only begin to incur when these functions are actually invoked and run. So there’s no real downside to creating many functions like this (aside from the slight overhead of having several similar functions in your account).

Now let’s circle back to that “Step 0” function I mentioned a few paragraphs ago. It might look something like this:

*this is simplified, but it follows the same basic logic I use in my actual Step 0 function*

const fileSizeInMB = await getFileSizeInMB(s3Bucket, s3Key)
const storageTierToUse = getNearestStorageTier(fileSizeInMB)
const snsTopicArn = getSnsTopicToTriggerDynamicStorageFunction(storageTierToUse)
await sns.publish(snsParams).promise()

This is the function that is first triggered. It could be triggered by an upload to S3, an API request, or something else. It acts as the buffer zone to figure out how big of a file we need to deal with. Once it knows that, it publishes the specific SNS topic that triggers the function with exactly the right amount of ephemeral storage — saving you a significant amount of money in the long run! Now instead of calling one catch-all function with 10 GB of ephemeral storage, you’re calling precisely the function that has the amount of storage you need to process the file.

Final Thoughts

I know the Serverless Framework has a robust plugins directory. If there is a way for me to build this into a plugin, and there are people who would use it, I will absolutely do that. I haven’t looked into the plugins ecosystem enough yet to know how it works or if this would be a feasible integration. If you have some insights here, please get in touch.

And if you know of a better way to do this, please let me know! I’d love to hear your thoughts, opinions, and feedback. This is simply an idea I had that proved to be useful without being overly complicated.

Thanks for reading. And remember: just because AWS doesn’t offer some feature out of the box doesn’t mean you can’t MacGyver it yourself!

Check out a fully deployable example of this on GitHub.