AWS Lambda: Going Beyond 5 Minutes
Timeouts? There’s a solution — Roll Your Own Lambda.
I’ve been using AWS Lambda for a few months now. Both just for fun to experiment with and then also for production at my day job. It’s been a great experience and I have to say I’m in love with AWS for (micro)services.
There are certainly some limitations with Amazon’s services. While some of the limits can be increased or lifted by contacting Amazon, others are there by design and can’t be helped. Be sure to read the fine print on limits when architecting your software.
One such limitation with AWS Lambda is the 5 minute execution time limit (previously only 1 minute).
Truthfully, Lambda is best for short processes. The true idea of a “microservice.” In fact, it’s better for your AWS bill too.
However, I feel like Amazon is a little bit at fault for making us believe we can use it for longer processes when they go on about “ETL” with the announcement of the time limit increase.
Or maybe you knew about the limit and built something that you never expected to exceed 5 minutes (like me)? What then?
ECS To the Rescue!
Good news! Migrating your Lambda to run in a Docker container managed by ECS isn’t that hard and can allow you to use your existing Lambdas so you don’t need to code anything over again. It’s just a bit of configuration work.
I built a little wrapper, to be used for Node.js Lambdas (or perhaps a Lambda using Go binary that was executed by Node.js), that mimics AWS Lambda. You can alter the code or Dockerfile to taste. It’s really just two files, though I’m hoping to make a more convenient command line tool around it all.
What this does is lets you build a Docker image so you can do:
docker run my-dockerized-lambda ‘{“event”: “message”}’
It works from any machine where docker is installed with the image. You could even use it to test your (Node.js) Lambdas locally.
Then once you have the image registered in ECS’ registry and an ECS Task Definition setup, you can then call Amazon’s “runTask()” method with a CMD override to pass the event message as if it was Lambda — only without an execution time limit.
Now a container will be placed onto an EC2 instance you configured via ECS and start going to work. It will exit when done, freeing up space on your cluster for other containers to run. It’s like rolling your own Lambda! Only you’re going to pay for the EC2 instance(s) by the hour of course.
Running It
Aside from the AWS SDK, there’s a variety of ways to invoke a Lambda. API Gateway, SNS, event triggers on a scheduler, and now also CloudWatch events. However, running an ECS Task is a bit more limited.
Outside of Amazon’s console and the CLI, your easiest path to running ECS Tasks is through Amazon’s SDK directly from your application’s codebase.
You can of course have another Lambda run the ECS Task or a variety of ECS Tasks depending on the event message passed. Here’s a nice blog post about that. You can then use API Gateway with that Lambda to run ECS Tasks via an API.
What You Get & Don’t Get
Of course you don’t get CloudWatch or some of the other benefits you get with Lambda such as an AWS session and the AWS SDK automatically available to just start making API requests.
Though you could add more advanced logging with CloudWatch and environment variables for an AWS user in the Dockerfile or the container overrides even when you ran the ECS Task.
You also don’t get the “set it and forget it” factor. While it’s going to run for you with nearly the same ease of Lambda, you still are running on top of an EC2 instance. Or should I say a cluster. So that’s something to pay attention to. While ECS’ scheduler does a good job, you might be needing to keep an eye on your instances and plan to bring more online or increase their size.
You also don’t get the API Gateway integration. This is one of the biggest downsides. Like I mentioned previously, you’d need a new Lambda now or you could setup Elastic Load Balancer. However small or easy that is, it’s yet more to setup and configure. Possibly more cost too.
However, the biggest benefit you now get is unrestricted execution time. Your process can take as long as it needs now. This, in my opinion, is a far better solution for your ETL tasks.
You also can maintain your existing workflow that you had with Lambda…Because let’s face it — it’s very easy to build, test, and run small bits of code. That’s one of the appeals of Lambda and “microservices” in general.
There’s certainly more AWS configuration to do here, but it’s really good to know that you can get more life and flexibility out of your Lambda functions. I find this particularly helpful as your application evolves and you start realizing Lambda may no longer be a good place for some particular process.
It really saves you from having to start all over again and it can help you quickly put out a fire when your supposed to be short running process ends up taking more than 5 minutes and there’s a panic.
