Extending Lambda with ECS and Fargate — Fat Lambda

AWS Lambda is an incredibly powerful tool that offers a range of benefits, such as reducing costs and improving application resiliency. It’s also incredibly user-friendly, making it suitable for a variety of use cases. However, if you’re dealing with a workload that runs for longer than 15 minutes, requires more than 3 GB of memory, or demands a greater level of troubleshooting control, you may be wondering if Lambda is the right fit for your needs. In these situations, Lambda may not be the best solution.

Fortunately, there are several strategies that you can adopt in order to overcome these limits, one very known strategy is what so called the fat Lambda. Before digging in, I’d like to explain a few terms that I’ll be using later on, mainly Docker, ECS and Fargate. If you used AWS Lambda before, chances are that you already know what these terms mean, but for the sake of it, let’s briefly go over them.

- Docker is a way to package your project into a “file”. This “file” (called a Docker image) can then be easily dropped and run on any server. This makes your project very portable and allows it to be better organized. Docker groups software like plastic containers group fruit.

- ECS or Elastic Container Service is a fully managed container orchestration service that helps you easily deploy, manage, and scale docker applications.

- An ECS task is the act of running a docker container on ECS following some configuration specified in a Task Definition

- AWS Fargate is a technology that provides on-demand, right-sized compute capacity for docker containers

You might be wondering why we decided to go with Fargate, well, several of reasons actually, fist of which is Lambda has a 15 minutes max runtime, vs no time limit on Fargate, second, Lambda has a 3GB memory limit, vs 30GB on Fargate, third of all, you can’t control how much CPU you get, vs (you guessed it) you can on Fargate. So, without any further due, how can you implement this Fat Lambda strategy?

In order for us to understand this better, let’s take an example. Say you have a workload that you need to run, say you want to run a complex calculation that ‘ll take more than 15 minutes to run. Here’s the overall architecture of what we’re trying to achieve

Step #1: Dockerize your application:

You start first by writing your app in python for example, then you dockerize your application, you can simply do this by creating a new file and calling it Dockerfile with no extension, inside of this docker file you specify how you want to execute your code in a docker container. Can be something similar to this

Python application code:

Step #2: Push Docker image to ECR

Next, we need to build and push our docker image to ECR (Elastic Container Registry). Think of ECR as a place where you store your docker images, as simple as that. Check out this documentation on how to create your own ECR repository

• To login to the ECR repository, run:

aws ecr get-login-password — region <aws_region> | docker login — username AWS — password-stdin <aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com

• To build our Docker image we need to run:

docker build -t <app_name> .

• To push our Docker image to ECR we need to first login by executing:

docker tag airflow:latest <aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/<app_name>:latest

• And then we need to tag and push by running:

docker push <aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/<app_name>:latest

Step #3: Prepare the trigger Lambda

Now that we have our Docker image ready, we need to create our Lambda. This one will help us trigger a task (runTask) job on ECS and that’s it. Our Lambda will look something like this:

We still need to give permission to our Lambda in order to run the task, for that we have to change the Lambda IAM role and add permission to access and run tasks on ECS, for this go to your lambda configuration and click on the IAM role you see there

From here go ahead and add a new Policy with the following JSON code

Step #4: ECS

Finally, we move to ECS. There are a couple of steps to be done here.

Step #4.A: create an ECS cluster

This is basically a place where to manage all of the tasks that we’ll be running. To create a cluster head to the ECS service on your AWS console and click on create cluster

Step #4.B: Create task definition:

Next step is to create a Task definition, this is the configuration that we want to have for running our tasks. Click on Task Definitions and then Create new Task Definition button, and specify the following configuration

Leave everything else as default and click on Create.

Step #5: Testing

Now all that is left to do is to test our setup, for this we’ll prepare a test case in Lambda, so go to your Lambda function, click on Test tab and specify the following configuration

Click on Save and Test, and if you check your ECS cluster you’ll see that there’s a new task running this means that we have successfully run our dockerized application from ECS using Fargate.

Conclusion

AWS Lambda by itself is an amazing solution provided by AWS, and has a lots of benefits, you should consider using it whenever is possible instead of going with the Fat lambda approach as it is way cheaper and simpler to set up, but if you can’t, cause of memory or run time limit, you now know that the FAT LAMBDA got your back!

Interested in joining our team? Browse our open positions or check out what we do at HeyJobs.