Deploy Python Lambda functions with container images

1. Introduction

Welcome to our beginner-friendly blog series! We’re diving into the essentials: AWS Deployment, CI/CD, and the tech behind Digital Transformation with a touch of AI. Why? Because I get the struggle. Whether you’re self-learning or in a small organization, these topics can feel like a maze.

I’ve been there, and now it’s time to break it down for you. No jargon, just clear insights. Let’s make AWS Deployment, CI/CD, and Digital Transformation accessible to all developers, regardless of experience.

In this first article, we’ll walk through the basics of deploying a Lambda function using Docker images. Let’s dive in together!

Code is available at this repo.

2. Rationale

There is a significant issue with using a Lambda layer in the deployment of a Lambda function — specifically, the 250MB limit on dependency packages. This limit is relatively easy to exceed. Fortunately, there is a workaround for this issue: deploying Lambda functions using container images.

3. Deployment steps

3.1. Prerequisite

• Docker
• AWS CLI

3.2. Handler

For illustration purpose, we will just have a very simple lambda handler. The handler simply logs the received event and context and returns “Hello world”.

./src/main.py

import logging

logging.basicConfig(level=logging.INFO, force=True)
logger = logging.getLogger(__name__)

def handler(event, context):
    logger.info(f"Received event\n{event}")
    logger.info(f"Received context\n{context}")

    return "Hello world"

3.3. Dockerfile

We will use an AWS base image for Python. You can check the base images for other coding language in AWS Documentation.

./Dockerfile

# Step 1: Pull the base image
FROM public.ecr.aws/lambda/python:3.10

# Step 2: Copy requirements.txt
COPY ./requirements.txt ${LAMBDA_TASK_ROOT}

# Step 3: Install the specified packages
RUN pip install -r requirements.txt

# Step 4: Copy function code
COPY ./src ${LAMBDA_TASK_ROOT}

# Step 5: Set the CMD to your handler
CMD [ "main.handler" ]

Note that, in AWS Lambda environments, the LAMBDA_TASK_ROOT environment variable is automatically set to the path where your Lambda function code is located. This variable helps your Lambda function discover the location of its code and dependencies at runtime. Hence, in second step and forth step, we copy the requirements.txt file and src folder to the location LAMBDA_TASK_ROOT within the Docker image.

Step 5 is to set the default command to be executed when the Docker container starts. In this case, it specifies that the Lambda function handler is main.handler. Adjust this line based on the actual structure of your Lambda function code.

3.4. Deployment scripts

3.4.1 Uploading Image

./deployment/upload_image.sh

# Step 1: Configure AWS credentials and application-related values such as application name, environment (e.g., dev, sqa, uat, and prod), and AWS account number
aws_profile="$1"
aws_region="$2"
env="$3"
app="$4"
aws_account="$5"

# Step 2: Formulate ECR values
ecr_repo="$app-$env"
base_uri="$aws_account.dkr.ecr.$aws_region.amazonaws.com"
image_uri="$base_uri/$ecr_repo:$env"

# Step 3: Upload docker image to ECR
# Step 3.1: Docker Image Built
echo "Docker Image Built"
docker build --platform linux/amd64 -t $ecr_repo:$env ../

# Step 3.2: Login into AWS ECR
export ecr_password=$(aws ecr get-login-password --region $aws_region --profile $aws_profile)
eval $ecr_password
echo "Successful ECR Login"
docker login --username AWS --password $ecr_password $base_uri

# Step 3.3: Tag the Docker image
docker tag $ecr_repo:$env $image_uri
echo "Successfully tagged the ECR image"

# Step 3.4: Push the Docker image to ECR repository
docker push $image_uri
echo "Successfully pushed the Docker image to ECR"

There are a few things to note right here:

• In the first step, we are taking from arguments when running the script.
• In the second step, I name my ECR repo with {app}-{env} format. You could change it to whatever you want. However, in my perspective, to make consistent and easy to track resources across the environment, you should keep this format.
• In step 3.1, it is important to use --platform tag as linux/amd64 to make it compatible with AWS machine. Docker will default to the architecture of the machine on which you are running the build command. Hence, if you are using Windows, it is fine to remove the --platform tag. However, if you are using MacOS, Docker will default to the architecture to arm64.

3.4.2 Deployment script

./deployment/deploy.sh

# Step 1: Get the environment
if [ -z "$1" ]; then
  env="dev"
else
  env=$(awk '{print tolower($0)}' <<< "$1")
fi

# Step 2: Specify your aws credentials and app name
aws_profile="AdministratorAccess-007985056474"
app="lambda-docker"
aws_account="007985056474"
aws_region="ap-southeast-1" 

# Step 3: Run uploading image script
source ./upload_image.sh $aws_profile $aws_region $env $app $aws_account

We have to replace above values for your own AWS credentials. The environment used for deployment will be dev by default if not specified.

3.5. Create repository in ECR

If you are following my naming convention as {app}-{env} in above steps, please use the same format here. In my case, it is lambda-docker-dev.

Configure ECR repo

Hit Create repository to create the ECR.

Create ECR repo

3.6. Push image to created ECR repo

• Navigate the ternimal to deployment folder
• Run `deploy.sh` file with the command bash deploy.sh

Run deployment script

After that, you should be able to see the image inside ECR via AWS console.

Image inside ECR repo

3.7. Create Lambda function

We are now able to create a Lambda function with the created image.

Please to the option to create with Container image as below. After that, name your Lambda function and copy the container image URI from ECR console. I chose the same naming convention for Lambda function here.

Configure Lambda Function

Hit Create function. That’s it. We finish.

Create Lambda function

4. Validation

By now, you should be able to see your Lambda function created.

Let’s test the function directly from the console. Go inside the created function, change to Testtab and hit Test button.

If the function is working properly, you should be able the success message.

Yay! It works.

You are now free to change the handler code in ./src/main.py and add all necessary packages in ./requirements.txt to meet your own needs.

5. Redeployment

After you changed the handler code and updated all dependencies, simply run the script like what we have done in step 3.6, and update the lambda function as below:

1. Open the Lambda function from AWS console
2. Click Deploy new image

3. Click Save and we finish.

6. What’s next

As you may see, in our deployment steps, there are a lot of manual work had to be done. Let’s see how we could automate this entire the process by utilizing Terraform. Stay tuned for my article.