How to lightweight your Python AWS Lambda functions with AWS Lambda Layer, Docker, and Terraform.

Since AWS released the Lambda development console, microservice development and debugging through console has been greatly improved. But we all face the same difficulties when your code scales up and your function size exceeds 3Mo. The console cannot display your code anymore.

Fortunately, AWS developed a new feature named AWS Lambda Layer to let you build common packages usable by the runtime.

We will dive into this use case: how can I package with ease a big Python library and use it through AWS console within one or multiple Lambda functions? Suppose that you have some data engineering stuff to do, you may need to use Pandas to workaround your application.

Software requirements:

• A Docker or Docker-CE installed,
• Terraform ~> 0.11.12 to use filebase64hash256 for Terraform to detect code changes, it can also work with base64sha256(file()) for Terraform earlier versions.

Project set up and code samples

First, create a structure as below :

iam.tf

resource "aws_iam_role" "role_for_lambda" {
  name = "role-lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012–10–17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
      "Service": "lambda.amazonaws.com"
    },
    "Effect": "Allow",
    "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "lambda_basic_execution_policy_attachement" {
  role       = "${aws_iam_role.role_for_lambda.name}"
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

File implementing a basic role for your Lambda function, add stuff according to your needs.

lambda.tf

data "archive_file" "lambda-archive" {
  type        = "zip"
  source_file = "lambda/src/main.py"
  output_path = "lambda/packages/lambda_function.zip"
}

resource "aws_lambda_function" "lambda-function" {
  filename         = "lambda/packages/lambda_function.zip"
  function_name    = "layered-test"
  role             = "${aws_iam_role.role_for_lambda.arn}"
  handler          = "main.handle"
  source_code_hash = "${data.archive_file.lambda-archive.output_base64sha256}"
  runtime          = "python3.7"
  timeout          = 15
  memory_size      = 128
  layers           = ["${aws_lambda_layer_version.python37-pandas-layer.arn}"]
}

resource "aws_lambda_layer_version" "python37-pandas-layer" {
  filename            = "lambda/packages/Python3-pandas.zip"
  layer_name          = "Python3-pandas"
  source_code_hash    = "${filebase64sha256("lambda/packages/Python3-pandas.zip")}"
  compatible_runtimes = ["python3.6", "python3.7"]
}

This file will deploy a Lambda function and a Lambda Layer packaging Pandas for Python3.

In your requirement file, add any packages according to your needs. Here, we will have a simple version of python-pandas.

requirements.txt

pandas==0.23.4
numpy==1.16.4
s3fs==0.2.1
pytz==2018.7

And then, here is the important part of the work, using LambCI Docker Image (we’ll use the python build tag but they have builds for each runtime supported by Lambda), build the deployment package for Lambda Layer as follows:

build_layer.sh

#!/bin/bash
export PKG_DIR="python"
rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.7 \
pip3 install -r requirements.txt --no-deps -t ${PKG_DIR}

Here you are, with a simple build.sh executing the code above and zipping the output.

build.sh

#!/bin/bash
export WRKDIR=$(pwd)
export LYR_PDS_DIR="layer-pandas"
        
#Init Packages Directory
mkdir -p packages/
        
# Building Python-pandas layer
cd ${WRKDIR}/${LYR_PDS_DIR}/
${WRKDIR}/${LYR_PDS_DIR}/build_layer.sh
zip -r ${WRKDIR}/packages/Python3-pandas.zip .
rm -rf ${WRKDIR}/${LYR_PDS_DIR}/python/

Note: AWS Lambda Layer allows users to upload other kinds of archive and files, for example, you can fill it with a .jar file if you work with Java functions

You’re now ready to launch Terraform to apply changes with a brand new Lambda function linked to a pandas Layer.

Now take a closer look to AWS Lambda development console, you can edit the Lambda function although a big package is linked to the function:

Now you can perform tests and build your own layer to ease development of AWS Lambda functions by yourself!

Afterwords

If you already worked with AWS SAM (Serverless Application Model) you must have heard about AWS Lambda Layer and it’s implementation within a SAM deployment which clearly is a better option if you’re trying to build a web service.

Another post talking about AWS Lambda Layer with step by step set up: here

Workaround with AWS Lambda Layer and NodeJS: here