Image for post
Image for post

How to lightweight your Python AWS Lambda functions with AWS Lambda Layer, Docker, and Terraform.

Tanguy Combe
Jun 11, 2019 · 3 min read

Since AWS released the Lambda development console, microservice development and debugging through console has been greatly improved. But we all face the same difficulties when your code scales up and your function size exceeds 3Mo. The console cannot display your code anymore.

Fortunately, AWS developed a new feature named AWS Lambda Layer to let you build common packages usable by the runtime.

We will dive into this use case: how can I package with ease a big Python library and use it through AWS console within one or multiple Lambda functions? Suppose that you have some data engineering stuff to do, you may need to use Pandas to workaround your application.

  • A Docker or Docker-CE installed,
  • Terraform ~> 0.11.12 to use filebase64hash256 for Terraform to detect code changes, it can also work with base64sha256(file()) for Terraform earlier versions.

First, create a structure as below :

Image for post
Image for post

iam.tf

resource "aws_iam_role" "role_for_lambda" {
name = "role-lambda"

assume_role_policy = <<EOF
{
"Version": "2012–10–17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}

resource "aws_iam_role_policy_attachment" "lambda_basic_execution_policy_attachement" {
role = "${aws_iam_role.role_for_lambda.name}"
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

File implementing a basic role for your Lambda function, add stuff according to your needs.

lambda.tf

data "archive_file" "lambda-archive" {
type = "zip"
source_file = "lambda/src/main.py"
output_path = "lambda/packages/lambda_function.zip"
}

resource "aws_lambda_function" "lambda-function" {
filename = "lambda/packages/lambda_function.zip"
function_name = "layered-test"
role = "${aws_iam_role.role_for_lambda.arn}"
handler = "main.handle"
source_code_hash = "${data.archive_file.lambda-archive.output_base64sha256}"
runtime = "python3.7"
timeout = 15
memory_size = 128
layers = ["${aws_lambda_layer_version.python37-pandas-layer.arn}"]
}

resource "aws_lambda_layer_version" "python37-pandas-layer" {
filename = "lambda/packages/Python3-pandas.zip"
layer_name = "Python3-pandas"
source_code_hash = "${filebase64sha256("lambda/packages/Python3-pandas.zip")}"
compatible_runtimes = ["python3.6", "python3.7"]
}

This file will deploy a Lambda function and a Lambda Layer packaging Pandas for Python3.

In your requirement file, add any packages according to your needs. Here, we will have a simple version of python-pandas.

requirements.txt

pandas==0.23.4
numpy==1.16.4
s3fs==0.2.1
pytz==2018.7

And then, here is the important part of the work, using LambCI Docker Image (we’ll use the python build tag but they have builds for each runtime supported by Lambda), build the deployment package for Lambda Layer as follows:

build_layer.sh

#!/bin/bash
export PKG_DIR="python"
rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}
docker run - rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.7 \
pip3 install -r requirements.txt - no-deps -t ${PKG_DIR}

Here you are, with a simple build.sh executing the code above and zipping the output.

build.sh

#!/bin/bash
export WRKDIR=$(pwd)
export LYR_PDS_DIR="layer-pandas"

#Init Packages Directory
mkdir -p packages/

# Building Python-pandas layer
cd ${WRKDIR}/${LYR_PDS_DIR}/
${WRKDIR}/${LYR_PDS_DIR}/build_layer.sh
zip -r ${WRKDIR}/packages/Python3-pandas.zip .
rm -rf ${WRKDIR}/${LYR_PDS_DIR}/python/

Note: AWS Lambda Layer allows users to upload other kinds of archive and files, for example, you can fill it with a .jar file if you work with Java functions

You’re now ready to launch Terraform to apply changes with a brand new Lambda function linked to a pandas Layer.

Now take a closer look to AWS Lambda development console, you can edit the Lambda function although a big package is linked to the function:

Image for post
Image for post

Now you can perform tests and build your own layer to ease development of AWS Lambda functions by yourself!

If you already worked with AWS SAM (Serverless Application Model) you must have heard about AWS Lambda Layer and it’s implementation within a SAM deployment which clearly is a better option if you’re trying to build a web service.

Another post talking about AWS Lambda Layer with step by step set up: here

Workaround with AWS Lambda Layer and NodeJS: here

WeScale

THE WESCALE EXPERTS BLOG — DEVOPS, CLOUD AND PASSION…

Tanguy Combe

Written by

WeScale

WeScale

THE WESCALE EXPERTS BLOG — DEVOPS, CLOUD AND PASSION Regularly find articles written by our consultants on the Cloud and DevOps technologies that make up our daily lives.

Tanguy Combe

Written by

WeScale

WeScale

THE WESCALE EXPERTS BLOG — DEVOPS, CLOUD AND PASSION Regularly find articles written by our consultants on the Cloud and DevOps technologies that make up our daily lives.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store