AWS python lambda layers

This guide is explaining how and why :) to create an AWS python λ layer

As mentioned in AWS docs

“Lambda layers provide a convenient way to package libraries and other dependencies that you can use with your Lambda functions. Using layers reduces the size of uploaded deployment archives and makes it faster to deploy your code.”

“A layer is a .zip file archive that can contain additional code or data. A layer can contain libraries, a custom runtime, data, or configuration files. Layers promote code sharing and separation of responsibilities so that you can iterate faster on writing business logic.You can create layers using the Lambda console, the Lambda API, AWS CloudFormation, or the AWS Serverless Application Model (AWS SAM).”

Why do we need layers

So apart from the sharing code feature (we may like to have shared functions or configuration files between our lambdas) whenever we would like to add a new library to our python lambda other than the ones that are pre-bundled already in AWS, see a list [here]we would have to create a layer. An example would be that we want to add these libraries slack-sdk, pandas, cryptography calls in our lambda handlers.

How layers work

For each Lambda runtime, the PATH variable includes specific folders in the /opt directory

Whenever a new execution environment is set up for a λ it loads also the layers associated with it. It extracts the layers contents into the /opt directory when setting up the execution environment for the function.

Python modules located in the following folders inside the /opt can be used as libraries to be imported. This could be your custom code or external libraries.

python

python/lib/python3.8/site-packages

The packaged libraries that would reside inside the above folders have to be compliant with:

· your lambda architecture (x86_64, arm64 etc), the library may have instruction set specific files that need to be compliant with your lambda arch.

· your python version (3.8, 3.9 etc). Libraries for non corresponding python versions may differ.

Instructions

Step by step instructions to create the content for a layer:

1. Create a Dockefile with amazonlinux

for setting up the appropriate python environment(e.g. we want python3.8)

We want to create an environment with python as much close as we can to the actual environment the lambda will run.

FROM amazonlinux:2022.0.20220504.1
RUN yum install -y python38 && \
yum install -y python3-pip && \
yum install -y zip && \
yum clean all

Build the image with

docker build -f Dockerfile -t lambda_layer_constructor:latest .

2. Run the docker image for setting python env

docker run --name my-container -it  lambda_layer_constructor:latest bash

Create a venv for installing the pip dependencies inside

python3.8 -m venv myvenv
source myvenv/bin/activate

Check your pip version and python version

python3.8 -m pip --version

3. Install your dependencies.

Since the amazonlinux is arm64 based in case your target is another architecture you will have to define it in the pip command following the manylinux paradigm as of [here]

So for installing pyjwt targeting x86_64 architecture and cryptography you will have to run:

pip install --platform manylinux2014_x86_64 --only-binary=:all: cryptography -t ./python
pip install --platform manylinux2014_x86_64 --only-binary=:all: pyjwt -t ./python

The installed packages will be located inside the python folder and now we want to transfer them outside the container for creating the zip for the layer.

docker cp my-container:/python.zip ./Desktop/

4. Transfer the python libraries

First zip them inside the docker container

zip -r python.zip ./python/

Then copy them

docker cp <container_name or id>:/python.zip ./Desktop/

5. Finally you can create your layer

with the zip folder immediately (in case it’s less than 50mb else you will have to upload it to s3 and create a layer with the s3 arn)

Also you can change the folder structure from python to python/lib/python3.8/site-packages if that is more suitable.

For example for creating a layer inside the aws console you would have to do something like the following specifying your zip, architecture, python version compatibility :

5. References

Creating and sharing Lambda layers

Preinstalled Python Packages in AWS Lambda Runtimes

GitHub - pypa/manylinux: Python wheels that work on any linux (almost)