Managing Python modules in AWS Lambda layers

4 min readAug 7, 2020

Introduction

The use of Lambda layers makes developers’ jobs much easier by introducing modularity across multiple Lambda functions. The creation of layers and deployment packages is very similar except on small change — the folder structure. Depending upon the runtime, this will change as listed here. Layers are added to the /opt directory and are included in the $PATH. This means that no other edits to sys.path or environment variables are required.

The Prerequisites:

EC2 instance using Lambda AMI for the python for the Lambda function. This step is important for python runtime; make sure we are using the correct AMI otherwise this will not work.

Install the minor version of python you want.
# Download python36$ yum list | grep python36
python36-pip.noarch$ sudo yum install python36-pip.noarch -y
$ python3 — version
> Python 3.6.11# Download and install pip3
$ curl -O https://bootstrap.pypa.io/get-pip.py
$ python3 get-pip.py
$ rm get-pip.py

2. Upgrade and configure the AWS CLI

$ pip3 install --upgrade awscli --user
$ aws configure

Let’s get started.

Install custom boto3 version rather than the runtime shipped.

Let’s SSH into the instance we created with the AMI, and get started with creating a custom boto3 layer to be used in the function. The .zip file must contain a folder named — python where all the dependencies must exist.

$ cd ~
$ mkdir boto3
$ cd boto3$ pip3 install boto3 -t ./python # installs the latest# Or use the required version from: https://github.com/boto/boto3/releases
$ pip install boto3==1.10.2 -t ./pythonzip -r boto3.zip python/# Publish a new layer version named boto3 with a description.
$ aws lambda publish-layer-version --layer-name boto3 --zip-file fileb://boto3.zip --compatible-runtimes python3.6 python3.7 --description "boto3 v1.14.37"# Associate the layer to a function, or use the console
$ aws lambda update-function-configuration --function-name <function-name> --layers <layer-arn>

At the time of writing this, the latest version of boto3 was v1.14.37. We can verify this inside Lambda once the layer is added by:

import boto3
import jsondef lambda_handler(event, context):
    print(boto3.__version__)return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

You should see the required boto3 version we just added: 1.14.37

2. Install the Requests module for Python

cd ~
mkdir requests
cd requests# Install the module under a new python directory
pip3 install requests -t ./python
zip -r requests.zip python/# Create a new layer version
aws lambda publish-layer-version --layer-name requests --zip-file fileb://requests.zip --compatible-runtimes python3.6 python3.7 --description "Request module for Python runtime"# Use the layer in an existing function
aws lambda update-function-configuration --function-name <function-name> --layers <layer-arn>

Within the function, start making API calls with the requests module!

import json
import requestsdef lambda_handler(event, context):
    response = requests.get('https://requests.readthedocs.io/en/master/')
    print(response.text)
    
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

3. Python: Pandas, NumPy, SciPy

You can also create the modules using a virtual environment, with something like:

cd ~
mkdir numpy-pandas-scipy
cd numpy-pandas-scipy/# start virtual env for python3
python3 -m venv v-env# activate it and install the modules, and deactivate the virtual environment
source v-env/bin/activate
pip3 install pandas numpy scipy
deactivate# Copy the contents to a folder to zip it
cp -r ~/numpy-pandas-scipy/v-env/lib/python3.6/site-packages/ ~/numpy-pandas-scipy/python# Zip
zip -r ~/numpy-pandas-scipy/pandasnumpy.zip ~/numpy-pandas-scipy/python# Create a new layer
# If the layer is too large (> 65MBs), use S3 to upload the layer.aws s3 cp ~/numpy-pandas-scipy/pandasnumpy.zip s3://<s3-bucket>aws lambda publish-layer-version --layer-name numpy-pandas-scipy --content S3Bucket=<bucket-name>,S3Key=pandasnumpy.zip --compatible-runtimes python3.6 python3.7 --description "Pandas, Numpy & SciPy"# Finally update the function to use the new layer
aws lambda update-function-configuration --function-name <function-name> --layers <layer-arn>

If the layer is too large, the API call to publish will fail and hence we need to send the .zip file to S3 and then reference the S3 file in the publish-layer-version CLI command as above.

Once done, you should be able to import all these modules without any errors as long as we are using the AMI for the runtime Lambda provides. If you run into permissions error, make sure the files and the directories have the correct read/write permissions before creating the .zip files

$ chown -R ec2-user:ec2-user python/$ find . -type d -exec chmod 755 {} +
$ find . -type f -exec chmod 644 {} +

Verify permissions and folder structure by:

$ zipinfo requests.zipArchive:  requests.zip
Zip file size: 992799 bytes, number of entries: 278
drwxrwxr-x  3.0 unx        0 bx stor 19-Oct-25 17:50 python/
drwxrwxr-x  3.0 unx        0 bx stor 19-Oct-25 17:50 python/urllib3/
-rw-rw-r--  3.0 unx     5943 tx defN 19-Oct-25 17:50 python/urllib3/fields.py
-rw-rw-r--  3.0 unx     5991 tx defN 19-Oct-25 17:50 python/urllib3/request.py
...

I’ve added a bunch of Lambda samples over GitHub for Python and CLI if you want to see more use-cases. Hope this helps! 🎉

Managing Python modules in AWS Lambda layers

Introduction

The Prerequisites:

Written by Shreyas Gaonkar