Lambda layer to use Numpy and Pandas in AWS Lambda Function

TLDR

• By using Lambda layers, you can import modules that are not included in the built-in Lambda Runtime.
• Make sure the folder structure of the module files you upload is correct.

Photo by Luigi Pozzoli on Unsplash

Motivation

When you use Lambda functions, many of the modules are included as built-in. However, in the case of Python for example, popular 3rd-party modules like Numpy, Pandas are not included. The Lambda layer enables you to import such modules. (Lists of all available Python modules are shown in this gist. )

In this post, I would show how to use Lambda Layer. Also, I describe the point where you often make mistakes.

Note: Lambda function with Python 3.7, x86_64 is used in this post.

Create Lambda layer

Error message in Lambda

If you try to import a module which is not built-in, you’ll see this error.

Function Logs
[ERROR] Runtime.ImportModuleError: Unable to import module ‘lambda_function’: No module named ‘xxxxxxxxxx’

xxxxxxxxxx might be numpy, pandas or other modules .(When I saw this error, I really didn’t understand because there was no error on my local machine. But I learned that we need Lambda layer for this kind of usage.)

Prepare zip of module

To create a lambda layer, a zip file of the module files is required. So here, let’s create zip files of numpy module and pandas module.

Create numpy lambda layer zip file

1. Go to pypi numpy page. (link)
2. On the Navigation pane, select Download files. Some versions of files are shown.
3. Download whl file for desired Python version, linux, x86_64. (numpy-1.21.4-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl)
4. Create a folder with the name python. Important: Other names than “python” cannot be used.
5. Move all the extracted files from whl to python folder . The structure of the folder will be like this.

python/
├── numpy
├── numpy-1.21.4.dist-info
└── numpy.libs

zip compress this python folder. The name of zip file is whatever, for example, numpy-lambda-layer.zip. Thus we prepared the zip file to upload as Lambda layer.

Create pandas lambda layer zip file

For pandas, you need modules of pandas and pytz. pytz is for timezone and dependency of pandas. (Note: numpy is also required to use pandas.)

From Pandas pypi page (link), download pandas-1.3.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl for Python 3.7 .

Visit pytz pypi page (link), download pytz-2021.3-py2.py3-none-any.whl .

As we did with numpy, create another python/ folder and move pandas and pytz. Zip to pandas-lambda-layer.zip .

python/
├── pandas
├── pandas-1.3.4.dist-info
├── pandas.libs
├── pytz
└── pytz-2021.3.dist-info

Upload zip to AWS

To upload the file, follow these steps on AWS console.

1. On the left side pane of the Lambda console, select Additional resources, Layers.
2. Select Create layer.
3. Set Name (ex. numpy-layer), choose Upload and set your file (numpy-lambda-layer.zip), Select Python 3.7 for Compatible runtimes.
4. Choose Create. The version of the layer is shown (start from 1).
5. (Do the same steps for pandas-lambda-layer.zip .)

Add layer and Use

Create a Lambda function with Python 3.7 runtime. On the console of this Lambda function, choose Add Layer.

In the Choose a layer section, select Custom layers radio button, set custom layer and version from drop-down fields. Select Add. You can add multiple layers to a Lambda function.

Once you’ve added layers, you can import modules!

Important Note: If you create the folder with name other than python in the previous step. Import fails, although lambda layers are created successfully.

This folder name is connected with PATH of Lambda. As shown in docs, lambda layer PATHs for Python Runtime are:

• python
• python/lib/python3.9/site-packages(site directories)

Therefore, please make sure to name python when creating a folder.

Summary

I have shared the step-by-step demo for creating Lambda layer. Using Python Runtime, 3rd party modules numpy and pandas are added as layers.

Appendix

Add layer — Use AWS layer

Actually, AWS supports layer for numpy and scipy by default. You can also use this.

Quota of Lambda layer

• size: 50MB
• num of layers: 5