AWS Lambda with Pandas and NumPy

AWS Lambda does not include Pandas/NumPy Python libraries by default. How use Pandas and NumPy with Lambda functions?

Image for post
Image for post

Problem statement

There are no default Pandas in AWS Lambda. You can see AWS Lambda execution environment and available libraries here. Let’s verify and create AWS Lambda Python 3.6 function with code below:

Lambda’s response:

Unable to import module 'lambda_function': No module named 'pandas'

Create new local directory with lambda_function.py file. Install Pandas to local directory with pip:

$ pip install -t . pandas

Remove *.dist-info and __pycache__. Prepare zip.zip archive with lambda_function.py file and Pandas:

$ rm -r *.dist-info __pycache__
$ zip -r zip.zip .
Image for post
Image for post

Create new lambda function (e.g. medium). Go to Function code section and select Upload a .zip file from Code entry type dropdown. Click Upload button. Upload zip.zip file. Finally click Save button:

Image for post
Image for post

Let’s check new Lambda function response:

Unable to import module 'lambda_function': Missing required dependencies ['numpy']

What? As you can see below, zip.zip archive includes NumPy.

Image for post
Image for post

Standard method does not work. AWS Lambda need special Pandas/NumPy. Let’s fix it.

Note: Do not forget clean your working environment, first. Remove pandas, numpy, and *.dist-info directories:

$ rm -r pandas numpy *.dist-info

Solution

AWS Lambda use Amazon Linux operating system. Idea is download Pandas and NumPy compatible with Amazon Linux.

Pandas. Navigate to https://pypi.org/project/pandas/#files. Search for and download newest *manylinux1_x86_64.whl package. In my case for Python 3.6 is pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl file.

NumPy. Do the same for NumPy. File is numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl.

Download whl files to directory with lambda_function.py. Unzip whl files.

$ unzip numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl
$ unzip pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

Note: Lambda with Python 3.7 requires pytz lib: $ pip install -t . pytz

Remove whl files, *.dist-info, and __pycache__. Prepare new zip.zip archive:

$ rm -r *.whl *.dist-info __pycache__
$ zip -r zip.zip .
Image for post
Image for post

Navigate to lambda function (e.g. medium). Go to Function code section and select Upload a .zip file from Code entry type dropdown. Click Upload button. Upload zip.zip file. Finally click Save button:

Image for post
Image for post

Let’s check new Lambda function response:

Execution result: succeeded

Congrats!

Written by

Python Developer and Artificial Intelligence Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store