Setting up a simple Lambda Layer in Python

Ben Mc
Ben Mc
Aug 2 · 5 min read
Photo by David Clode on Unsplash

Premise

I’ve been using the Serverless Framework in Python 3 for a while now and have had a lot of fun with it. It’s quick to set up, easy to deploy, and there is an ever-growing trove of documentation and tutorials out there.

However, despite all the fun, one of the main annoyances was deploying very large zip files. Python packages can get pretty big!

How big is big, you say?

Let’s say you need to use NumPy, one of the most downloaded Python packages. This package clocks in at around 85 MB unzipped.

Zipping up and deploying functions that utilise this package can be incredibly large and time-consuming, which can significantly slow down deployment and development time.


The Solution: Layers

At the end of November 2018, AWS released Lambda Layers as a way to solve this very problem and more. Lambda Layers acts as a way to split out your reusable code into separate components. That way, you can deploy changes to your Business Logic lambda functions only and keep your other, rarely changed, more common code, e.g. a NumPy library, in a separate layer.

Below is an example of how layers can split out common code in your lambda functions:


Example Time!

tl;dr: See a working example on github.

Below is a step-by-step guide of how to set up a working example using Lambda Layer.

Setting up a simple serverless Python function

Create a python 3 serverless service and create a virtual environment:

sls create -t aws-python3 -p sls-layers-python
cd sls-layers-python
virtualenv -p python3 venv
. venv/bin/activate

Install numpy:

pip install numpy

Create a simple hello function using numpy in the handler.py file:

Test this locally:

sls invoke local -f hello

Deploy time!

Now, before we deploy, let’s check out how big our zip file is, so we have a baseline to compare to later on. Run serverless package to zip up the function locally first and then check the zip file size that we would be deploying:

sls package
ls -lh .serverless/*.zip

From here we can see that the current zip file will be 42MB.

Let’s split this function up!

Before we begin, a bit of context. At the time of writing, the Serverless Framework can’t actually deploy layers separately. This is due to a limitation in CloudFormation. As mentioned in the linked article:

The best way to deal with this is by keeping your layer and your function in separate stacks.

Due to this, the current best practice is to split our code as follows.

Deactivate the virtual environment and change back to the root directory. Then, create another serverless stack, this time called sls-layers-python-requirements:

deactivate
cd ..
sls create -t aws-python3 -p sls-layers-python-requirements
cd sls-layers-python-requirements

This will create the following directory structure:

Then, re-initialise and reinstall the numpy package:

virtualenv -p python3 venv
. venv/bin/activate
pip install numpy
pip freeze > requirements.txt

Now, we need to ensure the Python packages are bundled in our deployment package. Let’s install the following plugin, serverless-python-requirements:

yarn add serverless-python-requirements --save-dev

Then, update the serverless.yml to the following:

And deploy:

sls deploy

This will take a while, but when this is done you should see the following:

In the above, we can see the output of a Lambda Layer. Let’s copy that and use it in our original sls-layers-python stack. In the directory, update the serverless.yaml file to the following:

Then, as before, let’s ensure that our serverless bundle can still reference the NumPy Python package when it is deployed:

yarn add serverless-python-requirements --save-dev
pip freeze > requirements.txt

Now let’s check the size of this new package before we wait for another 41 MB deploy. Run the following:

sls package
ls -lh .serverless/*.zip

6.2 KB is much better!

Let’s deploy this package using the following command:

sls deploy

Now that we have NumPy and our Python Business Logic both deployed, we can test this by running the following:

sls invoke -f hello

Happy days!!


What Limitations Exist?

Though this is a huge advancement in serverless development, there are a few caveats that you should still be aware of. The main ones, in my experience, are the following:

  • The overall deployment package size (Lambda Layers and Business Logic) can’t exceed 250 MB unzipped.
  • You can only use a maximum of five Lambda Layers per function.

Bonus Round!

One additional feature is the ability to reference layers from other AWS accounts. Due to this, many individuals and organisations have made specific Lambda Layers accessible to the public for everyone to use!

A list of curated AWS Lambda Layers can be found here.

What’s next?

If you have questions about any of the above, feel free to drop a question either in the comments below or in the GitHub repo linked in this article!

Happy coding everyone!

Better Programming

Advice for programmers.

Ben Mc

Written by

Ben Mc

Software Developer based in Sydney | Serverless | Javascript | Python

Better Programming

Advice for programmers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade