How to fix the lambda package size limit exceeded error caused by python libraries

Python package deployment to AWS lambda

I’m using the K-means clustering algorithm to create a recommendation engine for an ecommerce application. As a result, I needed the Scikit-learn and pandas libraries, which increased the size of my deployment package beyond the AWS lambda package limits.

During the deployment of my stack I got this error:

Package Size limit exceeded

Deployment Package Limits

When developing a serverless application, you should keep some soft and hard limits in mind for each AWS service. These restrictions are intended to protect both the customer and the provider from unintentional use. In addition, guardrails are provided to ensure that best practises are followed.
AWS Lambda has a hard limit of 50MB for compressed deployment packages and a hard limit of 250MB for uncompressed deployment packages.

How to deal with size limits when we are using libraries like scikit-learn, pandas etc?

To help with managing potentially large dependencies, functionality for compressing libraries is available (for example: pandas, numpy, scipy, and scikit-learn). To decompress them, you’ll need to make some changes to your code.

Firstly add the following configuration to your serverless.yml file.

custom:
  pythonRequirements:
    zip: true

Then add this piece of code to the beginning of our handler module in your handler.py file.

try:
  import unzip_requirements
except ImportError:
  pass

Note: Handler function in handler.py is the entry point of any lambda function. It is the first function that is executed when your lambda function is invoked.

These modification in your code resolve your issue of package size limit exceeded.

A Lambda Layer is another option for dealing with large dependencies.

For this include the layer option in your configuration in serverless.yml file.

custom:
  pythonRequirements:
    layer: true

A layer will be automatically produced after the requirements are compressed into a zip file. The reference must be added to the functions that will use the layer.

Add this reference to the function define in serverless.yml file.

functions:
  myfunction:
    handler: handler.myfunctions
    layers:
      - Ref: PythonRequirementsLambdaLayer

You can also add this custom configuration into your layer in serveless.yml file :

custom:
  pythonRequirements:
    layer:
      name: ${self:provider.stage}-layerName
      description: lambda layer for python requirements
      compatibleRuntimes:
        - python3.8
      licenseInfo: GPLv3
      allowedAccounts:
        - '*'

Key Note: We can also use theImage package type, which has a much larger file size limitation but it requires more configuration as compared to above mentioned methods.

We all know that when we build a machine learning model the algorithms which we use consist of high packages libraries. So, to overcome the lambda package size limit while deploying these model we can use the above approaches.

Keep Learning Keep Growing 🥂