Serverless: Packaging User-Defined Python Modules

image courtesy of Serverless, Inc. © 2018

Using Serverless to create Python distribution packages for AWS Lambda can, at times, evoke a feeling of hopelessness and despair, especially when one finds oneself needing to bundle unique user-defined modules across multiple functions within the same Serverless project. Fortunately, there exists an excellent plugin to combat this exact feeling — the Serverless Python Requirements plugin.

This plugin allows one to easily bundle Python packages and the associated dependencies thereof, as defined by requirements.txt, during the Serverless package/deploy phases. To get an understanding of how we can configure Serverless and the python-requirements plugin to handle user-defined modules, I’m going to start out by using a stock-standard Serverless Python example that creates a simple HTTP endpoint, and then modularize it, and determine the best method to ensure a working Lambda distribution package.

The project layout — flat, simple, no dependencies, no user-defined modules.

In my experience, though aesthetically I do not like it, I’ve always found it easiest to work with Python in Serverless when the Python handler file is in the project root directory, as is the case in this example. The serverless.yml file is rather straight-forward:

serverless.yml — no package configuration, just a pointer to handler.endpoint, which will be the entry-point for the currentTime Lambda function.

So what happens if we do not like having handler.py in the root project directory? It often doesn’t make sense for larger projects, simply adding clutter where most other runtimes would, for example, at least be found contained within something akin to a src/ directory. So let’s do that here. We’re also going to change the runtime to Python3.6 because stop using Python 2.7

Project structure — handler.py is now contained within src/

serverless.yml — updated the runtime to python3.6, pointing the handler to src/handler.endpoint

So far so good! Our API returns 200 despite moving the handler.py file into another directory. This is because we’re still telling Lambda exactly where to find that file via the handler path in our serverless.yml. Another way of expressing this, and what Lambda is technically doing behind the scenes, would be:

from src.handler import endpoint

Which seems fine at first… But if we have to import from src, then surely the handler is being executed from the root project directory? Let’s find out…

handler.py — importing os, and returning the current working directory in the response

Using the above, we can determine the current working directory for the Lambda function to be /var/task/, which suggests that our handler would be found in /var/task/src/handler.py — so what is the problem? Well, what happens if we do something like this?

Separating out some common functions into a custom module

src/lib/common.py — separating out a response_builder function to handle forming responses

src/handler.py — using an absolute import for lib/common.response_builder and having it handle creating the response

Granted, this is a pretty simple and entirely unnecessary change, however it does illustrate the point — sooner or later, you are going to want to be able to import user-defined modules within your handler, so you might end up doing something similar to the above. All we’ve done here is create a common location for functions we may want to reuse. We’re using an absolute import to get response_builder(), as this is recommended by PEP 8. However, invoking this Lambda function results in the following:

oh snap.

Of course, absolute imports are recommended, but this assumes that the entry-point for your application exists within the root of the working directory in which it is executed. But Lambda doesn’t do this, because Serverless isn’t packaging it with the src/ as the root of the distribution package, so Lambda checks the handler path we’ve configured, and imports it as a module:

from src.handler import endpoint

And then, in handler.py, we’re trying to import lib.common, but that isn’t a valid file path in the root folder, so we’re not going to find it.

So what do we do now? Try a relative import? Yeah, that’s what I thought too:

handler.py — using a relative sibling import

double snap.

Moving the import statement into the handler.endpoint function results in similar errors, as we’re still trying to import a module of which the execution runner does not know the location. And generally we want to avoid faffing with sys.path in every project we write.

So how can we use serverless-python-requirements to help us out of this? So far, everything I’ve found regarding the use of this plugin was to enable easy packaging of external module requirements (numpy, pandas, etc.), but in this circumstance, we’re specifically wanting to enable the use of user-defined modules. If we were to sum up the issue we face when using user-defined modules with Python and Serverless, it is thus:

Serverless will, by default, package a function with the location of the serverless.yml file as the root folder. This breaks Python’s ability to import user-defined modules in a sane way when the defined handlers are not in the root project folder.

By using the serverless-python-requirements plugin, we are able to force Serverless to redefine the distribution package structure such that handler.py is the root folder. For example, let’s look at the distribution package structure differences between using Serverless, versus using Serverless with the python-requirements plugin:

Serverless base distribution package structure without any plugins. Note the presence of the src/ directory, as well as all the superfluous files — why on earth is README.md included? (because we’re not performing individual packaging in Serverless)

serverless-python-requirements plugin distribution package structure. The src/ folder is not present, nor are any other superfluous files.

The latter, of course, works correctly. handler.py is in the root folder of the distribution package, which allows the Python imports to work as expected.

serverless.yml — we’ve included the serverless-python-requirements plugin

We’re now telling Serverless to package functions individually, and we’ve modified the configuration of the currentTime function. serverless-python-requirements uses the module option to determine the folder location of your handler entry-point. Thus, where previously handler was configured to src/handler.endpoint, we now have it configured to just handler.endpoint, and module is configured to src. We also need to include an empty requirements.txt file within src/ otherwise serverless-plugin-requirements throws an error — probably because it is usually used to package external dependencies as opposed to just hackily avoiding the default Serverless Python distribution package structure. Now our imports are simple again:

With serverless-python-requirements ensuring our defined handler is in the root folder of the distribution package, we can go back to using recommended import syntax.

Now what if currentTime was one of several Lambda functions within this Serverless project? We don’t just want to store everything for all the functions in src/. Fortunately, this process can easily be expanded to include additional functions in alternate directories by modifying the module option:

serverless.yml — we can further nest our handler within src/ by specifying the path in the module configuration

Here we’ve nested our currentTime handler within src/currentTime/handler.py. This way we can have multiple functions contained within their own separate folders within src/.

In conclusion, we’ve discovered that the default way Serverless structures Lambda distribution packages is not inherently compatible with the way Lambda (or Python, for that matter) expects the package to be structured. We used serverless-python-requirements to handle the restructuring of the default Serverless distribution package such that our handlers are packaged within the root folder thereof. This can be applied to multiple functions with the use of the module configuration option, provided we configure individual packaging in serverless.yml

The modified aws-python-simple-http-endpoint code can be found on my Github.