Serverless and Python

Setting up a local environment

If you are anything like me, you want, nay need a working version of your API/service running locally when you are developing. When using node as the runtime with the serverless framework this process is pretty painless. Install the serverless offline plugin with:

npm install serverless-offline  -- save-dev

Add the plugin to your serverless.yml, and start it with:

serverless offline start

It could hardly be simpler, and I hope it can be forgiven that I expected it to be just as simple when using python as the runtime. Unfortunately here’s the catch, that lovely generic sounding serverless offline plugin actually expects your code to be written in node. So in steps the serverless simulate plugin.

On first glance it looks great, offering a detailed simulation of the API gateway and lambda services. With lambdas being run through docker-lambda, we should be able to simulate any runtime we want. So lets get started. I’m going to assume you have docker installed, if not, you’ll need it. First step install the simulate plugin, and let’s have the python requirements plugin too (though you may already have this).

npm install serverless-plugin-simulate --save-dev
npm install serverless-python-requirements --save-dev

Then add a bit of configuration to the serverless.yml. We want to enable both plugins, and tell the python-requirements plugin to build the libraries in a docker (just to make sure they are compatible with where you are going to run them).

plugins:
  - serverless-python-requirements
  - serverless-plugin-simulate

custom:
  pythonRequirements:
    dockerizePip: true

You will likely also need the (node) requests library which the simulate plugin uses, so lets install that too.

npm install requests --save-dev

You might now assume we would be ready to launch the local server, and if you have no python requirements you would be right.

serverless simulate apigateway

This should start up everything you need (assuming you don’t need to invoke any lambdas from your handlers). If you do have any requirements though your endpoints will error, when you try hitting them. This is because there is no interaction between the serverless-python-requirements plugin and this simulate plugin. The requirements weren’t bundled with your code. So, what to do.

The simulate plugin will be working with a copy of the entire directory (where your serverless.yml sits). So we need to build our requirements there, and convince docker-lambda to use them. We have the python requirements plugin, which will build them with the right architecture for us so lets do that:

serverless requirements install

Now they should be sat in the directory ./serverless/requirements which will be copied to the docker image that our local lambda is running, so all that remains is to add those requirements into the PYTHONPATH.

So here is where it gets a bit tricky and pretty hacky. We have to build our own version of the docker image that the simulate plugin will use. Since docker will pull from our local images first this should trick it into using our local image rather than the one on dockerhub. So let’s make our image. I am going to assume your project is in python3.6, though you could follow similar steps to make this work with python2.7. First we will get the default image that is being used for your local lambda, then we are going to tweak it a bit and build it locally. So let’s get started.

git clone https://github.com/lambci/docker-lambda.git
cd docker-lambda/python3.6/run

Open up the Dockerfile and you’ll find a line like:

PYTHONPATH=/var/runtime

Let’s change this to:

PYTHONPATH=/var/runtime:/var/task/.serverless/requirements

Now let’s build the new image:

docker build -t lambci/lambda:python3.6 .

Now when we launch our local simulation, docker-lambda will take our newly changed image, and we should be all gravy.

serverless simulate apigateway

For an excellent summary of how the simulate plugin really works I found John McKim’s post invaluable.