Developing and Testing Lambdas with pytest and LocalStack

Ciaran Evans
Jul 29 · 6 min read

Background

Recently I moved from being a Software Engineer within the UK Hydrographic Office to a new role as a Senior Data Engineer. The project my new team are working on is a pipeline composed of multiple AWS Lambda functions, processing large satellite image tiles from the European Space Agency and performing predictions on them with a model produced by our Data Science team.

To start, we had two options for testing our Lambda functions:

  • Copy + paste our code into the Lambda console and run them in production on live data (Risky and makes me wince 🤨)
  • Write extremely verbose tests using Moto (Time consuming, requires knowledge on how AWS works to a low level, and makes test code very noisy 🔈)

Clearly neither of the above are ideal, nor are they particularly geared towards fast feedback & delivery of value. After some Googling, I came across LocalStack; when asked ‘Why Localstack?’ they reply with:

LocalStack builds on existing best-of-breed mocking/testing tools, most notably kinesalite/dynalite and moto. While these tools are awesome (!), they lack functionality for certain use cases. LocalStack combines the tools, makes them interoperable, and adds important missing functionality on top of them

After some discussion it was deemed worth a look and after about a week and a half of experimentation, we have managed to integrate LocalStack into our development pipeline.

To discuss how we use LocalStack and how to get it working yourself, I’ll be writing a series of blogs in the currently proposed order:

  • LocalStack and pytest running locally 💻 (This blog)
  • LocalStack and pytest dockerised 🐳
  • How to deal with Lambda Layers and Geospatial dependencies 🌍

LocalStack and pytest running locally

If you would like to view the code in this blog, or indeed run it yourself, you can find it on my GitHub. The code displayed in the blog is via gist just so it looks nice! 💅

To start, you will need Python installed (The repo is written in 3.6.6 but most >3 versions should work). You will also probably want to have some way of creating Python virtual environments, there are plenty of choices but I prefer Pyenv with the Virtualenv plugin.

Once you’re happy with your environment, install the required packages for the examples. You can do this by running the following in the root of the project:

It would also be beneficial if you were comfortable with Boto 3.

The basics

To start, lets take a look at the code for the basic Lambda function. We’ll start with lambda.py within lambda/basic_lambda.

As you can see, this is about as simple as it gets. The Lambda will just log a string and return a response with a message.

The main brunt of the code is within testutils.py, this provides a few methods that make deploying a Lambda to LocalStack easier and means that our test file can be nice and succinct.

As you can see above, all the clients we instantiate with Boto3 take in empty values for aws_access_key_id and aws_secret_access_key, this is because LocalStack doesn’t need, nor know them. You merely provide values for the client you want, the region_name and most importantly endpoint_url. This is the major difference between writing applications that talk to AWS vs. LocalStack.

In the situation where you run LocalStack locally, it will expose ports on http://localhost:<service-port>.

Testutils provides methods for creating the .zip you upload to your Lambda, creating your Lambda, invoking your Lambda, and tearing it down.

Finally comes our test file test_lambda.py:

How small is that?! 😱

When you run your test, it creates a package with your Lambda function code, deploys that Lambda, calls the Lambda, verifies its response and cleans up after you.

Lets run it! 🏃‍♀️

First you will want to open a terminal window and run the following within the root of the repo:

Now that you have LocalStack running, open up a new terminal window or a new pane in your multiplexer of choice. Make sure you’re in the root of the repo. It’s time to run our test! 😬

😎 It passed! You’ve successfully run your first test against LocalStack!

If you take a look at the window with LocalStack running you should see something like:

Interacting with ‘AWS’ within a Lambda

So, that was cool. But all it did was return some text and log a string.

How about we try and interact with ‘AWS’ within our new LocalStack Lambda?

Take a look at lambda.py in lambda/s3_lambda:

The above is slightly different from our basic one. You’ll notice the method get_s3_client(), this returns us a S3 client based on whether the Lambda is in Production (therefore using the Lambda consoles built in authentication etc.) or anything else, where we’ll provide the values needed for LocalStack.

The handler simply logs another message and then adds a dummy object to the S3 bucket a-bucket which is created within LocalStack by the test file.

Let’s have a look at this Lambas testutils.py:

The above is similar to the basic Lambdas testutils, I’ve added four additional functions: one to retrieve the S3 client, one to create a bucket, one to list the objects within a bucket, and one to delete a bucket).

We also updated the Lambda to add the STAGE environment variable.

Again, the only difference between these functions and interacting with AWS normally are the clients.

Finally, lets take a look at test_lambda.py:

As you can see, we still have a nice clean succinct test file. This test adds one more setup and teardown step, we then invoke the Lambda twice, ignoring its response and asserting that the bucket does indeed contain two items.

Final thoughts

So, lets quickly summarise what we covered.

  • Running LocalStack locally
  • Writing some very basic AWS Lambda functions
  • Using pytest
  • Testing the packaging, deployment, and invocation of a Lambda function
  • Not spending a penny. ⛔️💰

Hopefully this will have been useful to you, it was certainly a fun week trying to get this stuff working. We’re hoping that we can fully integrate LocalStack into our CI/CD pipeline to streamline our Lambda development.

If you’ve any questions, please don’t hesitate to leave an issue on the repo or message me on Twitter @Ciaran_Evans.

Big thanks to all the folks who contribute to LocalStack, it’s bloomin’ awesome.

Watch this space for Part 2, where we’ll be covering LocalStack and pytest dockerised 🐳

Ciaran Evans

Written by

Senior Data Engineer @ UK Hydrographic Office

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade