Developing and Testing Lambdas with pytest and LocalStack
- Update 06/2021 — I should point out that this is not maintained and I believe LocalStack has moved on a fair bit since I wrote this blog (I know that all services are exposed via the same port for one.). Hopefully though, from the traffic I get on this, there is still some use to you!
- Update 08/2021 — I’m just linking this PR made on the GitHub repository that refactors the examples here to all run in Docker — https://github.com/ciaranevans/localstack_and_pytest_1/pull/6
Background
Recently I moved from being a Software Engineer within the UK Hydrographic Office to a new role as a Senior Data Engineer. The project my new team are working on is a pipeline composed of multiple AWS Lambda functions, processing large satellite image tiles from the European Space Agency and performing predictions on them with a model produced by our Data Science team.
To start, we had two options for testing our Lambda functions:
- Copy + paste our code into the Lambda console and run them in production on live data (Risky and makes me wince 🤨)
- Write extremely verbose tests using Moto (Time consuming, requires knowledge on how AWS works to a low level, and makes test code very noisy 🔈)
Clearly neither of the above are ideal, nor are they particularly geared towards fast feedback & delivery of value. After some Googling, I came across LocalStack; when asked ‘Why Localstack?’ they reply with:
LocalStack builds on existing best-of-breed mocking/testing tools, most notably kinesalite/dynalite and moto. While these tools are awesome (!), they lack functionality for certain use cases. LocalStack combines the tools, makes them interoperable, and adds important missing functionality on top of them
After some discussion it was deemed worth a look and after about a week and a half of experimentation, we have managed to integrate LocalStack into our development pipeline.
LocalStack and pytest running locally
If you would like to view the code in this blog, or indeed run it yourself, you can find it on my GitHub. The code displayed in the blog is via gist just so it looks nice! 💅
To start, you will need Python installed (The repo is written in 3.6.6 but most >3 versions should work). You will also probably want to have some way of creating Python virtual environments, there are plenty of choices but I prefer Pyenv with the Virtualenv plugin.
You will also need to have Docker Compose installed.
Once you’re happy with your environment, install the required packages for the examples. You can do this by running the following in the root of the project:
$ pip3 install -r requirements.txtIt would also be beneficial if you were comfortable with Boto 3.
The basics
To start, lets take a look at the code for the basic Lambda function. We’ll start with lambda.py within lambda/basic_lambda.
As you can see, this is about as simple as it gets. The Lambda will just log a string and return a response with a message.
The main brunt of the code is within testutils.py, this provides a few methods that make deploying a Lambda to LocalStack easier and means that our test file can be nice and succinct.
As you can see above, all the clients we instantiate with Boto3 take in empty values for aws_access_key_id and aws_secret_access_key, this is because LocalStack doesn’t need, nor know them. You merely provide values for the client you want, the region_name and most importantly endpoint_url. This is the major difference between writing applications that talk to AWS vs. LocalStack.
In the situation where you run LocalStack locally, it will expose ports on http://localhost:<service-port>.
Testutils provides methods for creating the .zip you upload to your Lambda, creating your Lambda, invoking your Lambda, and tearing it down.
Finally comes our test file test_lambda.py:
How small is that?! 😱
When you run your test, it creates a package with your Lambda function code, deploys that Lambda, calls the Lambda, verifies its response and cleans up after you.
Lets run it! 🏃♀️
First you will want to open a terminal window and run the following within the root of the repo:
$ mkdir localstack-temp ^ This is due to LocalStack being a bit of pain with /tmp$ TMPDIR=./localstacktemp docker-compose up -d$ docker logs -f localstackAfter a few seconds, your terminal should look like:Starting local dev environment. CTRL-C to quit.
Starting mock S3 (http port 4572)...
Starting mock SNS (http port 4575)...
Starting mock IAM (http port 4593)...
Starting mock API Gateway (http port 4567)...
Starting mock DynamoDB (http port 4569)...
Starting mock Lambda service (http port 4574)...
Starting mock CloudWatch Logs (http port 4586)...
Ready.
Now that you have LocalStack running, open up a new terminal window or a new pane in your multiplexer of choice. Make sure you’re in the root of the repo. It’s time to run our test! 😬
$ cd lambda/basic_lambda
$ pytest -s .
================ test session starts ================
platform darwin -- Python 3.6.6, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /Users/ciaran/dev/localstack_and_pytest_1/lambda/basic_lambda
collected 1 itemtest_lambda.py
Setting up the class
.
Tearing down the class================ 1 passed in 0.32 seconds ================
😎 It passed! You’ve successfully run your first test against LocalStack!
The docker-compose.yml contains an entry LAMBDA_EXECUTOR=docker which tells LocalStack to run each invocation of a Lambda in a separate docker container using lambci. This keeps our Lambdas dependencies separate from our host and gives us a chance to see what CloudWatch show.
To check this, run the following:
$ docker ps -a # You'll want to get the name of the container run with the image: lambci/lambda:20191117-python3.6$ docker logs <the containers name>START RequestId: 8e231e97-8878-4bd1-b84f-eec5f1aedfc7 Version: $LATEST
[INFO] 2020-02-16T12:32:13.951Z 8e231e97-8878-4bd1-b84f-eec5f1aedfc7 I've been called!END RequestId: 8e231e97-8878-4bd1-b84f-eec5f1aedfc7
REPORT RequestId: 8e231e97-8878-4bd1-b84f-eec5f1aedfc7 Duration: 0 ms Billed Duration: 100 ms Memory Size: 1536 MB Max Memory Used: 19 MB{"message": "Hello pytest!"}
Interacting with ‘AWS’ within a Lambda
So, that was cool. But all it did was return some text and log a string.
How about we try and interact with ‘AWS’ within our new LocalStack Lambda?
Take a look at lambda.py in lambda/s3_lambda:
The above is slightly different from our basic one. You’ll notice the method get_s3_client(), this returns us a S3 client based on whether the Lambda is in Production (therefore using the Lambda consoles built in authentication etc.) or anything else, where we’ll provide the values needed for LocalStack.
The handler simply logs another message and then adds a dummy object to the S3 bucket a-bucket which is created within LocalStack by the test file.
Let’s have a look at this Lambas testutils.py:
The above is similar to the basic Lambdas testutils, I’ve added four additional functions: one to retrieve the S3 client, one to create a bucket, one to list the objects within a bucket, and one to delete a bucket).
We also updated the Lambda to add the STAGE environment variable.
Again, the only difference between these functions and interacting with AWS normally are the clients.
Finally, lets take a look at test_lambda.py:
As you can see, we still have a nice clean succinct test file. This test adds one more setup and teardown step, we then invoke the Lambda twice, ignoring its response and asserting that the bucket does indeed contain two items.
$ cd lambda/s3_lambda
$ pytest -s .
================ test session starts ================
platform darwin -- Python 3.6.6, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /Users/ciaran/dev/localstack_and_pytest_1/lambda/s3_lambda
collected 1 itemtest_lambda.py
Setting up the class
.
Tearing down the class================ 1 passed in 0.51 seconds ================
Final thoughts
So, let’s quickly summarise what we covered.
- Running LocalStack locally with Docker Compose
- Writing some very basic AWS Lambda functions
- Using pytest
- Testing the packaging, deployment, and invocation of a Lambda function
- Not spending a penny. ⛔️💰
Hopefully this will have been useful to you, it was certainly a fun week trying to get this stuff working. We’re hoping that we can fully integrate LocalStack into our CI/CD pipeline to streamline our Lambda development.
If you’ve any questions, please don’t hesitate to leave an issue on the repo or message me on Twitter @Ciaran_Evans.
Big thanks to all the folks who contribute to LocalStack, it’s bloomin’ awesome.
