Developing and Testing Amazon Lambda Functions

A Step-by-Step Example — using Kinesis Events — that Speeds Local Development and Improves Code Quality

I am psyched about Amazon’s new Lambda service for asynchronous task processing, but the ideal development and testing cycle is really left to the engineer. While Amazon provides a web-based console, I prefer an approach that uses Mocha. Below you will find the gritty details using Kinesis events as a sample input.

Amazon’s Web-Based Testing Console

Certainly better than nothing, the web-based testing console that Amazon provides gives you an interface to see execution results for Lambda functions given sample events.

Amazon Lambda Edit & Test Console

Of course, this makes for a laborious workflow during the development cycle. Each time you need to test out the execution results of the function, you have to precompile, package, zip, and upload before manually invoking it with a sample event you have copied in.

If there are multiple data scenarios to test, you have to copy those in one by one and review the output manually. Obviously the console is not intended for this.

Local Development and Testing

I found this approach useful for local development and automated testing:

  1. Create development and test environments
  2. Capture real events from Kinesis to use as testing inputs
  3. Simulate Kinesis event creation and Lambda execution
  4. Call the handler function and test outcomes using Mocha
  5. Automate testing with Circle CI

Create Development and Test Environments

If your function requires environment variables, you will need to create these locally. I use dotenv and a local .env file. If the local environment variables are nonsensitive (e.g. they do not include AWS Access Keys or other third party services credentials), you can check this file into source control. Otherwise, add to .gitignore.

In this case of this example, we use Firebase and have different URLs for development, test, staging, and production. These are reflected in our environment files.

Capture Real Events from Kinesis

If you have just started getting your hands dirty with Kinesis it can be useful to capture the specific output that Kinesis events generate. Piping this output to stdout can give you a bit of trouble when you copy from the console, so I also pipe directly to a flat file. Here’s a simple example:

Big thanks to @aj0strow for pointing me to to simplify this script.

If your Kinesis stream is live and generating a high volume of events, you might want to throttle the pipe. In any case, after capturing a few events, you will have an idea of precisely what Lambda will be seeing.

Simulate Kinesis Event Creation and Lambda Execution

Having viewed the live events, it is easy enough to write a test function that reliably encodes a raw payload the way your upstream code does, like so:

The events you capture from the above method will be strings, but when they are sent to Lambda they will be base64 encoded and packaged within an object containing an array of records. For testing, this must be replicated. In addition, if your code is asynchronous, Lambda expects you to call context.done() so that it can exit its own process.

Thankfully, this results in two boilerplate test functions that can be used across projects. Since the context object is called after the Lambda function concludes, it can be useful to create for each test a custom context including the test’s assertions.

In the same vein, we have found it useful to expose a boilerplate handler function within index.js itself:

With this function in place, we can create a private function handleData to do the actual work for each record as a javascript object.

Call the Handler Function and Test Outcomes

With all the Lambda, Kinesis, and Firebase sugar sorted out, tests with Mocha are nothing unique. You can use whichever assertion library you prefer, and each test can call upon different raw payloads of data to ensure that all data scenarios are covered.

We create a custom test context below with our assertions. These are checked after the Lambda function executes.

Once initial local development is complete, it is still a good idea to manually invoke your function against your sample events using Amazon’s web-based console, but provided you are getting the same results at critical milestones, you are probably safe to lean on Mocha for ongoing development.

Automate Testing with Circle CI

Since you have your development and production environments abstracted to environment files (if needed), it is a short hop to creating a test environment and automating the testing process.

We use Circle, while some use Travis or Jenkins. The point is to have a place where your tests will be executed each time code is pushed.

We also have notifications in Slack for pushes to Github, error and resolutions in Circle, and Deployments to AWS. Ultimately, with the simple machinery and workflow above, developing, testing, and deploying Lambda functions is no different than our approach to Ruby or Javascript development, and that’s a good thing.

To see all of the code in the same place, feel free to check out the source code on Github and the test results on Circle. Hope this helps!

This is article 2 of 3 in a series on Amazon Lambda. If you found the article useful, please recommend it and share with others, and perhaps check out the other two:

Thanks so much!