Running AWS Lambda and API Gateway locally: serverless-offline

On previous posts we talked about how to build unit tests for our Serverless application. While unit testing works great for TDD (Test Driven Development), in some moment we will want to run our Lambda application locally integrating it with the API Gateway.

The Serverless Framework offers a lot of good plugins. One of them gives us exactly what we need: serverless-offline.

This plugin leverages Docker containers to build a local environment where a simulated environment will be created, exposing the same REST endpoints the API Gateway would expose for our Lambdas in AWS.

The Serverless plugin takes advantage of Docker volumes, which don’t play well with the way Windows uses system paths. When I tried to run this plugin in Windows 10 I got errors because of this. If you plan to use serverless-offline, try a Linux distribution like Ubuntu. There is no mention of this fact in the plugin documentation.

Edit: Probably I did something wrong in my first tests. After I completed this post I went back to test it in Windows and now it works.

Now, in the last post we also used Dynalite to create local a DynamoDB server for our unit tests. However, the serverless-offline plugin plays well with another Serverless plugin, serverless-dynamodb-local. If we add both plugins, serverless-offline will seamless integrate with serverless-dynamodb-local and will create database instances based on our definition file serverless.yml.

This integration is great for us, since we will have to define our database in that file when we want to deploy our application to AWS.

Let’s use our trusted taco-gallery project, which we created in previous posts, and install the plugins:

npm install serverless-offline serverless-dynamodb-local --save-dev

The Serverless Framework will need us to configure access to AWS. This can be accomplished by running

sls config credentials --provider aws --key KEY --secret SECRET

Where KEY and SECRET are our AWS Key and secret key. We are not deploying to AWS, but the serverless plugin needs this configuration to exist in order to work correctly.

Let’s open our serverless.yml file and add these lines at the end:

plugins:
  - serverless-dynamodb-local
  - serverless-offline

The order is important, per serverless-dynamodb-local documentation.

We need to setup our Docker environment. Make sure you install Docker in your local computer, and then run the following command:

docker pull lambci/lambda

This will pull the Docker image required for serverless-offline to work correctly.

Next, let’s define the DynamoDB we are going to use for the application. Remember that this definition can also be used to create an instance of the database in AWS.

Using the same DB definition we setup for our unit tests, let’s add the following to serverless.yml:

resources:
  Resources:
    TacosDB:
      Type: 'AWS::DynamoDB::Table'
      Properties:
        AttributeDefinitions:
          -
            AttributeName: id
            AttributeType: S
          -
            AttributeName: name
            AttributeType: S
        KeySchema:
          -
            AttributeName: id
            KeyType: HASH
          -
            AttributeName: name
            KeyType: RANGE
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        TableName: TacoGallery

This definition will create a CloudFormation script, will will serve as the basis for our infrastructure.

Just like we did for our unit tests, we create a table “TacoGallery”, with two keys: id as a HASH key, and name as a RANGE key. This presentation provides a good explanation on using RANGE keys.

Run the following command to initialize the DynamoDB plugin:

sls dynamodb install

When we execute this plugin, it will start a server running in localhost.

sls dynamodb start -p 8000  --migrate true

The “migrate” option will take care of creating the database we defined in serverless.yml.

Make sure DynamoDB is running in the port 8000, because serverless-offline will try to use that port by default.

Keep this console running.

The serverless-dynamodb-local plugin allows you to check the database contents through a web application which will start at http://localhost:8000/shell. This application can also be useful to play around with DynamoDB and learn more about its syntax

The problem here is that the AWS SDK will require configuration in our code to indicate we are using our local server. We want to be able to deploy the same code we run locally to AWS. ANode package serverless-dynamodb-client takes care of detecting if we are running our server locally or in AWS, and will configure accordingly.

Install it with:

npm install --save serverless-dynamodb-client

And, instead of using the AWS SKD to create a DocumentClient in our application, use this plugin:

var dynamodb = require('serverless-dynamodb-client');

class TacoGallery{
   constructor(){
      this.db = dynamodb.doc;

This will be enough to configure a local DynamoDB development environment.

Let’s update our serverless.yml configuration file to create our REST endpoint:

functions:
  saveTaco:
    handler: handler.saveTaco
    events:
      - http:
          path: taco
          method: post

Serverless will take care of starting the API Gateway which will expose our Lambda, both in our local environment and in AWS.

Also, since we are using Babel for ES5 features like classes, we need to add this to the configuration file too:

custom:
  serverless-offline:
    babelOptions:
      presets: ["es2015"]

The next step is to start the serverless-offline plugin:

sls offline start -r us-east-1 --noTimeout

We will see the following in our console:

> taco-gallery@1.0.0 start /home/pedro/Dev/Serverless/taco-gallery
> sls offline start -r us-east-1 --noTimeout
Serverless: Starting Offline: dev/us-east-1.
Serverless: Routes for saveTaco:
Serverless: POST /taco
Serverless: Offline listening on http://localhost:3000

Our Serverless application is now running in our local environment!

We can test it with a curl command:

curl -d '{"name":"Al pastor","description":"A good taco"}' -H "Content-Type: application/json" -X POST http://localhost:3000/taco

And you should see a response in the console running Serverless:

Serverless: POST /taco (λ: saveTaco)
Serverless: The first request might take a few extra seconds
Serverless: [200] {"statusCode":200,"body":"{}"}

We just used our Serverless application to save a record to our local DynamoDB.

Now, to keep track of all the commands we have run, let’s add them to package.json:

"scripts": {
  "install": "docker pull lambci/lambda",
  "setup": "./node_modules/.bin/sls config credentials --provider aws --key KEY --secret SECRET",
  "start": "./node_modules/.bin/sls offline start -r us-east-1 --noTimeout",
  "install:dynamodb": "./node_modules/.bin/sls dynamodb install",
  "start:dynamodb": "./node_modules/.bin/sls dynamodb start -p 8000  --migrate true"
},

So, let’s review everything we changed.

Here is our serverless.yml file:

Here is our TacoGallery.js:

And our Node’s package.json:

The serverless-offline plugin is a different approach from what we have discussed before; it gives us a lot of benefits:

• Test your serverless application locally, including the API Gateway integration. This helps you to build clients locally without having to deploy to AWS
• Running DynamoDB locally will allow us to debug out application without costs caused by traffic or storage.
• Since it uses serverless.yml to define resources, we can make sure that, whatever we deploy to AWS will be tested locally first.

The downside of this approach is:

• The plugin requires Docker and be run in a computer which plays well with Docker volumes. If you use Windows as your development machine, this method might not be useful to you.
• The need of having a Docker environment can be an obstacle too if, for building your project, you use a Continuous Integration server where you can’t install Docker at will. A mixed approach of these plugins and Dynamolite can be a good solution for this challenge.
• If you want to be rigorous with your TDD approach, using this plugin can make it easy for you to not write all the tests you need for your app. You have to be strict with yourself to make sure your tests cover your project correctly.

You can find the source code in my Github repo (just be advised that it will change as I will continue to use this project in future posts): https://github.com/pfernandom/taco-gallery