NodeJS Runtime Environment with AWS Lambda Layers

9 min readNov 30, 2018

With AWS re:Invent 2018 underway, AWS has been releasing pretty neat little (and big) features, updates to existing services, and brand new services. Today was serverless day, and of-course with serverless there was a huge focus on Lambda.

What’s New with AWS Lambda?

There’s quite a bit new in the latest updates to Lambda, but here are the major ones-

Lambda Layers
Lambda Runtime API
C++ and Ruby native support
AWS Toolkits for PyCharm, IntelliJ, and VSCode IDEs (preview)
Improved Lambda container startups on a VPC (coming 2019)

You can read more about these features here, here and here. We are going to focus more on the first two and see how to setup Runtime Environment with shared dependencies with NodeJS.

Runtime Environment? What gives…?

So I am going to leave the C++ and Ruby features to the C++ and Ruby experts. But let’s talk about the meat of the matter here — Layers and Runtime API.

Until now, language support in Lambda have been pretty slim with native support for Python, Java, NodeJS, Go, and C# only. With the two top most new features in Lambda a.k.a. Lambda Layers and Lambda Runtime API, Lambda can now virtually support a large variety of programming languages.

Lambda’s containerization may not be new to folks familiar with Lambda functions, but the very stateless nature of every function made data or even code sharing between independent Lambda function elaborate or in some cases impossible. Well, not anymore! With Layers you can now manage and share dependencies, configurations, or even data between multiple Lambda functions. So we are going to look at a typical use case of Lambda functions written in NodeJS with multiple node dependencies.

Use Case

A typical Lambda function with NodeJS and dependent module would look something like this in the Lambda inline code editor-

As you can see, a single Lambda function that may have a few node dependencies will need to have the modules installed locally. You typically do that by -

npm install --save node_module

This would locally install a node_module inside of your Lambda function project directory. You would then need to zip the entire project directory’s content, and upload it to Lambda either via console or using AWS CLI. However, this can quickly get tiresome with growing number of Lambda functions. Repetitive installations of the same node modules for every single Lambda function not only result in redundant work but also hassle upgrading the node modules for every single Lambda function. It is also a huge challenge to maintain a “mono repo” git repository with all your Lambda code, since individual functions would have individual dependencies. In short, until now there wasn’t an elegant way to manage dependencies for ALL your lambda functions. The code size of your Lambda function may also be impacted due to larger dependencies, forcing you to break down a function into multiple functions (although the higher limit of a Lambda function code size is about 50 MB, but it’s nonetheless a limit). Layer intends to solve these issues, and much more.

Managing Dependencies Using Lambda Layers

AWS has made it dead simple (atleast for NodeJS) to create a centralized, dependency manager for your entire Lambda Runtime using Layers. Layers allows you to create a dependency bundle with all the libraries that you may want to be used by all (or most) of your Lambda functions. You can maintain multiple versions of Layers and use different Layer versions for different Lambda functions. You can also manage different permissions for different Layers. But won’t delve much into versioning or permissions in this article. That’s a topic for a different article. A simple use case for a Lambda Layer for dependency management purposes could look something like this-

Lambda-Layer-1 (Version 1)
├── nodejs
      └── package.json
      └── node_modules
              └── axios(version x.x.x)
              └── lodash(version y.y.y)

And

Lambda-Layer-2 (Version 2)
├── nodejs
      └── package.json
      └── node_modules
              └── axios(version x.x.y)
              └── lodash(version y.y.y)
              └── mysql(version z.z.z)

As you can see, the two layers Lambda-Layer-1 and Lambda-Layer-2 contains different node modules and different module versions or new modules. This makes Layers more appealing especially if you are managing Production and Non-Production stages in Lambda. You can now have Layers specific to a particular Stage/Team/Group of Functions and so on. That is HUGE!

Let’s take a look at how to get started.

How to create a Lambda Layer?

Creating a Lambda Layer for NodeJS dependencies is pretty simple.

Create a directory for your Layer (name it whatever you want) —

$ mkdir lambda-layer-test

Create another directory inside it and name it nodejs. Note: the nodejs directory name is not random and must be nodejs. More on environment path’s here.

$ cd lambda-layer-test
$ mkdir nodejs

Once you have the nodejs directory created, initialize npm —

$ cd nodejs
$ npm init

At this point you should have a directory structure like this

lambda-layer-test
├── nodejs

Now it’s time to install all your favorite node modules. So go ahead and install the modules locally.

$ npm install --save node_module1 node_module2 ...

Once the installation is complete you should have something like this.

lambda-layer-test
├── nodejs
      └── package.json
      └── node_modules
              └── node_module1
              └── node_module2

Pretty standard stuff right? and that’s pretty much how to create a dependency bundle really. Now you need to zip up the nodejs directory-

Creating Zip Archive with Node dependencies

Now we are ready to create a Lambda Layer. Log on to your AWS Console, and navigate over to Lambda Console. Click on Layers on the left panel, and then “Create Layer”.

The next section is pretty self explanatory. Give your Layer a name, description and upload your nodejs zip archive file. Make sure you select the appropriate runtime, in our case we selected Node.js 8.10. Runtime is important, since the runtime will dictate the environment path that your Lambda functions will refer to in order to import the dependencies. Once done, click “Create”.

Your Lambda Layer should now be ready to use.

Now it’s time to use your newly created Lambda Layer for NodeJS in a Lambda function. Head back to Lambda Console and click on “Functions” on the left panel and start creating a function from scratch.

You should now see a “new” option below the Lambda function name as “Layers”. Clicking on it reveals a section which lets you add your new Layer. We will do just that —

Click on “Add Layer”. On the next screen, select “Select from list of runtime compatible layers”. NOTE: Your Lambda Node version must match with the Lambda Layer Node Version. Chose your Lambda Layer from the drop down and then chose the “Version”. Click “Add”.

Don’t forget to save your function. That’s all as far as Layer configuration goes, now it’s time to write your Lambda code. There is nothing special about writing NodeJS Lambda function with this setup, with the exception that you can now simply open up the inline editor and start coding and importing the node modules that is included in the Lambda Layer.

Here’s a simple Lambda Function that makes call to an API using axios module that is added in the Lambda Layer.

const axios = require('axios');exports.handler = async (event, context) => {
    const instance = axios.create({baseURL:'http://api.yourdomain.com'});
    try{
        const response = await instance.get(`/get/bookImages`);        
        const images = response.data.images;
        return {code: "200", message: "Success", images};
    }catch(error){
        console.log(`Error obtaining book images - ${error}`);
        return {
            code: "400",
            message: `Error obtaining book images - ${error}`
        };
    }
};

As you can see from the image below, there are no local node modules in the lambda function and I imported the axios module in the Lambda function just the regular way using require('axios').

That’s about it. You can now test your function. This is not just limited to node modules, you can add any piece of NodeJS custom code in the Layer and refer/import it in any Lambda function (speaking of efficient code re-use). Of course, everything you saw above can be done using AWS CLI as well, however certain IAC tools such as Hashicorp’s Terraform or ServerLess Framework are yet to support Lambda Layers, but the speed at which things are going, I won’t be surprised if they already start support for Lambda Layers by the time you stumble upon this article.

Update: IAC tools such as Terraform, Cloudformation, Serverless Framework, and SAM (Serverless Application Model) all support Lambda Layers now. I have personally used Terraform, Cloudformation and SAM in multiple production apps with Lambda Layers and haven’t faced any issues so far. I have not used Serverless Framework personally although their documentation suggests that it also supports Lambda Layers.

Custom re-usable code/data using Layers

So far, we have seen how to use node_modules with Lambda Layers, but what about shared code or data? Maybe you have a common piece of utility function such as utlity.js or maybe you have a JSON file like options.json and you want to be able to include these into the Lambda Layer and be able to use them in all Lambda functions.

In order to do that simply, add your files under the nodejs directory as can be seen in the image below.

Once you have your desired files inside the nodejs directory, zip/archive the directory as explained previously and upload it into a Lambda Layer. Once we have the Lambda Layer assigned to our Lambda function, this is how we would import and use them-

Importing and using the custom (shared) files

According to documentation -

Your function can access the content of the layer during execution in the /opt directory. Layers are applied in the order that's specified, merging any folders with the same name. If the same file appears in multiple layers, the version in the last applied layer is used.

This means the contents of the Layer are accessible using the /opt directory at runtime of the Lambda function, which gives us easy access to any shared piece of code or data that exists in the Lambda Layer. This not only applies to NodeJS, but is also applicable to any other language that is supported by Lambda and it’s corresponding Layer.

What’s Next?

This is barely scratching the surface with Lambda Layers and I am just starting to get my hands dirty with it. There’s quite a bit of documentation available now, and if you are interested checkout this amazing list of Lambda Runtimes by the open source community on Github D̶o̶c̶u̶m̶e̶n̶t̶a̶t̶i̶o̶n̶ ̶i̶s̶ ̶l̶i̶t̶t̶l̶e̶ ̶s̶p̶a̶r̶s̶e̶,̶ ̶b̶u̶t̶ ̶t̶h̶a̶t̶’̶s̶ ̶j̶u̶s̶t̶ ̶a̶s̶ ̶a̶l̶m̶o̶s̶t̶ ̶a̶n̶y̶ ̶o̶t̶h̶e̶r̶ ̶A̶W̶S̶ ̶S̶e̶r̶v̶i̶c̶e̶. I highly recommend playing around with Layers, and as for me I̶ ̶m̶i̶g̶r̶a̶t̶e̶d̶ ̶2̶0̶ ̶s̶o̶m̶e̶t̶h̶i̶n̶g̶ ̶L̶a̶m̶b̶d̶a̶ ̶F̶u̶n̶c̶t̶i̶o̶n̶s̶ ̶o̶n̶ ̶o̶u̶r̶ ̶D̶e̶v̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶ ̶t̶o̶ ̶u̶s̶e̶ ̶L̶a̶y̶e̶r̶s̶ ̶a̶n̶d̶ ̶h̶a̶v̶e̶n̶’̶t̶ ̶f̶a̶c̶e̶d̶ ̶a̶n̶y̶ ̶i̶s̶s̶u̶e̶s̶ ̶s̶o̶ ̶f̶a̶r̶. As of 2020, all of our production as well as non-production Lambda workloads now extensively use Lambda Layers and we have adopted Continuous Deployment with Serverless Application Model (SAM) in developing and deploying Lambda functions at scale.