Connecting AWS Lambda Node.JS to Redshift or PostgreSQL? Try AWS Lambda Layers!

AWS Lambda Layers is the best thing that happened to AWS Lambda since… AWS Lambda itself! It solves a problem that has been bugging me for many years: How to call NPM packages without the deployment hassle. And with Lambda Layers it’s really easy to connect our Node.js Lambda Function to Redshift or PostgreSQL.

Let’s say we are writing a Node.js Lambda Function to SELECT some rows from a table in Redshift (or PostgreSQL, since it’s compatible). To do this in plain Node.js we would have to install the node-postgres NPM package…

npm install pg

But how do we run npm install when coding Node.js with the web IDE?

Well we can’t. We need to package all our Node.js files (including the node_modules folder containing node-postgres) into a ZIP file and deploy to Lambda.

Create a Lambda Layer

But that’s before AWS Lambda Layers! Today we can create a “Layer” that contains node-postgres and attach it to our Node.js function. Just enter this into a Windows or macOS command prompt:

mkdir nodePostgres
cd nodePostgres
mkdir nodejs
cd nodejs
npm init
(Keeping pressing Enter to accept all defaults)
npm install --save pg

Open the nodePostgres folder in Windows Explorer or macOS Finder.

Zip up the nodejs folder inside. You should see nodejs.zip inside the nodePostgres folder (check the screenshot).

Launch the AWS Lambda Console, click the menu at top left and select Layers.

Click Create Layer

Set the Name to nodePostgres

Click Upload and select the nodejs.zip file that we have created

Set Compatible Runtimes to Node.js 8.10

Click Create

Create a Lambda Function

Let’s create a Lambda Function to test the Lambda Layer. Copy this sample code from node-postgres that displays the message “Hello World”…

Here’s a more realistic sample: The code below selects one row from my tstatus table in Redshift that contains three fields: id, timestamp, isoccupied.

Using an NPM package in Lambda Layers works the same way as plain Node.js. require(‘pg’) works as expected because the NPM module pg is already packaged in the nodePostgres Lambda Layer. Yay!

Set the Lambda Environment Variables

node-postgres connects to Redshift and PostgreSQL using credentials provided in the Environment Variables. So set the Lambda Function’s Environment Variables accordingly for your database: PGDATABASE, PGHOST, PGPASSWORD, PGPORT, PGUSER. We may also provide the credentials in the Node.js code, as documented here.

For Redshift, the more secure way to connect to the database is to use IAM by calling the getClusterCredentials API. If you would like to see sample code, drop me a comment below!

Attach the Lambda Layer

Now we attach the Lambda Layer to our Lambda Function…

In the Designer section, click Layers

Click Add A Layer

Select nodePostgres, version 1. Click Add

Test the Lambda Function

Save the function and click Test. You should see the output of the SQL query, e.g. “Hello World”. And that’s how we connect Lambda Functions to Redshift and PostgreSQL databases with Lambda Layers… Really easy right?

Pure JavaScript only, please

Could we do the same with other NPM packages, creating Lambda Layers for each NPM package that we wish to attach? With node-postgres we could create the Lambda Layer easily because it’s Pure JavaScript and doesn’t call any functions specific to the operating system (Windows, Linux, …) or hardware platform (x86, ARM, …).

Many NPM packages have dependencies that are operating system and hardware-specific. What happens when we use Windows to add one of these NPM packages into nodejs.zip and call it as a Lambda Layer? It will fail because Lambda Functions are hosted by AWS on Linux, not Windows.

The right way to create a Lambda Layer for such NPM packages: Create nodejs.zip under the proper Lambda Execution Environment, with the same version of Linux used by AWS Lambda. Which becomes a hassle again.

Perhaps someday somebody will figure out a way to publish the popular NPM packages as Lambda Layers. AWS Lambda coding would become so fun and simple!

Lup Yuen Lee 李立源

Written by

Techie and Educator in IoT 物聯網教師

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade