Connecting AWS Lambda Node.JS to Redshift or PostgreSQL? Try AWS Lambda Layers!
AWS Lambda Layers is the best thing that happened to AWS Lambda since… AWS Lambda itself! It solves a problem that has been bugging me for many years: How to call NPM packages without the deployment hassle. And with Lambda Layers it’s really easy to connect our Node.js Lambda Function to Redshift or PostgreSQL.
Let’s say we are writing a Node.js Lambda Function to
SELECT some rows from a table in Redshift (or PostgreSQL, since it’s compatible). To do this in plain Node.js we would have to install the
node-postgres NPM package…
npm install pg
But how do we run
npm install when coding Node.js with the web IDE?
Well we can’t. We need to package all our Node.js files (including the
node_modules folder containing
node-postgres) into a ZIP file and deploy to Lambda.
Create a Lambda Layer
But that’s before AWS Lambda Layers! Today we can create a “Layer” that contains
node-postgres and attach it to our Node.js function. Just enter this into a Windows or macOS command prompt:
(Keeping pressing Enter to accept all defaults)
npm install --save pg
nodePostgres folder in Windows Explorer or macOS Finder.
Zip up the
nodejs folder inside. You should see
nodejs.zip inside the
nodePostgres folder (check the screenshot).
Launch the AWS Lambda Console, click the ≡ menu at top left and select
Upload and select the
nodejs.zip file that we have created
Compatible Runtimes to
Create a Lambda Function
Let’s create a Lambda Function to test the Lambda Layer. Copy this sample code from
node-postgres that displays the message “
Here’s a more realistic sample: The code below selects one row from my
tstatus table in Redshift that contains three fields:
id, timestamp, isoccupied.
Using an NPM package in Lambda Layers works the same way as plain Node.js.
require(‘pg’) works as expected because the NPM module
pg is already packaged in the
nodePostgres Lambda Layer. Yay!
Set the Lambda Environment Variables
node-postgres connects to Redshift and PostgreSQL using credentials provided in the Environment Variables. So set the Lambda Function’s Environment Variables accordingly for your database:
PGDATABASE, PGHOST, PGPASSWORD, PGPORT, PGUSER. We may also provide the credentials in the Node.js code, as documented here.
Attach the Lambda Layer
Now we attach the Lambda Layer to our Lambda Function…
Designer section, click
Add A Layer
nodePostgres, version 1. Click
Test the Lambda Function
Save the function and click
Test. You should see the output of the SQL query, e.g. “
Hello World”. And that’s how we connect Lambda Functions to Redshift and PostgreSQL databases with Lambda Layers… Really easy right?
Could we do the same with other NPM packages, creating Lambda Layers for each NPM package that we wish to attach? With
Many NPM packages have dependencies that are operating system and hardware-specific. What happens when we use Windows to add one of these NPM packages into
nodejs.zip and call it as a Lambda Layer? It will fail because Lambda Functions are hosted by AWS on Linux, not Windows.
The right way to create a Lambda Layer for such NPM packages: Create
nodejs.zip under the proper Lambda Execution Environment, with the same version of Linux used by AWS Lambda. Which becomes a hassle again.
Perhaps someday somebody will figure out a way to publish the popular NPM packages as Lambda Layers. AWS Lambda coding would become so fun and simple!