Dev IRL: How to ingest Heroku Log Drains with Nodejs on AWS CloudWatch? Part 2: Get the logs

8 min readOct 24, 2022

Getting the logs from Heroku

One way to digest Heroku drains is to add a new HTTPS drain in your app. Simply provide a POST endpoint via the Heroku CLI, and Heroku will begin to send it all the logs generated by your app (from the web and worker processes, or the Heroku router) via their Logplex system. The command is as follows:

heroku drains:add https://myendpoint.com/mydrain -a myapp

It’s worth noting that the Logplex router resides in the US region, regardless of your app’s region. It means that all the informations contained in the logs are processed in the US at some point in time. Keep that in mind if you’re bound by strict compliance requirements, and consider using Private Spaces if that’s the case.

This story is the second of a 6 parts series. You can find all the other parts below:

Creating the endpoint

Create the heroku-drains Lambda function

Keep in mind that every entity on AWS must have the proper permissions to execute or access a resource, defined by a policy. So, while there are several ways to create an API Gateway endpoint, an easy one is to create your Lambda function first and then create the API Gateway as a function trigger. This will automatically add the said permissions for you.

Go to your Lambda dashboard and click on Create function. Select Author from scratch, give a unique name to your function and select your runtime and architecture, Node.js 16.x and arm64 respectively.

From now on, we’ll reference this function as the heroku-drains Lambda function (but you can named it the way you want).

You can test your Lambda function by going in the Test tab and simply click Test. You’ll see a generic “Hello world” message, and the log output.

Please note that besides the function itself, AWS has also created a new CloudWatch Log group dedicated to this function and a new role to access this Log group through a new policy. This is good to know if you want to delete the function and clean all the related resources (AWS won’t do it for you), but also to debug your Lambda function by consulting the Log group.

Create the trigger

Our first goal here is to trigger this function via an API call. To do so:

Click on the Add trigger button
Select the API Gateway source
Select Create a new API
Select HTTP API
Configure security as Open

From here, you should have a working API endpoint in the form https://[TOKEN].execute-api.[YOUR_REGION].amazonaws.com/default/[YOUR_FUNCTION_NAME] that shows the “Hello World” message!

Customizing the endpoint

So far we have a unique endpoint that execute our Lambda function, but remember that we must provide a specific endpoint for each of our Heroku app. Moreover, when the function is executed we must know which app is concerned, to put its logs in the right Log stream. To achieve this, we must customize a part of the endpoint’s URL and retrieve it within the code.

To do so, we’ll use the path variable functionality of API Gateway that’ll contain the Heroku app’s name which is unique, hence a unique endpoint by app:

Go to your API
Go to the Routes section
Click ANY (the protocol of the Route)
Click on Edit
Add /{herokuapp} at the end of your current Route and click Save

⚠️Your attention please⚠️, since the route has changed, the “link” between the Lambda function and the API Gateway trigger is now “broken” and you need to “update” it! Nothing complicated here though:

Go to the Integrations tab of the API
Click on the Lambda function
Click on Edit
Just click on Save, without updating anything

Update the Lambda integration after any changes

If you return to your Lambda function’s triggers, you’ll now see 2 triggers, the old one and the new one (with the path variable). You can safely delete the old one.

At this point, the last section of your API is dynamic, and you can retrieve its content in your Lambda’s Code section. Let’s see how to do it.

Go to your function’s Code tab and set the following code:

Note that all the path variables are stored in the pathParameter property of the handler’s event parameter, so we’ve just stored it the herokuapp const and put it in the response’s body.

Deploy your function by clicking on the Deploy button and go to your endpoint, by adding some string as its last part. You should see it in the message.

🎉 Congrats! You have now a fully customizable endpoint to give to your Heroku app as its HTTPS drain!

Bootstrap your Lambda functions

Like I said in the introduction of this series, for the sake of simplicity I will exclusively use the AWS Lambda code editor to implement the Lambda functions. That said, you can create a repo that you’ll sync with your functions (and this is the recommended way to develop complex Lambda functions within a team). There is an official VSCode extension to help with the process.

But before implementing the function itself, we’ll need some bootstrapping:

Configuration of the function
Tools and packages
Helpers to manage our architecture

Convert the Lambda function as an ES module

To communicate with the various AWS services we’ll use the AWS Javascript SDK (whose V2 comes out-of-the-box in Lambda functions). We’ll also need the heroku-log-parser NPM package to help us with the log parsing and finally a JSON file for the alerts’ configuration.

Among other things, converting our Lambda function into an ES module make the importation of resources easier (trust me on that). Let’s start with declaring our function as an ES module: simply rename your index.js file to index.mjs in the file explorer.

Then, update the declaration of the handler:

Note the slight syntax difference from exports.handler = async (event) => {} to export async function handler(event){} . Deploy the update and check the result via an API call: nothing have changed BUT we have now an ES module!

Importing resources

There will be 3 main kinds of external resources:

built-in packages: like the AWS SDK
custom packages: like NPM packages
static resources: like other ES modules in your environment

To import built-in packages, just import them from the /var/runtime/node_modules path. For the AWS package, import it with:

import AWS from '/var/runtime/node_modules/aws-sdk/lib/aws.js'

To import custom packages, the process is slightly different. You’ll need to create a Layer and upload the said packages as a .zip file, then access them from the /opt/nodejs/node_modules path. Let’s see an example with the heroku-log-parser package:

Open a terminal on your system with nodejs and NPM installed
Create a new folder named after your function, then a folder named nodejs
Run npm install heroku-log-parser --save
Make a .zip of your nodejs folder
Go to the Layers section of your Lambda function and click on Add a layer
See the tiny blue You can also create a layer sentence? Click on it.
Name your layer after your Lambda function
Upload your .zip, choose the same architecture and runtime as your function and click on Create
Return to your Lambda function’s layers and add the new one as a Custom layer

A neat thing about Layers is that you can add or remove them from your Lambda stack, but also make versions of them. So if you have a regression or a bug with a new version of a package, you can easily revert to the previous safe one. Anyway, once added to your function you can import the heroku-log-parser package with:

import herokuLogParser from '/opt/nodejs/node_modules/heroku-log-parser/heroku-log-parser.js';

Last but not least, you can create your own modules and import them in your function with a relative path. Let’s create a simple module that exports a JSON-like object. Right-click in your Lambda function’s environment file explorer, create a new file named error-codes.js next to your index.mjs file and copy-paste the following into it:

You can now simply import this object in a variable with:

import ErrorCodes from './error-codes.js'

Let’s wrap all of it together, and test it! Update your index.mjs like that:

Test your function by clicking the Test button and you should see the herokuLogParser class name, the AWS SDK current version and the JSON-like object in the Execution Results tab! We’ve now all the necessary tools to implement our function.

What have we learned?

How Heroku HTTP drains works, with their Logplex system
How to create AWS Lambda functions
How to trigger a Lambda function with API Gateway
How to convert a Lambda function to an ES module
How to create Layers for Lambda functions
How to import a NPM package in Lambda functions

Now, go to the next part: Handle the drains 🚀