Dev IRL: How to ingest Heroku Log Drains with Nodejs on AWS CloudWatch? Part 3: Handle the drains

8 min readOct 24, 2022

Creating Helpers to build our architecture

Do you remember that we need a specific SQS and Log stream for each of our app? If not, got to the Part 1: Architecture 😉

To have them, we’ll need to create them before adding the new HTTPS drain Heroku-side so when the logs come, everything should be ready to ingest them. To do that we can use our heroku-drains Lambda function, but as you can imagine it’ll begin to handle different tasks (at least two: creating and ingesting). And if you create something, you should also be able to delete it. And to list it. In other words Create, Read, Update and Delete items so yes, CRUD operations.

This story is the third of a 6 parts series. You can find all the other parts below:

Handle the HTTP methods

We can take advantage of the API Gateway’s HTTP methods to implement a kind of RESTful API. I mean “kind of” because the POST HTTP method will already be used by Heroku to send us the logs.

All in all, we could call our API with:

GET: to list the drains
POST: to ingest the logs
PUT: to create a new drain
DELETE: to delete a drain

By default, the API can be accessed with any kind of method (remember the ANY keyword in the Route’s details?) so the trick here will be to detect the method within our function. Let’s see how to do that:

Detect and handle the HTTP method

As you can see, you can simply get the HTTP method in the event.requestContext.httpMethod property, and execute different actions depending on that. In this example, we just return the method used along the app’s name as JSON response. You can test the various method with a tool like Postman.

We can now build our helpers to manage the SQS queues and Log streams! Just a quick note regarding the order of the methods, I test the POST first as it’ll be the most used by the Heroku drain.

List the drains

Let’s define that, in our context, a drain is composed by:

a Log stream
a SQS queue
a Queue event source (the “link” between a queue and our Lambda function)

But remember one thing: a Log stream resides in a Log group. So we must first create a Log group and then give our function the permissions to create, read and delete Log streams in that Log group.

Quick tip here: keep in mind that every action you make with an AWS API call must be granted through your Lambda function policy. So every time you use a new API endpoint or manipulate a new resource, you must update your policy. This is a best practice as you must give only the minimum permissions to an entity.

Go to your CloudWatch dashboard, create a new Log group and set its retention setting to 7 days (you’ll be able to tune this setting later if you need to).

Now go to your heroku-drains Lambda function’s Configuration > Permissions tab and click on its role name. Here, you can see the current policy applied to your function. We’ll add some others to manage our new Log group. Click on Edit and select the JSON editor (or use the visual editor if you’re more comfortable with).

Add the following statements (be aware that the names could change and I obfuscated the IDs):

{
    "Effect": "Allow",
    "Action": [
        "logs:CreateLogStream",
        "logs:DescribeLogStreams",
        "logs:DeleteLogStream",
        "logs:PutLogEvents"
    ],
    "Resource": [
        "arn:aws:logs:eu-west-1:XXXXXXXX:log-group:/heroku/apps-test:*"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "sqs:DeleteQueue",
        "sqs:SendMessage",
        "sqs:GetQueueAttributes",
        "sqs:CreateQueue"
    ],
    "Resource": [
        "arn:aws:sqs:eu-west-1:XXXXXXXX:*"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "sqs:ListQueues",
        "lambda:ListEventSourceMappings",
        "lambda:CreateEventSourceMapping"
    ],
    "Resource": [
        "*"
    ]
}

Okay, now we can use the AWS APIs to handle our Log streams, queues and queue event sources. One thing is missing though: we have created a Log group, but how could we access its name? It would be a bad idea to hardcode it in the Lambda function since it would turn the function specific to this very log group.

Instead, we could take advantage of the Lambda function environment variables! This will grant us the power of customization without updating the function’s code. Go to the Configuration > Environment variables tab of the function and click on Edit. Add a variable (e.g. LOG_GROUP_NAME) and set the Log group name. The variable will be exposed in the process.env.LOG_GROUP_NAME property within the function.

Getting a drain’s ressources

We’ll write a few helper functions to help us getting the resources. We’ll declare and define them outside the main even handler method though, to reuse them at each function invocation instead of recreating them. Typically, you must put this above the export async function handler(event){ statement:

Okay, that’s a lot of helpers 😅 But believe me, its worth the effort! The various functions are pretty self-explanatory, so I won’t get into the details.

Just one thing though: keep in mind that the listQueues , describeLogStreams or listEventSourceMappings are search functions. This is the reason why we must get the result that is an exact match of the search params.

We can now call getDrain when our function is called with a GET method, and return the result as a JSON object:

You should see an empty data result as there is no drains created at the moment, but this is intended. Note that I’ve added prefixes to the Log stream and queue names, respectively [APP] and APP_ , to tag them as “belonging to an app” and easily identify or search for them in the AWS console.

Creating a new drain

Okay let’s get serious now! We can write the helper function to create a new drain, again outside the event handler of the Lambda function:

Note that:

we check the existence of each resource before creating them to ensure their unicity.
this is during the creation phase that we set several parameters, like the MaximumBatchingWindowInSecond or BatchSize ones we talked about in the Part 1: Architecture, that will take care of the invocation rate.
the queue event source is referencing the Lambda function dedicated to store the logs, whose name is stored in a new environment variable called DRAIN_STORAGE_FUNCTION. We need to create this function now to avoid an InvalidParameterValueException.

From now on, we’ll reference this function as the heroku-drains-storage Lambda function. Of course you can name it as you want, so go to your heroku-drains function’s Configuration > Environment variables tab and add this name to a new DRAIN_STORAGE_FUNCTION variable.

A basic Lambda setup for now

As we only need a valid reference to create the queue event source, we can just create a basic setup for our heroku-drains-storage Lambda function that will take care of the log storage itself. Go to your Lambda dashboard and:

create a new function, with the same name as your createEventSourceMapping 's FunctionName parameter.
choose Node.js 16.x as your runtime and arm64 as your architecture
rename your index.js file as index.mjs to convert it to an ES module
update your code to log the incoming event and inform the SQS queue that everything went OK (i.e. no batch failures occurs) as below:

As we discussed earlier, creating a new resource means updating its permissions! Here, our function will need to receive messages from SQS and obviously put logs in the app’s dedicated Log group. Go to the Configuration > Permissions tab of your function and click on the role’s name. Add the following statements:

{
    "Effect": "Allow",
    "Action": [
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage",
        "sqs:GetQueueAttributes"
    ],
    "Resource": [
        "arn:aws:sqs:eu-west-1:XXXXXXXX:*"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "logs:DescribeLogStreams",
        "logs:PutLogEvents"
    ],
    "Resource": [
        "arn:aws:logs:eu-west-1:XXXXXXXXXX:log-group:/heroku/apps-test:log-stream:*"
    ]
}

Again, beware the names of your ressources and your IDs!

Test the endpoint

Now, let’s call createDrain when our heroku-drains function is triggered with a PUT method:

Note the prefixes I previously mentioned, this is where they are created 😉 So, after a PUT call you should see the API responding with the newly created resources (at least the queue and the queue event source, as the createLogStream method returns a empty body).

Now if you call the function with a GET method, you should see the Log stream and queue infos of your app. You can also check your various resources via the AWS console:

a new Log stream in the app’s Log group
a new SQS queue
a new SQS trigger on the heroku-drains-storage Lambda function that will store the logs

The resources created for this new drain.

Delete a drain

The last step is to delete a drain. Let’s define our helper:

Again, this is pretty self-explanatory, we just ensure that the resource exists before deleting it. Now for the invocation during a DELETE call:

Try it and you should be granted a success message 🎉

A CLI interface

Wouldn’t it be nice if all of those CRUD operations could be executed by a single command line? Of course it would!

So the goal here is to write a small bash script with several options to help us list, create and delete log drains. We’ll assume that nodejs, NPM and the Heroku CLI are globally available in your system since, well, you’re likely to have those if you develop nodejs apps on Heroku 😄

I won’t go into the implementation details of this script (and it could be optimized for sure), but here it is:

As a reminder, what we call a “drain” here is:

a Heroku HTTP drain subscription
an app’s specific SQS queue
a Log stream dedicated to an app

So all you have to do is updating the BASE_URL according to your own API Gateway endpoint, and execute:

sh heroku-drains-cli -a <myapp> --get : to get infos about this drain
sh heroku-drains-cli -a <myapp> --put : to create a new drain
sh heroku-drains-cli -a <myapp> --delete : to delete a new drain
sh heroku-drains-cli -a <myapp> --help : to show the CLI usage

With these commands, both the Heroku and AWS sides are handled.

What have we learned?

How to add environment variables to a Lambda function
How to manage Lambda functions policies to give them proper permissions
How to use different HTTP methods with the same Lambda function
How to write a little bash script to handle the drains

Now, go to the next part: Sending events with SQS 🚀