Dev IRL: How to ingest Heroku Log Drains with Nodejs on AWS CloudWatch? Part 4: Sending events with SQS

Photo by Mehdi Sepehri on Unsplash

In the previous parts of this series we’ve achieved the backbone of our architecture:

• we can manage our drains (through CRUD-like operations)
• we can scale our SQS queues (one for each app)
• we can put our app’s logs in specific Log streams (for easier analysis)

The last step would be the alert system, but let’s focus on the log’s ingestion for now.

This story is the forth of a 6 parts series. You can find all the other parts below:

• Part 1: Architecture
• Part 2: Get the logs
• Part 3: Handle the drains
• Part 4: Sending events with SQS <<< 📍You are here!
• Part 5: Ingest the logs
• Part 6: The alert system
• Bonus Part: SpeedRun

Sending messages to a queue

First, we’ll ensure that we can send some events to our heroku-drains-storage Lambda function with a POST call to our API. Let’s write a helper for that, again outside the heroku-drains Lambda function event handler, along the other helpers:

Nothing too complex here, just take a look at line 7: the QueueUrl is based on a new environment variable, called SQS_BASE_URL. This URL’s pattern is https://sqs.[YOUR_REGION].amazonaws.com/[YOUR_AWS_ID]/ (e.g. https://sqs.eu-west-1.amazonaws.com/1234567890/) and since it contains your AWS ID, its always a good idea to store it in a safe manner. So go to your heroku-drains function’s Configuration > Environment variables tab and put your own SQS base URL in a new SQS_BASE_URL variable.

Let’s call this sendEventsToQueue helper when you hit your API with a POST method:

So if you call your API with a POST method, you’ll see nothing but a 200 status: that’s expected as this is a requirement from Heroku Logplex that does not support chunked transfer encoding of responses, hence no body in the response and a Content-Length equals to 0.

The interesting thing though is to check the logs of the heroku-drains-storage Lambda function triggered by the SQS queue. Go to your CloudWatch dashboard and select the Log group dedicated to this Lambda function. Here you should see the logs generated by the invocation, and more specifically the one generated by the console.log('Records:’,event.Records); statement.

heroku-drains-storage Log group

You should see the payload you sent through the SQS message, i.e. all you need to store a new log entry in CloudWatch:

• the Log group’s name
• the Log stream’s name
• the log content

Parsing the logs

Let’s go back to the way we handle the POST method: we just have generated a dummy events array for now, like const events = [{"name":"event1"},{"name":"event2"}];. It’s time to parse a real Heroku Logplex event!

We’ll assume a couple of things:

• you have a running Heroku application that generate some logs
• your application is using the morgan middleware (this is not mandatory, but it helps a lot for the log parsing)

Next, we’ll need some Regular Expressions and update the way we handle POST requests. Add the following RegExp just before the event handler of the Lambda function:

And update the way we handle the POST requests like this:

I won’t go into the details of the parsing (as it is specific to my needs) but as you can see the herokuLogParser does the heavy lifting, and I take advantage of the morgan middleware’s log structure to get most of the informations.

Finally, add a new drain on the Heroku app with the Heroku CLI. Double-check:

• your API endpoint URL (I obfuscated mine)
• the last part of your API (you can use the one you previously checked with the dummy event)
• your Heroku appname (provided as the -a option)

Execute the following command in your terminal:

heroku drains:add https://xxx.execute-api.eu-west-1.amazonaws.com/default/heroku-drains-test/myapp -a my-heroku-app

BTW, you can check your app’s current drains with the command:

heroku drains --json --app my-heroku-app

You can now generate some logs from your app (e.g. visit it with your browser) and you should see a Heroku formatted event in your heroku-drains-storage Lambda function logs instead of the dummy one!

Troubleshooting your process so far

You finally get a formatted log entry in your heroku-drains-storage Lambda function, ready to be stored in CloudWatch! 🎉

Or not?

If you don’t see any logs in your storage Lambda function, check the following:

• Is your Heroku app set with the correct drain ? Check it with the command heroku drains --json --app my-heroku-app.
• Is your drain created AWS-side? Call your API endpoint with a GET method and check the log_stream , queue and queue_event_source properties, and check all your resources in the AWS dashboard.
• Is your API endpoint responding as it should? Call it with a POST method.
• Is one of your Lambda functions shows any errors? Check their logs in their dedicated Log group.
• Have you a permission related error? Check your Lambda’s role permissions.

What have we learned?

• How to trigger a Lambda function with a SQS queue message
• How to parse a log
• How to add a Heroku HTTP drain with the Heroku CLI

Everything should be alright now, so let’s dive in the log storage process with part 5: Ingest the logs 🚀