Monitor your Heroku app in Datadog

Timothy Carbone
Dec 17, 2018 · 3 min read

At Unsplash, we love Heroku but sometimes, it’s not easy to monitor whatever happens in your Heroku app.

Image for post
Image for post
Photo by rawpixel on Unsplash

Most of our dev-ops monitoring happens in Datadog. Unfortunately, there’s not an easy and clear way to serve Heroku data points to Datadog for monitoring. These data points being measurements like dyno memory usage, database load or even in-app measurement like query times etc…

A while ago, we found a lonely git repo that creates that missing bridge between Heroku and Datadog pretty neatly by parsing Heroku logs and sending data to Datadog accordingly.

So we forked it and quickly hacked around it to do some of the things we needed it to do… and that’s it. We started collecting Heroku data points in Datadog, monitoring our web and worker dynos, our Heroku Postgres databases and a couple of custom metrics like the execution time of our ETL scripts.

Today, as we’re adding a new metric to measure, we thought it’d be worth refactoring the code and making it so that anyone can build their own Heroku-Datadog logs parser in NodeJS, easily.

There you go:

By setting up a log drain in Heroku towards the logs parsing app, whenever Heroku logs something like a web request, a database or dyno status update, an error or a console log from your application, the log will get pushed to your logs parsing app.

Your parsing app will be able to read every single log line and to react accordingly. By default, the app parses the dyno metrics logs and the Heroku Postgres database metrics logs.

The interesting bit is that you’re absolutely free to add your own custom parsing function to parse Heroku logs and trigger custom Datadog measurements.

Here’s an example of a custom log parser we built for ourselves:

Whenever one of our ETL scripts ends, we log this in the console (Javascript):

console.log(‘ETL job job:name done in 2 mins and 20 secs’)

This makes our worker app write a log that will be drained and read by our log parsing app. Our custom parsing function reads the line, extracts the task name and its duration and sends the the measurement to Datadog.

Image for post
Image for post
Scripts execution times charted in Datadog

If you want to build your own custom log parsing, create a parser in app/parsers using the runtime.js example in the project.

You can easily see that with this technique, you can pretty much get in Datadog anything that’s important to you and happening in Heroku:

  • Log anything in your Heroku application
  • Parse the log in your parsing app and trigger Datadog measurements

You could log execution times, CPU usage measurements, database request results, database query times … pretty much anything that you’re able to code, plus some out of the box logs from Heroku.

You can host your app in Heroku. Follow the directions in the README:

You can check out this repo that achieves the same goal by formatting your logs rather than writing parsers. It’s a different approach that you may or may not be more comfortable with.

Unsplash Blog

Behind the scenes building the open photography movement at…

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store