Fixing Node.js CloudWatch Streaming

A workaround for a sudden stop in streaming of node.js logs from AWS’ ElasticBeanstalk / EC2 to CloudWatch

I’ve been enjoying a fun bug recently. Our ElasticBeanstalk instances just stop streaming their node.js logs after a while (typically three days). It’s not been a massive issue, as our delivery is continuous enough that an instance doesn’t typically last that long. However, it does mysteriously seem to mean that whenever I want to check the logs in CloudWatch, they’re not there; which is especially annoying for historic instances- as they are apparently burned after reading.

In the end I got annoyed enough to brave the song-and-dance that is AWS Support. And, after a bit of backing-and-forthing and a few bewildering missteps, they revealed that this is, in fact, a known issue- and there’s a workaround. So today I learned how to stop your node.js log stream to CloudWatch just casually stopping.

Like, seemingly, all my solutions of late- it involves adding the following file to your project’s .ebextensions folder:

If you don’t know what an .ebextensions folder is, you can find out a bit more about it here. In essence this file replaces the hourly log rotation script for the node.js log files with a new one; revealing the underlying reason for the issue in the first place.

One of the most embarrassing things that can happen to a sysadmin is that their server fills up with logs. To reduce the chances of this ultimate shame, linux distros typically come with a utility called logrotate. Its whole purpose in life is to ‘rotate-out’ logs when they get to a certain size and/or a age. It then stores a limited history of these rotated-out logs; ensuring that even the most verbose of applications can only take-up a fixed amount of space on the drive.

It turns out that the awslogs service is not very tolerant of these log rotations; appearing to maintain a reference to the rotated out file, rather than the new, latest, log. This replacement version of the cron.logrotate.elasticbeanstalk.nodejs.conf script adds a line that restarts the service after each rotate- so that it once again monitors the latest node.js log file. A ‘turn-it-off-and-turn-it-on-again’ fix, if you will.

AWS haven’t given a timeline for getting a fix out; but the workaround will work, for now.

tomincode

thomas michael wallace

Written by

writes about the stuff he does in code.

tomincode

tomincode

writing daily lessons on the stuff tom does in code.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade