Fixing Node.js CloudWatch Streaming
A workaround for a sudden stop in streaming of node.js logs from AWS’ ElasticBeanstalk / EC2 to CloudWatch
I’ve been enjoying a fun bug recently. Our ElasticBeanstalk instances just stop streaming their node.js logs after a while (typically three days). It’s not been a massive issue, as our delivery is continuous enough that an instance doesn’t typically last that long. However, it does mysteriously seem to mean that whenever I want to check the logs in CloudWatch, they’re not there; which is especially annoying for historic instances- as they are apparently burned after reading.
In the end I got annoyed enough to brave the song-and-dance that is AWS Support. And, after a bit of backing-and-forthing and a few bewildering missteps, they revealed that this is, in fact, a known issue- and there’s a workaround. So today I learned how to stop your node.js log stream to CloudWatch just casually stopping.
Like, seemingly, all my solutions of late- it involves adding the following file to your project’s .ebextensions folder:
If you don’t know what an .ebextensions folder is, you can find out a bit more about it here. In essence this file replaces the hourly log rotation script for the node.js log files with a new one; revealing the underlying reason for the issue in the first place.
One of the most embarrassing things that can happen to a sysadmin is that their server fills up with logs. To reduce the chances of this ultimate shame, linux distros typically come with a utility called logrotate. Its whole purpose in life is to ‘rotate-out’ logs when they get to a certain size and/or a age. It then stores a limited history of these rotated-out logs; ensuring that even the most verbose of applications can only take-up a fixed amount of space on the drive.
It turns out that the awslogs service is not very tolerant of these log rotations; appearing to maintain a reference to the rotated out file, rather than the new, latest, log. This replacement version of the cron.logrotate.elasticbeanstalk.nodejs.conf script adds a line that restarts the service after each rotate- so that it once again monitors the latest node.js log file. A ‘turn-it-off-and-turn-it-on-again’ fix, if you will.
AWS haven’t given a timeline for getting a fix out; but the workaround will work, for now.
