PM2 and memory usage in Node.JS Apps

Real htop shot from a production server running our WebCache

One of the projects I'm working on required a Redis+Node.JS WebCache for storing products' available stock and prices. Retrieving this information was quite slow, so caching this information was a great improvement in the performance of the system.

The problem: We needed a WebCache for several clients, each one of them with different server configurations, sharing the same Redis server. Some clients generated lots of traffic over the server, that meant more memory usage.

We decided to run the node app using PM2. PM2 somehow ensures that an app keeps running, IMHO far better than forever, and far easier than a Linux service, with more management options thru the command line.

I'll keep talking about the WebCache when referring to the node app, but you can make an abstraction inside your mind and change it to your own node app :) .
To run a WebCache using PM2 we use the following command (inside the WebCache folder):

pm2 start index.js --env production --name WebCache.apple -c "0 0 * * *" --max-memory-restart 850M --watch

(just for the record, it's a real PITA to make double dashes when posting a Medium article, had to type _-_- and delete the _ )

Explanation:

  • --env production: sets the environment variable to production
  • --name WebCache.apple: sets this instance of the app to be called WebCache.apple (that's WebCache.<company name>). As we have many companies using a WebCache, a name is required to identify each of them (for restarting and stuff).
  • -c "0 0 * * *": restarts the app at 00:00 hrs. every day. It uses a cron formatted string.
  • --max-memory-restart 850M: auto-restarts the app if its memory usage exceeds 850 megabytes
  • --watch: restarts the app if it detects any change on any files related to the app (as require libraries, or other imported scripts).
    BEWARE: If for some reason you're using some log library that writes log files inside the app folder (or any child folder), this option will restart your app on EVERY file change. Not good… My advice, if using PM2 don't log to files, log to console as PM2 will listen to the log stream and make it's own log to file. By the way don't use console.log there are better libraries for logging (my personal favorite: Winston). If you have to ignore any files on your watch take a look at the --ignore-watch="folder folder2 etc" attribute.

After a few clients this worked great, but as we included new companies the memory usage also increased. And as it did, the app performance for some companies decreased in about a 20% for each request (you know, processes racing for memory resources). After running htop we noticed our server's memory usage was almost at 90%, and it went back "to normal" after restarting all running PM2 processes with: pm2 restart all

So I made a little bash script (heavily inspired from here) and made it run as a cron job every 10 minutes:

This frees the memory, performance goes back to the expected times per request, but… it doesn't fix the underlying problem. The WebCache was created to hold a platform that works on legacy software, where integrations with external software (such as a REST API that makes certain requests to retrieve information) are resource expensive operations.

The moral of this story:

  1. Even if you use PM2 with memory limit parameters, sometimes you will need an external memory monitor that frees memory usage within certain range.
  2. Poorly coded legacy software always leads to developer's headaches.

Hope you enjoyed this article. Clap or follow if you like, and if you have any other tips regarding PM2 or memory usage share them in the comments.

Jose I Santa Cruz G

Written by

Polyglot software engineer, amateur guitar & bass player, geek, husband. Not precisely in that order.