Solving server-side memory leaks on Nuxt.js
At Le Collectionist, we rent the most beautiful houses with exceptional services and memorable experiences. Our public website is one of our main showcase for our customers. We built it with Nuxt.js and Vue.js on the front-end, and with a Ruby on Rails REST API. It is hosted on Heroku.
A few months ago, we encountered memory leaks on our production website, which started to crash every few hours. This was followed by a week of relentless research to discover what created the leak, and how we could fix it.
In this article, you’ll find our historic with server-side memory leaks on Nuxt.js, as well as the process we established to detect them and fix them.
TL;DR — Finding server-side memory leaks on Nuxt.js
- Do a git bisect to find the commit that created the leak
2. At each step of your bisect:
- Launch your Node server in debug mode
`node — inspect node_modules/nuxt/.bin/nuxt start` - Take a first memory heap snapshot of your server with Chrome devtools
- Simulate user traffic and take a second snapshot to know if there’s a leak with this commit
3. At the end you should know which commit created the leak. For us it was always linked with a new plugin or an upgrade, and we either found a fix by googling “Memory leak <name of plugin>” or going back to the former version of the plugin
How we first encountered memory leaks
Let’s get back to our first big leak. We started receiving notifications that the website crashed. We took a look at the app’s memory usage, and it kept going up until it crashed at 2GB every few hours.
And thus every few hours, the site displayed this magnificent message to our customers:
Our first reaction was to google “Nuxt.js memory leaks”, and we found several leads, essentially linked with some of our plugins. Turns out, if you google “<plugin name> memory leak”, you’ll always find something.
However, we didn’t know at the time how to properly check for memory leaks on the server side on our local environment. So we tried to remove some plugins, to go back in our Git history, but it seemed like we had leaks all the time.
We tried following the Nuxt.js protocol for memory leaks described in this github issue, but it didn’t help, maybe because the issue was not coming from Nuxt itself.
After reading many articles to understand how to detect memory leaks in Nuxt.js and Node.js, we found out that we weren’t testing the leaks correctly on our local. So all our tests had been in vain…
Check for leaks on local server
Here’s what we do now to test properly if we have memory leaks on a local server.
To detect memory leaks, we use the Memory tool from Chrome devtools to record and compare heap snapshots. A heap snapshot shows the memory used in your app at a given moment.
We can’t take snapshots directly when we inspect our local nuxt server on localhost:3000, because it would analyze the client side. We want to find memory leaks that are on the server side, so it’s a bit more tedious to test, and that’s what we got wrong in the first place.
1. Setup
We’re going to use a tool called siege to simulate user traffic.
In order to set the siege for a limited time period, we use gtimeout on Mac (provided by coreutils)
For Mac users:
brew install siege
brew install coreutils2. Build & start in inspect mode
You have to build your Nuxt.js project and start it in inspect mode in order to open node.js debugger. That’s how you can check if you have leaks on the server side
To do so, we created 2 scripts in our package.json, build-dev and prod-debug.
First, we build our project in dev mode, in order to access the right local environment variables:
npm run build-dev
# alias for `cross-env NODE_ENV=dev nuxt build`You might not need it if you don’t have any trouble with your environment variables, and use the regular npm run build
Then we start the server and enable Node inspector:
npm run prod-debug
# alias for `node — inspect node_modules/nuxt/.bin/nuxt start`3. Take heap snapshots
Inspect on Chrome
Go to chrome://inspect on Chrome.
Your local server should appear below Remote Target. If not, check the debugger port on your terminal (should be 9229 by default), click on Configure button and add localhost:9229.
Click on the inspect link below your target, it will open a regular Chrome inspector on your node server.
Go to the Memory tab and take a first heap snapshot.
Launch a siege
Now create a siege in a new terminal, for example:
gtimeout 30 siege -c 20 http://localhost:3000/=> it simulates 20 concurrent users during 30 seconds on localhost:3000
Wait for the end of the siege and take a second snapshot.
Compare Heap snapshots
You can now compare the snapshots sizes. If there’s a memory leak, the difference should be really big, for instance from 50MB to 130MB.
If you’re not sure, you can try again:
- Clean the memory by clicking on the bin icon.
- Take a new snapshot.
- Launch a new siege. You can also put a timeout of 60s if you want.
- Take another snapshot at the end.
If the snapshots sizes are similar, you don’t have a leak 🎉
/!\ Each snapshot takes twice as much size on your computer. You can clean the memory by clicking on the bin icon.
4. Find what caused the memory leak
At first we tried to make a comparison between our heap snapshots. To do so, you can click on the top left dropdown (displaying Summary), select Comparison, then select the snapshot you want to compare. You can take a look at the Object section.
That’s the theory, but in fact we had a hard time getting a clear answer or even a clue as to where to look for the leak.
For us, the best thing to do is to make a git bisect to find the culprit commit.
The idea is to set a bad commit (the one where you have your leak), and a good one : an old commit that had no memory leak. Then through the bisect you’ll go to different commits and decide whether they’re good or bad to narrow down to the commit that introduced the memory leak.
git bisect start
git bisect bad
git bisect good <old-good-commit-hash>At every step of the bisect, make sure to start again step 2 and step 3. If you don’t build and start your project, you won’t be testing the right commit. Yes, we’ve made the mistake and it’s already a time-consuming process, so don’t forget it.
Also don’t forget to free the memory by clicking on the bin icon.
After repeating step 2 and 3, you should be able to tell if the commit of the current step of the bisect has a leak or not by comparing your heap snapshots. You can indicate this with the following commands:
# no leak
git bisect good# leak
git bisect bad
Repeat this until the console gives you the culprit commit:
<commit hash> is the first bad commit
To return to your original commit and end the bisect, run:
git bisect resetAnd tadaa you know where your memory leak came from! 🎉
Conclusion
Now you know how we find or detect memory leaks on Nuxt.js!
It’s quite a long process but it worked for us, and when we got a leak a second time, we knew how to proceed.
We also make sure to create a separate commit when we upgrade a plugin or add a new one, to help in case we need to go through a git bisect.
Sources
You can find other tutorials here:
https://marmelab.com/blog/2018/04/03/how-to-track-and-fix-memory-leak-with-nodejs.html
https://blog.risingstack.com/finding-a-memory-leak-in-node-js/
