An Insufficient Memory Deployment Failure
If one deploys a Node.js application to an Amazon Web Services Elastic Beanstalk instance that does not provide enough RAM for the application’s npm install, the deploy may fail without any clear indication of the problem. One can indirectly determine if insufficient memory is the cause of failure and then pursue several techniques to resolve this type of issue.
This post was peer reviewed by Lance Lacoste
During the development process a developer typically wants to deploy to a production environment, which can be an arduous task. Luckily, the modern developer has many managed deployment solutions they can leverage. The Amazon Web Services(AWS) suite, specifically the Elastic Beanstalk(EB) service, provides one such solution. Other options include Microsoft’s Azure Platform, the Heroku Cloud Application Platform, and Digital Ocean Droplets.
Each of these services offers a varying set of features, but all will prompt for a choice on the underlying computing instance’s hardware specifications (e.g. # of CPUs, amount of RAM, etc). This choice will affect the overall performance of the deployed web application and the associated execution cost. Unfortunately, it might not be obvious what the hardware requirements of a deployment environment for a web application might be.
Folks with enough NodeJs experience have most probably run up against a problem unique to the NodeJs platform. During a typical NodeJs application deployment, the npm install command is executed in order to download and install all of a project’s dependencies. npm install(as of at least npm v3.10) seems to not free up any RAM during the sequential loading of project dependencies, instead only requiring more RAM as the process progresses (this memory is freed up at the very end of the process). This quirk can cause surprisingly difficult to debug deployment errors if the deploy environment lacks sufficient memory.
Deployment failures due to insufficient memory are difficult to diagnose with AWS EB tools
An EB deployment will fail if the underlying instance does not have enough RAM for the npm install. Unfortunately, this can be a very tricky failure case to determine directly from the EB console and logs.
Determining that a particular deployment has failed is easy enough. The EB console will show errors on failure. More obviously, if one attempts to navigate to the url associated with the EB project, the associated web server will refuse to make a connection. On occasion, the server will make a connection but respond with a 502: bad gateway.
Evidence of the cause of the failure can be found in the EB console. In the Recent Events feed, one should notice several Error type events, including one with the details:
Failed to run npm install. Snapshot logs for more details.
This message is a bit ambiguous but it’s a hint. One can dig deeper.
EB generates a bunch of logs, both those associated with specific applications (e.g. NodeJs) and system wide output. Clicking on the Logs sub-menu option brings one to a page that provides the ability to request these logs. An instance’s logs can be requested via the Request Logs => Full Logs option. This generates a downloadable zip file of an assortment of system & application logs.
The logs download provides a lot of content, but most interesting in this context are var/log/eb-activity.log and var/log/nodejs/npm-debug.log. One can search eb-activity.log for “npm install” but will find nothing more descriptive than what the EB console already revealed:
Running npm install: /opt/elasticbeanstalk/node-install/node-v6.9.1-linux-x64/bin/npm
Setting npm config jobs to 1
npm config jobs set to 1
Running npm with --production flag
Failed to run npm install. Snapshot logs for more details.
UTC 2017/01/08 18:33:39 cannot find application npm debug log at /tmp/deployment/application/npm-debug.lognpm-debug.log will be just about empty if npm install failed:
<date-of-deploy> cannot find application npm debug log at /tmp/deployment/application/npm-debug.log )How to estimate the memory requirements of a project’s npm install
If one can determine that a project’s npm install requires more RAM than is allocated on a deployment instance, then they can know that insufficient memory is at least one of the problems with the failed deployment. This can be accomplished by using the program execution monitoring tools provided by the development machine’s operating system:
- open the OS’s process monitoring program (e.g. Linux: System Monitor, MacOS: Activity Monitor, Windows: Task Manager)
- navigate to the project’s root directory
- clear the npm cache:
rm -rf ~/.npm & npm cache clear - delete the current
node_modulesdirectory:rm -rf node_modules - execute
npm install - pay attention to the associated RAM usage.
If development is occurring on a MacOS/Linux environment, the following technique might prove more helpful than the program execution monitoring tool:
- navigate to the NodeJs project’s directory
- clear the npm cache:
rm -rf ~/.npm & npm cache clear - delete the current
node_modulesdirector:rm -rf node_modules - execute
npm install & top -p $! - Pay attention to the
REScolumn. This is a good approximation of the physical RAM that npm is using
For example, one has a NodeJs project with a package.json that includes the following list of dependencies:
"dependencies": {
"axios": "^0.15.2",
"babel-core": "^6.17.0",
"babel-loader": "^6.2.5",
"babel-polyfill": "^6.16.0",
"babel-preset-es2015": "^6.16.0",
"babel-preset-es2017": "^6.16.0",
"babel-preset-react": "^6.16.0",
"babel-preset-stage-0": "^6.16.0",
"bcrypt": "^1.0.1",
"body-parser": "^1.15.2",
"classnames": "^2.2.5",
"compression-webpack-plugin": "^0.3.2",
"cookie-parser": "^1.4.3",
"css-loader": "^0.25.0",
"dynamodb": "^0.3.6",
"ejs": "^2.5.2",
"enzyme": "^2.6.0",
"express": "^4.14.0",
"extract-text-webpack-plugin": "^1.0.1",
"helmet": "^3.3.0",
"html-webpack-plugin": "^2.24.1",
"identity-obj-proxy": "^3.0.0",
"jest-cli": "^17.0.3",
"jsonwebtoken": "^7.1.9",
"nconf": "^0.8.4",
"postcss-cssnext": "^2.8.0",
"postcss-import": "^8.2.0",
"postcss-loader": "^1.1.1",
"react": "^15.4.1",
"react-addons-test-utils": "^15.4.1",
"react-css-modules": "^3.7.10",
"react-dom": "^15.4.1",
"react-router": "^2.8.1",
"rimraf": "^2.5.4",
"style-loader": "^0.13.1",
"uuid": "^3.0.1",
"webpack": "^1.13.2",
"webpack-merge": "^2.0.0",
"webpack-node-externals": "^1.5.4"
}Using the technique above reveals that roughly 700 mB of memory is required in order to successfully execute npm install for the project. Wow!
What to do if an npm install requires more RAM than the current deploy instance provides
Upgrade the EC2 instance size
After ascertaining the amount of RAM a project’s npm install requires, one can determine if the deployment instance is sized properly. Deployments on the EB platform take place on Elastic Compute Cloud (EC2) instances, of which there are many varieties. One can see at https://aws.amazon.com/ec2/instance-types/#instance-type-matrix that all instance types except for the t1.micro and the t2.nano provide more than the roughly 700 mB of RAM that the package.json from above would require.
If a deployment is failing and the amount of RAM required for the npm install is not available to the currently used instance type, larger EC2 instances can be used. Larger instances should resolve the memory problem but be mindful when scaling up instance sizes: larger instances cost more money!
Declare non-production dependencies as devDependencies
Some project dependencies are simply unnecessary for a production deployment. Dependencies associated with testing, linting, development tooling, etc. (i.e. dependencies not required by the application during run time) can be declared in the devDependencies section of the package.json. These dependencies won’t be installed when npm install with the production tag is invoked (i.e. npm install --production, this is how EB invokes it). Consequently, the deployment’s npm install will require less RAM.
Build/Bundle the front end application and serve from static asset server
If a NodeJs project includes a web front end, chances are that there are a lot of dependencies in the package.json that are associated exclusively with this part of the application (e.g. react, redux, babel tooling). If the application has front end code transpilation (e.g. babel) and/or asset bundling (e.g. webpack, browserify, etc.) steps, these steps do not need to be executed on the deployment server.
To cut down on the dependencies required for a deployment (and consequently the total RAM required), one can choose to instead perform front end build tasks on the development machine, or as part of an automated build process, before deploying to EB. The resultant front end assets can then be uploaded and served from a Content Delivery Network (e.g. AWS Simple Storage Service). Having a NodeJs app serve front end assets directly is convenient in development environments, but delegating these tasks to a CDN allows a production app to scale more effectively.
The project’s package.json can then be trimmed of all of the now-unneeded front end dependencies. Like the section above, the dependencies that are only needed for the build/bundling of the front end step can be relegated to the devDependencies section of the package.json.
Create and enable a swap area
The idea here is to provision a part of the compute instance’s hard drive as virtual RAM. Basically, when the physical RAM runs low, the OS will free it up by temporarily storing parts of its contents on the hard drive’s designated Swap area. This is obviously a much slower process than using only the system’s physical RAM but it can be a quick fix while determining a more permanent solution. This solution is not recommended for production deployments as it is not very performant and may degrade the integrity of the instance’s SSD hard drive due to the increased number of reads.
Check out https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04 for a walk through or https://github.com/Cretezy/Swap for an easy-to-use script.
Dockerize the application
A more advanced solution and consequently a more technically challenging one is the dockerizing of the application. Docker is a great tool that is beyond the scope of this post but should be in the toolbox of every modern developer.
At a high-level, this approach would involve the generation and deployment of a docker image of the NodeJs application after npm install has successfully concluded. EB supports the deployment of apps from docker images. Unfortunately, this solution requires the extra step of generating a new docker image for each deployment version which ultimately adds more time/complexity to a deployment.
Yarn
While I have not personally tested this, I would be remiss not to mention it. The Facebook-born, npm alternative could possibly pull down project dependencies while requiring less RAM. If this is the case and one wanted to try substituting Yarn for npm, they would have to set up an EB deploy configuration (e.g. /.ebextensions/yarn.config ) that disables the running of npm install and executes the Yarn process instead. If one is more familiar with Docker, configuring a Docker container to make this substitution might prove more effective.
In summary, NodeJs applications can be extremely memory demanding during deployments due to the dependency fetching algorithm that npm install uses. This post discusses how to determine if an Amazon Web Services’ Elastic Beanstalk deployment failed due to insufficient memory. With this knowledge, one can either redeploy to an appropriately-sized EC2 instance or follow the outlined techniques to reduce the deployment memory requirement. Deploy early and deploy often!
