Reducing size of NPM based projects
Recently, I purchased a new computer and I was setting up a new development environment with a cloud backup. When I started to move files from one computer to another, I learned that my 43 javascript projects (web, mobile apps, NPM and Meteor packages) contain 2.7 MILLION files in total of 42 GB! My internet cloud backup has estimated 250 days to complete. I felt similar to this:
When I analysed what consisted of these files I found out that 93% were stored in node_modules folders with 90% redundancy, some being copied 30 times or more! I guess, when you just go on typing those npm installs you do not realise how much stuff is landing on your computer.
The reason is quite simple, each npm install puts in your node_modules folder the latest versions of packages as specified in your package.json and all of their dependencies. Since each project is in a different state and uses different versions, logically they will keep different files. Yep … but as I’m trying to maintaim all my projects up to date, I end up with all the redundancies, having the same NPM package copied over and over to a different directory.
When I posted my question in NPM issues, I got no meaningful response (I got 1 in total :). But I did not give up and set out to solve this problem myself by writing an npm extension that would symlink modules into the node_modules directory. This brought upon me yet another shitstorm of problems, but I ended up studying following thread that pointed me towards the AMAZING pnpm project.
Not only pnpm is blazing fast, much faster than npm, it solved all the problem with symlinking the modules into so called store inside the project directory.
Long story short, using pnpm I have reduced the amount of files to little over 300.000 and size dropped to 4GB. That’s around 89% reduction in amount of files and directory size! My cloud backup finished in reasonable time and moving projects and copying boilerplates is a breeze.
Since pnpm create a separate store in each project, I had to tweak the settings of pnpn to create a global store, shared by all the projects. You can follow the update on this issue here. What you can do to make it work now?
- First, fork the pnpm project and clone it into your directory
2. In the cloned directory open the ‘lib/config.js’ file and modify path of the store to the directory of your desire.
module.exports = require(‘rc’)(‘pnpm’, {
...
store_path: ‘/Users/tomi/.npmstore’,
...
})3. In the cloned directory run npm link to create a global module
DONE!
All you can do now is to delete your node_modules folder run ‘pnpm install’ and watch the magic happen!
