Shrink the Node_Modules Folder Size in Yarn(1.*) Project by Deduplicating Dependencies

We can reduce our node modules size by these actions:

• find and remove all unused dependencies with depcheck.
• deduplicate all the duplicated dependencies with yarn-deduplicate.

Step 1 — remove unused dependencies

first, we will install this tool:

npx depcheck

and then we will run it with this command(for advanced uses you can look on the docs):

depcheck

it will give you a list of all Unused dependencies, Unused Devdependencies, and Missing dependencies — for example:

Unused dependencies
* @aws-amplify/auth
* @fullcalendar/core
Unused devDependencies
* @babel/core
* @babel/plugin-transform-typescript
Missing dependencies
* serverless: ./serverless.yml

now you can remove those dependencies (only) with yarn remove (its a bad idea to remove it by simply delete the files or changing the package.json or yarn.lock manually — for a lot of reasons…):

yarn remove @aws-amplify/auth @fullcalendar/core @babel/core @babel/plugin-transform-typescript

and then run

yarn install

again, to update your node modules library.

in the docs there is this quote:

Depcheck just walks through all files and tries to find the dependencies according to some predefined rules. However, the predefined rules may not be enough or may even be wrong.

There may be some cases in which a dependency is being used but is reported as unused, or dependency is not used but is reported as missing. These are false alert situations.

so — it’s possible to get a false alert and remove dependent that is in use — so its recommended to remove the unused dependencies one by one and make sure that your app is working properly after removing this dependency.

Step 2- deduplicate all the duplicated dependencies

*** This part is relevant only if you use Yarn v1. Yarn v2 supports package deduplication natively and you don’t need to use this tool at all ***

the second step is to clean up yarn.lock by removing duplicates.

A duplicate package is when two dependencies are resolved to a different version, even when a single version matches the range specified in the dependencies.
if you “host” other npm packages that use different versions of heavy dependencies(material-ui, babel, jest, etc…) you can easily go to crazy node_modules folder size — that will affect developing, build and deploys times and performance.

you can check how much duplications you have with by running this command:

npm ls — parseable | xargs -L1 sh -c ‘basename $1’ dummy | sort | uniq -c | grep -v “^ *1 “ | sort -rn

to get the names of duplications with the numbers:

or by run:
yarn-deduplicate — list
and get the names and the versions:

you can also see the size of your node_modules and understand which packages make the noide_modules very big with this:
cd node_modules
du -h -d 2 | sort -h
you will get a list of all libraries and their size:

and it can help you to focus on the specific packages that you should focus on.

Why to use it(yarn say that they do it automatically but they don’t 😞)
Yarn documentation seems to suggest this package shouldn’t be necessary.
For example, in https://classic.yarnpkg.com/en/docs/cli/dedupe/, it says

The dedupe command isn’t necessary. yarn install will already dedupe.

This is, however, not exactly true. There are cases where yarn will not deduplicate existing packages. For example, this scenario:

• Install libA. It depends on libB ^1.1.0. At this point, the latest version of libB is 1.1.2, so it gets installed as a transitive dependency in your repo
• After a few days, install libC. It also depends on libB ^1.1.0. But this time, the latest libB version is 1.1.3.

In the above scenario, you’ll end up with libB@1.1.2 and libB@1.1.3 in your repo.

How to use:
in your root project folder run:

yarn add yarn-deduplicate -D; # local. this is best practice!
yarn yarn-deduplicate -s fewer yarn.lock;
yarn install # Don’t forget this step.

To understand better what the deduplication do,
I recommend to ensure your yarn.lock and package.json are committed, run the tool, examining the diff, and commit the changes!

you should see something like this in your yarn.lock:

as we can see in this example, yarn-dedupe changes change the yarn.lock and now these “@babel/helpers@⁷.7.4” will use version 7.10.4, and in the next time that we will yarn install

the @babel/helpers@⁷.7.4 will remove from our node_modules tree.

here is an example of a node_modules tree of an npm package of other teams that we host in our main app before and after deduplication:

and this is only a little part of the diff — so you can understand how it can shrink the size of the main node_modules.

there are a lot of options, strategies, and more details to learn in this tool — highly recommended to read the docs and this great article.

if your yarn.lock is very big and it’s possible deduplicate every package separately, you should do it with this command:

yarn yarn-deduplicate — packages <package1> <package2> <packageN>

it is possible that things can stop workes after this tool (you actually change dependencies of your dependencies) in some cases you would delete your node_modules library and install it again with:
rm -rf node_modules/ yarn install

anyway — don’t forget to yarn install after dedupe and make sure to test your app after using this tool and make sure everything will work as expected.