yarn-deduplicate — The Hero We Need

Yarn’s logo

You may skip reading the article and just jump to the tool’s readme
yarn 2 users: you don’t need it, read here

After a while of using yarn, when i lost the fear of sniffing inside the lock file, I’ve noticed something suspicious:
The yarn.lock file contains different versions of the same package. and lot of them.
Some of them were even just a different patch version!
And the requiring semver expressions have a common range!

To make things worse, in some cases i got cryptic TypeScript errors:

Two different types with this name exist, but they are unrelated 🧐

And of-course, a lot of the dependencies had their own recursive node_modules like good-old npm 3 days 🤩

A Simplified example:

mkdir yarn-dedup-examples
cd yarn-dedup-examples
yarn init -y
yarn add @types/node@^8 # lts/carbon
yarn add @types/fs-extra # promise wrapper of node's fs

Which will give us duplicate @types/node package

Content of project before yarn-deduplicate

yarn add yarn-deduplicate -D; # local. this is best practice!
yarn yarn-deduplicate -s fewer yarn.lock;
yarn install # Don't forget this step.

And we will end up with:

After yarn deduplicate

Which is the desired state.

To understand better what the deduplication do,
I recommend to ensure your yarn.lock and package.json are committed, run the tool, examining the diff and commit the changes!
And of course to read the documentation

Usage in CI

You can run yarn-deduplicate --list --fail as part of your CI, before yarn install, to ensure no duplications are added over time. Don’t forget to add your strategy flag if your are using the none-default.

https://github.com/atlassian/yarn-deduplicate#usage-in-ci

When also to run this tool?

• After Installing / removing dependencies
• After dependencies upgrades (if you don’t know yet yarn upgrade-interactive you must try it now!)
• rebases/merges that lock file/package.json changes are involved

Other solutions?

https://yarnpkg.com/lang/en/docs/selective-version-resolutions/ this can somewhat help. I personally never used it.

Some background — yarn is “lying” to us

yarn has internal dedupe that prints a text that tries to convince you that you don’t need any dedupe 🤭

yarn dedupe v1.13.0
error The dedupe command isn't necessary. `yarn install` will already dedupe.
info Visit https://yarnpkg.com/en/docs/cli/dedupe for documentation about this command.

But if you search for issues in yarn’s github, you will find a related one: [feature] improve resolution deduping

The discussion is very interesting, the practical part is a link to a tool called yarn-tools, which is an early iteration of yarn-deduplicate.

You can manually do whatever this tool do,
and also change the lock file in a way that will trick yarn, to do many things (Not recommended)

Why yarn won’t do what yarn-deduplicate by itself?

The reason for this was to ensure stable dependencies over time while allowing new installs to use the latest available.

So Yarn preserves the old resolution but prefers the newest when installing a new dependency. Looking back, I think the logical thing to do would have been to use the existing resolution if we were aiming for stability.
@madbyk, yarn v1 team, from the article’s responses

Worth noting, yarn 2 has this feature built-in

Main takeaways:

lock files are important!
When reviewing pull requests, review also lock file changes!
Lock files are changing for a reason, and you need to make sure the changes makes sense.
if the diff is huge — something might be wrong.

Thank you for reading!
Bnaya