Photo by EJ Strat on Unsplash

Make your CI faster and improve developer experience: Upgrade to Yarn >2 with (or without) Plug’n’Play

In 2020, Yarn maintainers released a bomb in the world of JavaScript package managers. Yarn 2, or “berry”, introduced a feature called Plug’n’Play, or PnP for short, meant to improve dependencies management, performances, and developer experience thanks to faster workflows.

2 years later and PnP has been ignored by a large part of the JavaScript ecosystem. Even when PnP is supported, it is often an afterthought. While it’s getting better, it’s hard for developers to justify upgrading to Yarn 2 with PnP, which often introduces a lot of new errors, paradigms and constraints.

The thing is, because of that, people have refrained from even upgrading to Yarn 2, often associating that release with PnP. I will tell you a secret: You don’t have to use PnP to use a lot of cool features from Yarn 2/3.

After reading this article, you’ll want to upgrade to Yarn 3, and you should be considering trying PnP.

Actually, let’s do it now, upgrade to Yarn latest version

$ yarn set version berry

In .yarnrc.yml , add:

nodeLinker: node-modules

In .gitignore, add:

.yarn/*
!.yarn/patches
!.yarn/releases
!.yarn/plugins
!.yarn/sdks
!.yarn/versions

Here you go. You’re using Yarn >2 without PnP. For most codebases, it will not break anything and you’ll benefit from the latest performance upgrades from the Yarn team.

But most importantly, you can now start using their new features, notably the one I consider the most important: the offline cache (more on this below).

At Inato, we upgraded most of our repositories to Yarn 3 that way without incidence on any of our processes.

You don’t have to download your dependencies all the damn time

The main great feature in my opinion that comes with Yarn 2 is called offline cache.

The concept is simple: you add your dependencies to your git repository. Not your node_modules directory, but zipped and much more lightweight versions of all the dependencies you use. Every time you add a dependency, you commit the new zip file. Upgrading a dependency? You commit the new zip file and remove the previous one!

Consider the pros:

• Coworkers pulling changes from your repository will also pull the zipped versions of the dependencies. Running yarn install will only unzip that dependency and copy the files to your node_modules. No need to fetch them from a remote registry.
• npm servers are down? No problem, yarn has everything you need to install dependencies, it only needs the zip files!
• What is true for you is also true for your CI/CD! Running yarn install is much faster since it doesn’t need to download any dependency. The need for caching and restoring your node_modules directory is very limited now, since yarn install only needs to extract the zip files to the node_modules directory.

To add offline cache capabilities to your repository, you just need to tell git not to ignore the cache subdirectory of .yarn, so add this to your .gitignore:

!.yarn/cache

Then git add .yarn and enjoy faster CI times and an overall faster workflow, also more resilient to network problems!

It may feel wrong to download that many zip files when you only need a small subset or even none of your dependencies, but this is a drawback we chose to embrace.

Note that since dependencies are added to the repository, you could expect that we’re just trading yarn install time with git clone time. In our experience, this is not true, checking out our code is stable and fast. This may vary depending on where you host your code, and where you run your CI of course. Anyway, you’re removing the npm Single Point of Failure, which is still a nice enhancement.

Here is the yarn install step for a relatively small repository that we have. Previously installing dependencies took around 40 seconds, now it only takes around 15 seconds for each of our jobs.

Other Yarn 2 features

Other interesting features include plugins (like the typescript plugin that automatically adds the @types/ dependency of dependencies you add), the portal protocol, yarn workspaces foreach… More information can be found here.

Ready for PnP?

You’ve improved your workflow, why stop now?

First, let’s talk about the benefits of PnP. The main benefit is that it takes the offline cache a step further by allowing Zero Installs. A yarn install is even faster with PnP, because there is no node_modules directory to copy the zipped files into. Instead, with PnP yarn tells node during runtime where to go look for files (directly into the zip files) when it does a require/import.

yarn install is then only useful to extract/build some binaries from the zip files and is therefore almost instantaneous.

It does that by generating JavaScript files .pnp.*. When a JavaScript file requires another JavaScript file, it will go through that .pnp file that will map the request to the correct file that’s still embedded in a zip.

That means that this .pnp file needs to be loaded by node whenever you use it. Yarn exposes a yarn node command that is correctly setup, and if you use node in a package.json script, it will automatically use yarn node . But if you use node directly, it won’t work.

This is the reason why migrating to PnP can be hard: you can be depending on tools that don’t use yarn node and instead try to access node from your $PATH , or some tools can try to resolve dependencies in specific way that is incompatible with the .pnp file… Dependencies can also forget to declare dependencies, thus making it impossible for Yarn to know which version it expects, resulting in an error (that can be resolved thanks to a feature called packageExtensions).

To try PnP, just change nodeLinker in .yarnrc.yml to pnp , and run yarn install

You will also need to setup your preferred IDE configuration, but Yarn helps you with that, just follow their guide.

In my experience, most tools today work with Yarn PnP, but you may need to wait for patches to use the latest versions sometimes, because as I said, sometimes PnP is still an afterthought.

With this, our yarn install time for our largest repository that has around 15 jobs in its CI that need to install dependencies is only 6 seconds. We use yarn workspaces focus to go even further and only install some of our monorepo’s packages dependencies.

A hidden bonus is that for most package changes, coworkers pulling changes from the repository will not even need to run yarn install.

Conclusion

Yarn 2/3 are largely misunderstood releases that give access to features everyone can benefit from.

At Inato, it mostly allowed us to run a lot of our CI steps in parallel, since the yarn install step is so cheap now.

Instead of typechecking/linting/running tests in series, we now do all of these in parallel jobs.

Upgrading to Yarn 2 was the starting point/building block of other improvements in parallelisation that brought our CI/CD Time to Production to only 8 minutes without compromising on the number of tests we run.

This led to faster product iteration, faster bug fixes, and happier users!

Drug discovery is a challenging, intellectually complex, and rewarding endeavor: we help develop effective and safe cures to diseases affecting millions of people. If you’re looking to have a massive impact, join us! https://inato.com/careers/