npm 5 and file: URLs

I wanted to switch a fairly large project in a monorepo to Node 8. Everything ran fine, except I had several issues getting the project to work with npm 5.

This is the layout of our Lerna-based monorepo:

project/
project/packages
project/packages/main-app
project/packages/sdk
project/packages/mobile-app

The project called “sdk” is used by the other client apps, so in this case it’s used by both main-app and mobile-app. Because we don’t want to have to version and release sdk, we’ve installed it using a file: URL. In npm 2–4, this causes npm to copy the entire folder, which means when actively working on both the sdk and client apps we have to use npm link.

With npm 5, this all fell apart. It wasn’t really npm’s fault so much as nested package resolution in our webpack config. We’ve used the modules property in the webpack config to automatically pull in dependencies from each package’s node_modules folder, but this doesn’t work in npm 5 because the symlink means files imported into main-app from sdk can’t then find sub-dependencies within sdk.

If you switch between npm 3 and 5, then you can test out what’s happening quite easily. Create two packages, and install one using the file: URL:

mkdir experiment-1
cd experiment-1
mkdir package-parent
mkdir package-child
cd package-child
npm init -f
cd ../package-parent
npm init -f
npm i --save ../package-child
ls -l node_modules

With npm 2–4, you should see a regular directory in node_modules, like this:

drwxr-xr-x  3 alex  wheel  102  6 Jul 11:03 package-child

with npm 5 you’ll see a link:

lrwxr-xr-x  1 alex  wheel  19  6 Jul 10:56 package-child -> ../../package-child

As I said, I worked around our issues with symlinks by forcing webpack to search for module paths in the sub-dependency. But what I thought was interesting about npm’s behaviour is I only found it after rereading the npm blog (clutching at straws after some failed Googling). Here’s npm’s blog post on the release of npm 5, which for me was the only real source of documentation for the changes in this release (other than the changelog):

npm install ./packages/subdir will now create a symlink instead of a regular installation. file://path/to/tarball.tgz will not change – only directories are symlinked. (#15900)

This helped me a lot, because otherwise I don’t think I’d have figured out this behaviour based on npm’s documentation:

As of version 2.0.0 you can provide a path to a local directory that contains a package. Local paths can be saved using npm install -S or npm install --save, using any of these forms:

../foo/bar
~/foo/bar
./foo/bar
/foo/bar

in which case they will be normalized to a relative path and added to yourpackage.json. For example:

{
"name": "baz",
"dependencies": {
"bar": "file:../foo/bar"
}
}

This feature is helpful for local offline development and creating tests that require npm installing where you don’t want to hit an external server, but should not be used when publishing packages to the public registry.

So either I’m not looking at the right place in the docs, or the explicit behaviour of file: URLs being linked isn’t mentioned. The reason it should be mentioned is if you dig into the issue referred to in the release blog post (https://github.com/npm/npm/pull/15900), it says that file: URLs will be soft deprecated for a new link: specifier.

file:-type specifiers that refer to directories will be soft deprecated, their behavior being identical to the new link: specifier and their existence becoming a footnote in the documentation.

This is not clear from the npm file specifier specification, so the behaviour is currently something you have to dig for a little bit to understand.

To summarise:

• npm now symlinks file: specifiers
• npm calls file: URLs “specifiers”, which is telling because the behaviour is different to URLs (link instead of copy)
• We probably should be using link: instead of file:, but the documentation isn’t clear enough yet (are GitHub issues documentation)
• If you can’t use symlinks, or if they upset your build tools, then try using tarballs instead
• When you’re trying to figure out npm issues, use --verbase. The CLI does that Unix thing of being quiet after running, which means the results of an action aren’t always clear