npm 5 and file: URLs
I wanted to switch a fairly large project in a monorepo to Node 8. Everything ran fine, except I had several issues getting the project to work with npm 5.
This is the layout of our Lerna-based monorepo:
The project called “sdk” is used by the other client apps, so in this case it’s used by both main-app and mobile-app. Because we don’t want to have to version and release sdk, we’ve installed it using a
file: URL. In npm 2–4, this causes npm to copy the entire folder, which means when actively working on both the sdk and client apps we have to use
With npm 5, this all fell apart. It wasn’t really npm’s fault so much as nested package resolution in our webpack config. We’ve used the
modules property in the webpack config to automatically pull in dependencies from each package’s
node_modules folder, but this doesn’t work in npm 5 because the symlink means files imported into main-app from sdk can’t then find sub-dependencies within sdk.
If you switch between npm 3 and 5, then you can test out what’s happening quite easily. Create two packages, and install one using the file: URL:
npm init -f
npm init -f
npm i --save ../package-child
ls -l node_modules
With npm 2–4, you should see a regular directory in node_modules, like this:
drwxr-xr-x 3 alex wheel 102 6 Jul 11:03 package-child
with npm 5 you’ll see a link:
lrwxr-xr-x 1 alex wheel 19 6 Jul 10:56 package-child -> ../../package-child
As I said, I worked around our issues with symlinks by forcing webpack to search for module paths in the sub-dependency. But what I thought was interesting about npm’s behaviour is I only found it after rereading the npm blog (clutching at straws after some failed Googling). Here’s npm’s blog post on the release of npm 5, which for me was the only real source of documentation for the changes in this release (other than the changelog):
npm install ./packages/subdirwill now create a symlink instead of a regular installation.
file://path/to/tarball.tgzwill not change – only directories are symlinked. (#15900)
This helped me a lot, because otherwise I don’t think I’d have figured out this behaviour based on npm’s documentation:
As of version 2.0.0 you can provide a path to a local directory that contains a package. Local paths can be saved using
npm install -Sor
npm install --save, using any of these forms:
in which case they will be normalized to a relative path and added to your
package.json. For example:
This feature is helpful for local offline development and creating tests that require npm installing where you don’t want to hit an external server, but should not be used when publishing packages to the public registry.
So either I’m not looking at the right place in the docs, or the explicit behaviour of
file: URLs being linked isn’t mentioned. The reason it should be mentioned is if you dig into the issue referred to in the release blog post (https://github.com/npm/npm/pull/15900), it says that
file: URLs will be soft deprecated for a new
file:-type specifiers that refer to directories will be soft deprecated, their behavior being identical to the new
link:specifier and their existence becoming a footnote in the documentation.
This is not clear from the npm file specifier specification, so the behaviour is currently something you have to dig for a little bit to understand.
- npm now symlinks
- npm calls
file:URLs “specifiers”, which is telling because the behaviour is different to URLs (link instead of copy)
- We probably should be using
file:, but the documentation isn’t clear enough yet (are GitHub issues documentation)
- If you can’t use symlinks, or if they upset your build tools, then try using tarballs instead
- When you’re trying to figure out npm issues, use
--verbase. The CLI does that Unix thing of being quiet after running, which means the results of an action aren’t always clear