Common npm mistakes

One of the best things about node development is npm. The large library of packages and ease of installing and publishing makes node an open source practitioner’s dream.

One of the worst things about node development is also npm. Actually, most of the problem is not with npm, but challenges in dealing with dependencies. I believe that many of these challenges stem from widespread misconceptions and poor practices.

Having personally built many large scale node applications  services, with big developer teams, I have seen firsthand many “dependency hell” struggles. I’d like to share the common mistakes I’ve observed with the hope that it can help you avoid the same pitfalls we encountered.

Mistake 1: using the wrong dependency type

There are so many dependency types within node that it’s a common mistake for the wrong one to be used. In general, you should determine the relationship of the dependency with your module and (if applicable) the consumer of your module. Eventually, I’d like to get around to authoring a more sophisticated decision flow chart around this, but here’s an abbreviated version.

devDependencies

Is the dependency used during the runtime of the application? If the answer to that question is “no,” then this is a dev dependency. Common categories of dev dependencies include tools for testing, building, minifying, bundling, transpiling, and linting. Sometimes this can seem ambiguous. What about projects using babel or webpack? The output from those tools is used at runtime, but those libraries are not. Some build tools also inject references to runtime modules, and those should not be dev dependencies. For example, babel-polyfill is a regular dependency, not a devDependency.

Be careful if you’re authoring a dev tool, like a gulp plugin or webpack loader. Although your module will be a dev dependency of an application, your own dependencies are not necessarily also dev dependencies themselves. If your build tool’s runtime functionality requires the use of a dependency, that is a regular dependency. But a build tool for your build tool is definitely a dev dependency.

peerDependencies

Is your module a plugin for some library or framework? Then that library/framework is a peer dependency. The consuming application is responsible for selecting and integrating a set of mutually-compatible plugins; the plugins themselves should not introduce said framework directly, but should instead specify a maximally-inclusive version range to make the job of integration as easy as possible. For example:

• Your module is an express middleware; if it requires request and response properties only available in version 4, you should have express@^4.0.0as a peer dependency.
• Your module provides gulp tasks or plugins. The application using your module obviously “owns” the gulp dependency, so your module should never provide its own gulp dependency.

optionalDependencies

Is your dependency something that is not supported on all platforms? And does your module also handle the case where the dependency is not installed? If the answer to both of those is “yes,” then this is an optional dependency. Do not use this dependency type if:

• You want to allow the application to decide whether this is installed or not; although “optional” sounds like it would enable this scenario, there is currently no support for this. Sometimes peer dependencies are used as a workaround due to missing peer dependencies only raising warnings, but that is semantically incorrect and causes shrinkwrap to fail.
• Your module cannot function without the dependency. If it cannot, the dependency is not optional. Consider using the engines property in your package.json to specify which environments your module is compatible with.

To put it another way, the only thing that optionalDependencies is good for is enabling an install to not fail if that dependency does not successfully install.

bundledDependencies

Personally, I have no exposure to these types of dependencies and have no guidelines for these.

dependencies (regular)

The remainder of dependencies are normal dependencies.

Mistake 2: avoiding peerDependencies

There’s a common misconception that peerDependencies in npm are “broken.” They aren’t broken! There are differences between how peerDependencies worked in v2 and v3, so if you’re expecting the behavior to stay the same, you may be forgiven for calling the new behavior “broken.” But I argue that npm 2 had the worse implementation.

With npm 2, if your package.json listed a peer dependency, when someone installed your package, that dependency was also automatically installed. This is definitely convenient in many ways, but it also caused other issues. See the GitHub issue for the behavior change for more details on this, but the gist of the problem is that it’s better to manually resolve peer dependency conflicts issues to avoid the accidental duplication that can occur during automatic installation. Hence npm 3 changed installation to warn you of missing peer dependencies instead, you can can explicitly add that dependency to your own package.json.

There’s also a misconception that peerDependencies is no-longer necessary because npm 3 introduced automatic de-duping. This is not correct! Its true that avoiding multiple installs of a package is one of the purposes of peerDependencies, but dependencies are only de-duplicated if the versions are all compatible. If they are not compatible, you’ll still get multiple versions installed.

To illustrate this, imagine if you had two express middleware dependencies:

{
 “name”: “my-app”,
 “dependencies”: {
   “middleware-a”: “*”,
   “middleware-b”: “*”
 }
}

Now suppose both of these list express as a dependency:

{
 “name”: “middleware-a”,
 “dependencies”: {
   “express”: “^3.0.0”
 }
}
{
 “name”: “middleware-b”,
 “dependencies”: {
   “express”: “^4.0.0”
 }
}

Now what do you get if you require(‘express’)? Will you get version 3 or 4? Obviously, whether your file structure looks like this:

node_modules
 express@4
 middleware-a
   express@3
 middleware-b

…or this:

node_modules
 express@3
 middleware-a
 middleware-b
   express@4

…either way it’s broken and may not work as expected. In contrast, if your dependencies look like this:

{
 “name”: “middleware-a”,
 “peerDependencies”: {
   “express”: “^3.0.0”
 }
}
{
 “name”: “middleware-b”,
 “peerDependencies”: {
   “express”: “^4.0.0”
 }
}

…when you install, you’ll be told that one requires express@3 and the other requires express@4. You’ll have to choose which express version your app will use and select different dependencies that are mutually compatible.

Mistake 3: not using shrinkwrap

If you are writing a module that is not itself an app/service, not using shrinkwrap is fine. However, I frequently hear this phrase from node developers that are authoring apps:

“Shrinkwrapping is annoying. We just use semver, so it’s completely unnecessary.”

This is wrongheaded! Semver versions simply convey developer intentions. If a developer intended a version to be backwards compatible, that’s not the same thing as it actually being backwards-compatible. Even if the package author correctly follows semver guidelines, your app may be relying on “buggy” behavior, and a fix for that bug, even in a patch revision bump, may break your app.

And of course, every release brings with it potential new bugs. Semver does not tell you anything about the quality of a release.

Time and time again, we dealt with breakage of our applications due to unsupervised version bumps. We learned that you cannot trust other developers to not break your app, and it’s not possible to have a test suite that will catch everything.

Shrinkwrap lets you determine exactly which versions of package you can expect to be installed. This way, if you are just updating your own code, you don’t shoulder the burden of also testing dependency upgrades you weren’t expecting.

Mistake 4: not shrinkwrapping devDependencies

If you use a CI/CD system like we do, you’ll save yourself a lot of grief if you shrinkwrap all dependencies, including devDependencies. Do npm shrinkwrap --dev every time. Otherwise your build servers will use different versions of build/test tools than yours, and you’ll get unexpectedly failing builds. And it’s not just the build tools you need to be concerned about, but the output of those build tools. For example, new babel plugin versions may cause unanticipated breakage in client JS.

With npm@4, dev dependencies are shrinkwrapped by default, which is a really good thing.

Mistake 5: ignoring peerDependency warnings

Your dependencies list their peerDependencies for a reason; they establish their compatibility. If you simply ignore peerDependency warnings, you’re ignoring what the package author told you would work, at your own peril.

Additionally, peer dependency warnings prevent shrinkwrap files from being written; this is a good thing, because you don’t want to shrinkwrap an install that is broken anyway.

Mistake 6: being overly-specific with peerDependencies

If you’re authoring a module that has peer dependencies, be as loose as possible with your version requirement; don’t just blindly use the latest version of those dependencies. Remember that the consuming application may be reconciling multiple version requirements; the less specific you are, the better.

Mistake 7: shrinkwrapping optional dependencies

If you’re working in a shop with multiple development environments, beware of platform-specific dependencies. Do not shrinkwrap fsevents, for example, if you expect an install to succeed on a Windows or Linux machine. You can avoid shrinkwrap including your dependency, even if it’s a sub-dependency, by listing it under your own package’s optionalDependencies.

Mistake 8: manually adding dependencies to package.json

Don’t manually add dependencies to your package.json, please! You suck at alphabetizing, and you’re always forgetting to updatenpm-shrinkwrap.json. Use npm i --save and npm i --save-dev only; this will automatically update both package.json and npm-shrinkwrap.json (including doing any deduping). If there is a conflict in generating the shrinkwrap file, you’ll receive warnings about any version conflicts you need to resolve.

That being said, there are some quirks and bugs that make npm CLI maintenance of your package.json annoying. One big annoyance for me is in how versions are saved in package.json; I expect any @ version patterns to be reflected in package.json when doing an install — save, but that’s not what happens. Even if I do npm i — save package@~1.0.0, the version is saved with a ^ in package.json.

There’s also a bug relating to shrinkwrap files. If npm install --save detects that your dependency is already present in your shrinkwrap file, it erroneously stores the version in package.json with the resolved URL of the dependency, not the version number. (update, this has been fixed in the latest npm@5)

Still, despite these annoyances, it still makes sense to use the CLI, but make sure to double-check your package.json before committing it.

Mistake 9: upgrading for the sake of upgrading

One approach our team has taken for maintaining shrinkwrap files was to do rm -rf node_modules && npm i && npm shrinkwrap --dev after every dependency change. Do not do this! You’re going to not-only get the new dependency you wanted, but you’re also introducing version changes for every other dependency at the same time.

This is a potentially controversial opinion, but you do not need to always upgrade to the latest version of everything!. Version bumps may introduce some valuable bug fixes, but more-often-than-not, this is what happens:

• Because your version uses ^, you receive new features (minor version bumps) that you aren’t actually using yet and therefore don’t need.
• You receive a patch version bump that fixes a bug that didn’t impact your app in the first place or that your app relied on behaviorally.
• You receive new bugs that your tests don’t catch. You now have to debug which of the many version bumps caused your issues.

It’s true that you may need to eventually upgrade your packages, but the YAGNI principle applies, even with version bumps; only upgrade once you actually need an upgrade. Treat every version bump with the same scrutiny as changes to your own code, and be sure to have integration tests to ensure that your dependency works as expected.

Summary

Make sure you understand how dependencies work with npm, or you may encounter the dependency hell that has plagued many of your fellow developers. And for your own sake, shrinkwrap to avoid the headache of random version upgrades and non-deterministic installation.

Do you have any other advice relating to npm dependencies? I’d love to hear your thoughts.