Embracing the ‘npm unpublish’ command

It’s been an odd week in the JavaScript land.

The StackOverflow 2016 survey came out and listed JS as the most popular front-end and back-end language. This means that the nodejs ecosystem has grown and matured and can no longer be seen as niche, experimental or just a hype.

Part of this ecosystem is the huge number of publicly available node modules (more than 250,000), usually available through npm registry and managed with the npm command. If you have experience with package managers, you must recognize that npm works incredibly well. It works so well that it gets easy to forget things can go south… and that’s the other thing that happened in the JavaScript land this week.

Left-pad, a module a lot of packages depended on, was removed from the package registry. Installation of modules that depended on it suddenly broke. Builds of apps that relied on those suddenly broke. The havoc was so big that the left-pad author’s removal of the module was reverted by npm’s staff.

At OutSystems, npm is an essential part of our build process, which is continuously running. But we didn’t notice any problem. And that’s because we designed our build infrastructure considering that npm modules and module versions can’t be taken for granted.

Accepting the Ephemerality of Software and Dependencies

We build a product with LTS (long term support) versions. As such, we must make sure that build dependencies of today will still be available for use in 2 years, should we launch a new patch for an old version.

Modules going away might be rare in the npm registry, but module versions going away happens a lot. We have to make sure that the exact version of any package we want will still be available for use to us.

One way of ensuring this for node modules is to add them along with the rest of your source code to the repository. The problem with that strategy is that node modules can explode in number of files and cause unwanted overhead.

So we opted for another strategy. We setup a local npm registry for our build processes. This registry is configured such that any package requested that it doesn’t know about is fetched from the official registry, but once it does that, the package gets cached locally. Once a package is cached, it will never be requested again from the official registry. One effect of this is that unpublished packages or versions on the official registry continue to be available to us.

To use this, we setup a local server running sinopia, which is a perfect fit for this use-case. We also make use of the npm shrinkwrap command to pin specific versions of the packages, and add only package.json and npm-shrinkwrap.json to source control. This way we can keep using npm and we’re sure that everyone gets the same packages every time.

A disadvantage of this solution is that some package installations perform downloads from other sources, so we have to cache those somewhere else. Fortunately the only case we found so far (the phantomjs package) is very easy to configure for alternative download locations, which can also be locally cached (also to make sure that they won’t disappear through 3rd party actions).

Solve little at a time, and then keep solving

The issue of the removal of left-pad from the registry will still give way to a lot of discussion about a lot of topics. Can you be bullied-out of a package name? Is the npm registry reliable? Does it make sense to write very small modules, or is it best to repeat those one-liners through your whole code-base? Lots of questions with lots of strong opinions will still run for a long time.

In the meantime, if you’re building a product and you need predictable, repeatable builds, you can solve the reliability problem of 3rd party systems by rolling your own local caches.