In Node.js development, have you ever done a fresh
npm install on a known working app only to have it start failing? In our experience, we run into this problem every once in a while. The reason is due to a common practice where we specify a semver range for the package we depend on using
~. For example,
^1.0.0 means anything that’s version
So if an app has a dependency
X@^1.0.0 and pulls in current version
1.1.0 but then the owner of
X publish a new version
1.2.0, a fresh
npm install would pull in the new one. Most of the time (like 99%) this is fine, but occasionally it doesn’t work out and your app starts failing.
Sometimes you have a fairly good idea of what packages were updated and it’s easy to figure out, but other times you might not even be aware that your problem is due to a downstream dependency being updated, and you’d be scratching your head for hours. While with the practice of lock file this is much better controlled, occasionally when you update your locks this could still happen. Usually we update locks one at a time, but could still result in getting new downstream from it, or sometimes we may want to do a refresh on the whole lock file.
In my daily job of supporting hundreds of Node.js developers, I have to debug this issue sometimes. This is a common problem that’s been around since early Node.js days. Before the practice of dependency lock file, we used to rely on
npm-shrinkwrap.json and using precise versions instead of semver in
Our most recent incident is due to
webpack getting updated from version
4.29.0. The issue itself affected a lot of people since
webpack is super popular and it’s being discussed here. As it turned out, this is due to a
peerDependencies related bug in npm.
We ran into this problem independently in one of our large apps. It pulls in a lot of packages with a full development
npm install. When the app first started to fail, we had no idea what was wrong.
We use babel and the dynamic import syntax plugin to transpile our dynamic import code. Since we got a syntax error related to dynamic import, we looked at the obvious suspects like our babel config and the dynamic syntax plugin. After some sanity checks we confirmed that babel was definitely still loading the plugin, so we were quite confused, and we were not immediately aware that the cause was a downstream dependency since we made a lot of our own changes.
After going through commit history we found that the problem occurred in a branch with a commit that was known to work because its PR was green, and that’s when we started to realize the cause was a downstream package got updated.
Now this is not anything new to us and we have a set of standard procedures to debug this. Before going into that, I am going to introduce fyn, a node package manager I wrote that evolved from experience and tools to help with development and debugging in Node.js. I will discuss how some unique features of fyn helped with hunting down the problematic package.
The usual way to isolate a problematic package is to first find what packages have updated. For this, we used to rely on
npm-shrinkwrap.json, and now
package-lock.json, that’s committed regularly. We’d
npm install two different
node_modules, one with a known working lock and another without. Then we have custom tools that would compare packages in the two
node_modules and show those that are different. The problem with this though, is that we generally only keep the lock file on releases and we don’t always have a lock file on the exact time we’d want.
In our incident, we know that the commit passed PR on 1/14, but the lock file was older. While we could go with that lock file, it’d be nice to be able to start on 1/14 to reduce the number of packages that are updated. This is the need for which the lock timestamp feature in fyn was implemented.
So I did
fyn install --lock-time=1/14/2019 without any locks and got a
node_modules with packages that were only published up to 1/14/2019. Next I ran
fyn install without locks to get all the latest packages. And now I can compare them.
Before I continue though, I want to point out that since in this incident the cause was a npm bug, normally we may not be able to reproduce it with
fyn, but coincidentally
fyn had the similar buggy behavior so it was possible or at least easier. However, if the cause was not npm related, then debugging it with
fyn would be the same.
node_modules with fyn is actually very simple. We don’t even need to keep two different copies of
node_modules, because fyn’s lock file can be diffed directly and get very readable results.
In our incident, just a little more than a week’s time we got more than 20 updated packages. Some are our internal packages, but most were public ones that were not direct dependencies in our app, in particular half were the babel packages:
The light bulb went on above my head and I thought to myself, “jackpot!”. I immediately updated fyn’s lock file to set these packages to the older versions and expecting good result, but alas, it was not to be. Luckily there were just about 10 other packages and most were obviously unrelated, and then I noticed that webpack was updated:
That was the most likely related package. Sure enough, after updating with fyn, the error was gone. Awesome!
Since this issue surfaced, webpack’s author was very quick to find the cause and submitted a PR to fix it in npm, which was very impressive as usual.
I wrote this to discuss what we went through with a common dependencies related problem in Node.js and how I used fyn to assist my debugging. fyn is the cumulative result of my experiences and various attempts at writing tools that help with managing
node_modules and dependencies, that I ultimately sunk hundreds of hours to put together as a fully functional node package manager to improve productivity and efficiency.
If the topics discussed here make sense to you and you have related experience or have your own tips and tools to share, then please drop a comment. I’d love to hear your thoughts.