Debug Node.js Dependencies

Node.js development, have you ever done a fresh npm install on a known working app only to have it start failing? In our experience, we run into this problem every once in a while. The reason is due to a common practice where we specify a semver range for the package we depend on using ^ or ~. For example, ^1.0.0 means anything that’s version 1, like 1.1.0 or 1.2.1.

So if an app has a dependency X@^1.0.0 and pulls in current version 1.1.0 but then the owner of X publish a new version 1.2.0, a fresh npm install would pull in the new one. Most of the time (like 99%) this is fine, but occasionally it doesn’t work out and your app starts failing.

Sometimes you have a fairly good idea of what packages were updated and it’s easy to figure out, but other times you might not even be aware that your problem is due to a downstream dependency being updated, and you’d be scratching your head for hours. While with the practice of lock file this is much better controlled, occasionally when you update your locks this could still happen. Usually we update locks one at a time, but could still result in getting new downstream from it, or sometimes we may want to do a refresh on the whole lock file.

In my daily job of supporting hundreds of Node.js developers, I have to debug this issue sometimes. This is a common problem that’s been around since early Node.js days. Before the practice of dependency lock file, we used to rely on npm-shrinkwrap.json and using precise versions instead of semver in package.json.

Our most recent incident is due to webpack getting updated from version 4.28.4 to 4.29.0. The issue itself affected a lot of people since webpack is super popular and it’s being discussed here. As it turned out, this is due to a peerDependencies related bug in npm.

We ran into this problem independently in one of our large apps. It pulls in a lot of packages with a full development npm install. When the app first started to fail, we had no idea what was wrong.

We use babel and the dynamic import syntax plugin to transpile our dynamic import code. Since we got a syntax error related to dynamic import, we looked at the obvious suspects like our babel config and the dynamic syntax plugin. After some sanity checks we confirmed that babel was definitely still loading the plugin, so we were quite confused, and we were not immediately aware that the cause was a downstream dependency since we made a lot of our own changes.

After going through commit history we found that the problem occurred in a branch with a commit that was known to work because its PR was green, and that’s when we started to realize the cause was a downstream package got updated.

Now this is not anything new to us and we have a set of standard procedures to debug this. Before going into that, I am going to introduce fyn, a node package manager I wrote that evolved from experience and tools to help with development and debugging in Node.js. I will discuss how some unique features of fyn helped with hunting down the problematic package.

The usual way to isolate a problematic package is to first find what packages have updated. For this, we used to rely on npm-shrinkwrap.json, and now package-lock.json, that’s committed regularly. We’d npm install two different node_modules, one with a known working lock and another without. Then we have custom tools that would compare packages in the two node_modules and show those that are different. The problem with this though, is that we generally only keep the lock file on releases and we don’t always have a lock file on the exact time we’d want.

In our incident, we know that the commit passed PR on 1/14, but the lock file was older. While we could go with that lock file, it’d be nice to be able to start on 1/14 to reduce the number of packages that are updated. This is the need for which the lock timestamp feature in fyn was implemented.

So I did fyn install --lock-time=1/14/2019 without any locks and got a node_modules with packages that were only published up to 1/14/2019. Next I ran fyn install without locks to get all the latest packages. And now I can compare them.

Before I continue though, I want to point out that since in this incident the cause was a npm bug, normally we may not be able to reproduce it with fyn, but coincidentally fyn had the similar buggy behavior so it was possible or at least easier. However, if the cause was not npm related, then debugging it with fyn would be the same.

Comparing node_modules with fyn is actually very simple. We don’t even need to keep two different copies of node_modules, because fyn’s lock file can be diffed directly and get very readable results.

In our incident, just a little more than a week’s time we got more than 20 updated packages. Some are our internal packages, but most were public ones that were not direct dependencies in our app, in particular half were the babel packages:

The light bulb went on above my head and I thought to myself, “jackpot!”. I immediately updated fyn’s lock file to set these packages to the older versions and expecting good result, but alas, it was not to be. Luckily there were just about 10 other packages and most were obviously unrelated, and then I noticed that webpack was updated:

That was the most likely related package. Sure enough, after updating with fyn, the error was gone. Awesome!

Since this issue surfaced, webpack’s author was very quick to find the cause and submitted a PR to fix it in npm, which was very impressive as usual.

I wrote this to discuss what we went through with a common dependencies related problem in Node.js and how I used fyn to assist my debugging. fyn is the cumulative result of my experiences and various attempts at writing tools that help with managing node_modules and dependencies, that I ultimately sunk hundreds of hours to put together as a fully functional node package manager to improve productivity and efficiency.

If the topics discussed here make sense to you and you have related experience or have your own tips and tools to share, then please drop a comment. I’d love to hear your thoughts.

Oh, one last thing, I am hiring JavaScript experts at Walmart Labs, to work on Node.js/ReactJS and anything that’s platform, infra, web, and cloud related. We do bash as well, and it would be super exciting if you are into TypeScript. Hit me up if you are interested.



We’re powering the next great retail disruption. Learn more about us —

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store