On Being Operationally Incompetent

(I originally posted this as a comment to a GitHub issue but decided it was important enough for a larger audience to read.)

To catch you up, 11 months ago the maintainers of the highly popular node request module dropped support for versions of node 0.x. These versions of node are no longer under LTS support and are considered harmful because they contain multiple critical security issues and other bugs. At the time this change was done, the version number of the request module was not updated to reflect what is in practice, a breaking change.

And so for 11 months, anyone with such an unsupported version of node have been getting ugly warning from the npm client that they are installing unsupported software. Of course, these folks ignored the warnings. Again, for 11 months.

Last few days, the maintainers of request updated one of its dependency that uses capabilities not available in node 0.x which broken any environment that automatically updates its dependencies. Shit hit the fan for many people.


I am not interested in the debate around what was the right version to publish this change and the rules of proper semver compliance. If you want that debate, go read the issue itself. Ironically, it was one of my personal modules that cause this in the first place, and I do derive some pleasure from that irrelevant fact.

I am interested in security.

Here is what I posted in response to the rage and entitlement expressed by those impacted by this incident:

Another way to look at this for anyone who got hit by this: your environment is fully exposed to the worse kind of DoS attack vector — operational incompetence.
I am being harsh here for a reason because you got lucky today and some of you still consider this to be someone else’ fault. This change caused your application to fail instead of leak data, lose money, or cause irreversible hard. What I hope you have learned today is that ANYONE with publish rights to ANY module you depends on, directly or indirectly, can shut down your entire environment. That is at best fucking ridiculous and at worst criminally negligent.
Think about what you are admitting to have created — an environment where hundreds (if not thousands) or people have the power, maliciously or innocently, to put you out of business (if only for a short period of time), without any accountability or recourse. Maybe open source isn’t for you.
Most of you don’t know me, don’t give a crap about what I do, and certainly don’t consider me to be an active security threat. And yet, I have publish rights to modules that are being downloaded by pretty much every fucking node deployment in the world. I have at least 95% access. I have publish rights to qs and I don't even maintain that module anymore - that's 2.5 million daily downloads. npm Inc. will stop me from deleting the module from the registry, a minor improvement over the last registry clusterfuck. But I can still push garbage and impact 2.5 million clients if it takes npm Inc a full day to shut me down.
There is always a delicate balance between productivity (using loose version requirements in your package file to automatically get new version) and security (not getting hit by random shit being posted to the registry). But an easy line to draw is between your local development environment and your production or operational environments.
“Can I afford this system to go down or be exposed to malicious code for a short period of time?” is a simple question to ask yourself. For your personal laptop? probably. For the team’s build servers? probably not. For your production system? fuck no!
If anyone can cause you to run whatever code they want, you have a massive problem. Again, I can publish right now, a version of qs, that when loaded into memory, will upload files from your machines to my cloud account. Maybe I'll grab your browser history and publish some interesting stats on the kind of porn node developers prefer... It won't last long as people will notice and npm Inc. will take action. But for a brief time, I will fucking p0wned your environment.
And no, you can’t event trust me because my systems can be hacked. If someone gets access to an npm access token from any of my machines, they gain the same power… I don’t even trust modules I am the sole publisher of.
Consider this experience an inexpensive lesson in open source operational security.
A single golf clap? Or a long standing ovation?

By clapping more or less, you can signal to us which stories really stand out.