Forecasting NPM Security Advisories

Ryan McGeehan
Jan 3, 2019 · 4 min read

This forecasting essay describes the likelihood of a malicious NPM package being introduced into a javascript development environment.

If you are not familiar with this sort of forecasting work, I’ve been searching the limits of various forecasting and probabilistic risk assessment methods to introduce into security engineering organizations.

The risk we’d like to measure involves NPM packages. High profile incidents involving this ecosystem would make it seem like package takeover issues are very common. This forecast was specifically inspired by a November 2018 incident involving the event-stream package. There was also an issue with eslint a few months earlier.

First, we’ll discuss the several scenarios you can discuss (or forecast!), as a security team.

Second, we’ll measure a measurable scenario with the panel.

Useful Scenarios

While each scenario is similar, they require very different mitigation approaches to reduce the likelihood and impact of an unwanted event.

If you publish an NPM, you’re likely concerned that it could be compromised and published to attack those who depend on it. This is very similar to the event-stream incident, given that you are the maintainer.

The likelihoods of this scenario occurring are highly dependent on the number of maintainers and their individual exposure, the build environment, the publishing NPM credentials, and so on.

For example, compare:

  • 100 non-MFA’d contributing developers using leaked Github credentials with an exposed jenkins server to publish package updates depended on by hundreds of thousands for high impact applications.
  • A lone, MFA’d developer publishing from a secured CI/CD environment, carefully publishing packages, depended on by a few, for low impact applications.

Indeed, you’d believe the likelihood of incident in these cases would be very different, and those beliefs are quantifiable.

Dependency on a package, of course, has far different risks than maintaining one. What is the likelihood that one of the packages you use will be compromised? This varies greatly depending on the number of packages you depend on, how they’re updated, and the likelihood that they are valuable enough to be targeted, among other things.

This scenario includes newly added packages, which could include a typo-jacked package. How simple is it for a developer to add a new package? In what environments? How often are new packages added? What is our risk if one is added locally, to a developer machine?

The Forecast

For this forecast, we wanted to really nail down our expectations that any NPM package in a repository could be compromised. We used NPM advisory data as the closest reference class to inform us about the frequency of hijacked packages, and picked the following scenario:

How many “malicious” advisories will NPM publish in December 2018?

These were the rough principles the judge (myself) followed:

With 90% confidence, the panel estimated the following interval of total advisories involving hijacked packages that would occur in December, would land between 0–2.777:


We were correct, there were six total NPM advisories in December, and none of them met this criteria. Zero sits well within the range of 0-2.777 advisories, and our resulting Brier score for the panel being correct is 0.01.

This is helpful for any ongoing scoring of prediction skill in Risk Measurement.

This offers some information that can be used for prioritizing mitigations. I can only offer my own insight. Given the sheer volume of packages and the low occurrence of wholesale package takeover, I would prioritize the risk of adding a typo-jacked package far more than a current packages being compromised and updated.

But, both are risks. The simplicity and frequency of a typo-jacked package getting into a given build environment seems strictly higher than the concern of an existing package being compromised. This can obviously fluctuate given the conditions of a specific environment (your company). But they’re both risks.

Mitigations could be shared to mitigate both risks, but the introduction of new packages would probably be a more narrow focus for the more likely scenario.

Ryan McGeehan writes about security on Medium.

Starting Up Security

Guides for the growing security team

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store