Forecasting NPM Security Advisories

This forecasting essay describes the likelihood of a malicious NPM package being introduced into a javascript development environment.

If you are not familiar with this sort of forecasting work, I’ve been searching the limits of various forecasting and probabilistic risk assessment methods to introduce into security engineering organizations.

The risk we’d like to measure involves NPM packages. High profile incidents involving this ecosystem would make it seem like package takeover issues are very common. This forecast was specifically inspired by a November 2018 incident involving the event-stream package. There was also an issue with eslint a few months earlier.

First, we’ll discuss the several scenarios you can discuss (or forecast!), as a security team.

Second, we’ll measure a measurable scenario with the panel.

Useful Scenarios

While each scenario is similar, they require very different mitigation approaches to reduce the likelihood and impact of an unwanted event.

“The package we maintain has been compromised.”

If you publish an NPM, you’re likely concerned that it could be compromised and published to attack those who depend on it. This is very similar to the event-stream incident, given that you are the maintainer.

The likelihoods of this scenario occurring are highly dependent on the number of maintainers and their individual exposure, the build environment, the publishing NPM credentials, and so on.

For example, compare:

• 100 non-MFA’d contributing developers using leaked Github credentials with an exposed jenkins server to publish package updates depended on by hundreds of thousands for high impact applications.
• A lone, MFA’d developer publishing from a secured CI/CD environment, carefully publishing packages, depended on by a few, for low impact applications.

Indeed, you’d believe the likelihood of incident in these cases would be very different, and those beliefs are quantifiable.

“The package we depend on has been compromised.”

Dependency on a package, of course, has far different risks than maintaining one. What is the likelihood that one of the packages you use will be compromised? This varies greatly depending on the number of packages you depend on, how they’re updated, and the likelihood that they are valuable enough to be targeted, among other things.

“We have introduced a compromised package.”

This scenario includes newly added packages, which could include a typo-jacked package. How simple is it for a developer to add a new package? In what environments? How often are new packages added? What is our risk if one is added locally, to a developer machine?

The Forecast

For this forecast, we wanted to really nail down our expectations that any NPM package in a repository could be compromised. We used NPM advisory data as the closest reference class to inform us about the frequency of hijacked packages, and picked the following scenario:

How many “malicious” advisories will NPM publish in December 2018?

These were the rough principles the judge (myself) followed:

With 90% confidence, the panel estimated the following interval of total advisories involving hijacked packages that would occur in December, would land between 0–2.777:

Conclusion

We were correct, there were six total NPM advisories in December, and none of them met this criteria. Zero sits well within the range of 0-2.777 advisories, and our resulting Brier score for the panel being correct is 0.01.

This is helpful for any ongoing scoring of prediction skill in Risk Measurement.

This offers some information that can be used for prioritizing mitigations. I can only offer my own insight. Given the sheer volume of packages and the low occurrence of wholesale package takeover, I would prioritize the risk of adding a typo-jacked package far more than a current packages being compromised and updated.

But, both are risks. The simplicity and frequency of a typo-jacked package getting into a given build environment seems strictly higher than the concern of an existing package being compromised. This can obviously fluctuate given the conditions of a specific environment (your company). But they’re both risks.

Mitigations could be shared to mitigate both risks, but the introduction of new packages would probably be a more narrow focus for the more likely scenario.

Ryan McGeehan writes about security on Medium.