Describing Vulnerability Risks

Improving vulnerability communication with quantitative risk forecasts

Ryan McGeehan
Apr 23, 2019 · 5 min read

Vulnerabilities can be accompanied with clever risk measurements that clearly explain why one particular vulnerability is worse than another.

This helps when traditional classification would consider two vulnerabilities to be equivalent.

The following will discuss the potential benefits of vulnerability measurement with forecasting. This proposes that vulnerabilities can be described with infinite flexibility to describe the risks of the environment they exist in.

Explicitly state a future outcome the vulnerability contributes to.

We might express the risk of a vulnerability with CVSS, critical, or maybe a color code. We might support this with textual / verbal argument, a proof of concept, or other things. None of these are bad. We’ll likely use them forever as they are informative.

An additional measurement approach can describe the vulnerability with a higher fidelity of the risks we are relating to the vulnerability. That is done with forecasting a scenario. People are naturally familiar with the components involved with this sort of measurement… It just needs to be practiced and used a specific way.

For this essay: Let’s describe the risks involved with a vulnerability as a handful of scenarios. We’ll accompany the vuln with forecasts of whether these scenarios will occur.

Some scenarios may be (Given: we do not fix this bug…):

The percentile is an assigned belief to each scenario that we would choose.

This allows for empirical approaches that let us score forecast skill over time, with highly contextual risks we need captured by the vulnerability.

The author (us) can be held accountable to the risks we’ve identified. These forecasts can be measured if we’re ultimately right or wrong. Once we publish these forecasts, we’ll have to measure it after one month/year, and account for the error.

This is a feature of forecasting that is impossible with critical, color codes, or verbal debate. We can’t account for our judgement over time with these methods. CVSS has very little context about the boutique risks of our environment and can’t reach that far into communicating them.

Why is this important?

It is possible to operate in professional environments as a calibrated forecaster.

This means, for instance… if we have a 90% belief in a future outcome, we are eventually correct as close as possible to 90% in all of our historical claims. This is an important concept in predictive measurement along with the concept of prediction error.

There is extensive research in expert prediction calibration dating back to the 50’s. It’s also the backbone of modern meteorology that we benefit from on a daily basis. In more uncertain settings, we see calibration being demonstrated on a number of topics by forecasting groups.

The 538 blog has been forecasting operationally for over a decade with the same measurements.

Image for post
Image for post
From the 538 blog

The above charts from 538 hug a calibration line (1:1). This means that the 538 blog expresses greater degrees of certainty only when they truly believe they will be correct with the same ratio (50% confident claims should be 50% correct over time). This is empirical evidence that suggests their methods are more reliable than “wild ass guesses”.

This cannot be achieved unless we keep track of our forecasts. And that is exactly where the opportunity lies in vulnerability reporting.

Can we build evidence that we reliably measure risk?

Let’s describe the possibilities of a vulnerability.

Imagine that we have discovered a severe vulnerability.

Let’s say it’s on a production server with sensitive data. It’s Struts. It’s the same vulnerability being exploited by numerous internet scanners. It was just exposed when the business made some changes to a network configuration.

Our intuition is screaming that this needs response right now.

Our “critical” classification doesn’t quite do it justice. A “10” CVSS score prioritizes it evenly with other 10's currently known at the company. This isn’t really in the same ballpark as kicking a door down.

All we have is our own expert belief that this is a big deal.

This context rich sense of urgency can be captured with measurable statements. We can express the severe urgency of this issue while also expressing that we are a reliable judge of severity.

Let’s use the following scenarios which might be crafted for this specific risk we are expressing.

We believe if no mitigation occurs:

And here’s the uppercut… We can express that we are already owned.

If we are calibrated and show a low historical error in prediction making… then the source of this risk measurement (us!) can be demonstrated as reliable.

How do we demonstrate that we are reliable in measuring risk?

We must make a lot of predictions to be able to show calibration. There are a lot of opportunities to make these predictions over time. Such as:

Prediction opportunities appear frequently in any environment. We just have to keep our eyes peeled for good ones.

We can also forecast all sorts of business metrics that have nothing to do with security. Of most important note: prediction skill is not entirely subject matter specific. We can regularly participate in prediction of other subjects and see benefits in other topics. For instance, participating in the Good Judgement Open will help keep skills sharp.

It’s most preferable to find these opportunities that are related to our subject matter, but it’s useful to widen this skill set with training.

High potential for automation

Lastly, it’s possible to train classic statistical models to apply forecasts to vulnerabilities at scale. I’ve done similar with a proof of concept here. While that PoC is focused on AWS configuration security, it’s easily adaptable to vulnerability risk. An automated process would simply need insight into categorical data about the vulnerability and can be fully customized to support some data that experts have already assessed. I think there’s interesting work to be done there.


Ryan McGeehan writes about security on scrty.io

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store