Risky Business

The existence of a security vulnerability in a system does not itself constitute anomalous or unintended behavior. Just because a vulnerability exists, doesn’t mean it can or will be exploited. Maybe 17 other things have to align before Little Bobby Tables comes a’calling. Rather, a vulnerability is the potential for failure. It’s an attribute or property of the system that makes it susceptible to failure or to exploitation or pwnage. It’s the front door key under the doormat that no one has found yet. It’s that funky-ass SQLi that’s been in prod for three years and no one — not the devs, not the black hats — has noticed. In old skool testing parlance, it’s a latent defect.

Now let’s stir things up a bit. Let’s say you wrote the code that has that vulnerability, but you’re super-smart and there’s no way you’d do something like that without a damned good reason. You knew that there are stronger alternatives, but you chose SHA-1. Deliberately and intentionally. Why would you do that? Or as my parents used to say, “What the hell were you thinking?”

Maybe you had a vendor lock-in situation with a library. Maybe you looked at the 17 things that have to go wrong before your hash becomes the last line of defense. Maybe you evaluated the time taken to crack the hash versus the time the hashed data lives. Maybe… I dunno, there are any number of reasons a smart person like you makes decisions that an outsider wouldn’t understand.

You probably discussed it with your AppSec gurus and even ran it all the way up to the CISO. Then you collectively decided it was OK to use a weak hash algorithm. You evaluated the pros and cons.

You looked at the cost of what might happen should that vuln be exploited, the damage that might be done and the value of the asset you’re protecting. You looked at the probability of it being exploited — those 17 hurdles that an attacker has to jump before they can exploit the weak hash. You looked at the cost of finding an alternative that avoids the problem. You decided, nope, SHA-1 it is.

In other words, you evaluated the risk.

In many shops, normal QA testing involves a lot of on-the-fly risk assessment. When a bug is found — a regular, functional, non-security-related bug — the team will discuss it with stakeholders. In Scrumish shops, that’s the Product Owner. In XP shops, those are the Customers. And together we perform a risk assessment. Oh, we don’t usually call it that; we’re more likely to say something like “let’s triage these bugs” or “we’re prioritizing the bugs” or “we’re grooming the backlog.” But trust me, we’re assessing the risk. We’re looking at how likely the bug is to happen in the wild. We’re looking at just how bad things can get if the bug is triggered. We’re looking at what it will take to fix it and what else might get clobbered if someone finally musters the courage to go into the code that guy Steve wrote without tests in a 36 hour streak before he quit in a huff over the beach volleyball incident.

It’s perhaps less formal than a CISSP would come up with, but it is a process (which makes it ISO-9000, right? 😇)

If you’ve been doing software for, I dunno, more than say 10 minutes, you’ll be very familiar with that feeling when a decision is made to leave a bug unfixed. “Let’s just make it a P4.” “We’ll fix it some day when we have time.” (Please don’t derail me now with that #zerobugs or #nobacklog talk. You know it’s a total trigger and you’ll have me frothing at the mouth in no time. Spoiler: I’m in favor.) It doesn’t feel good. As Engineers, we like to clean things up and releasing with known bugs seems like betrayal. But we go along with it. By deferring that bug, the team has decided to accept some risk and live with a known vulnerability.

So good ol’ functional bugs and security vulnerabilities are the same thing, right? Ish.

They’re the same up to a point, and this is where we get into “security architecture” and how some vulnerabilities can’t be found by scrutizing the code, but are systemic, an artifact of the system design. But let’s leave that aside for now.

Let’s look instead at another class of latent defects that are so, so tragically misunderstood: what we used to call usability bugs.

The world of UI design (and its cousins UX, user centered design, CHI, human factors, etc.) has always struggled for a place at the table when defects and failure are concerned. We’ve struggled to be taken seriously. How often do you see a usability bug triaged as a P0 showstopper? I know things are improving, especially in large orgs, but usability has long been considered a nice-to-have, not central to the product, the business. Functionality — whatever that means — trumps usability. (Oh, and substitute “security” for “usability” in the last two sentences and it still holds true.)

Usability flaws don’t always result in immediate anomalous or unintended behavior, but they sure can be vulnerabilities in the broader sense: things that, once triggered by human (in-)action, can result in failure. The Hawaiian missile early warning system was functionally operational but it had a major usability problem that led to failure. I bet all the tests passed. It had probably undergone a pretty comprehensive security review too. But it had a usability bug. That bug was probably there for years and everything was just fine until 17 things lined up just right to cause that poor operator to pick the wrong menu item. (Note: the deets of this incident are pretty murky and seemed to change over time. I have not read the official report.)

Let’s call that class of usability bug a vulnerability. (Saw that coming, amirite?) The system behaved as intended, but it has an attribute or property that makes it susceptible to failure or to exploitation or to causing mayhem and confusion among the residents of Hawaii.

So what do we do with vulnerabilities? Well, first we have to know they’re there, and that’s what security audits and testing and pentesting and red teaming and QA and fuzzing and usability studies and all those things are for. Then, once we know where the vulnerabilities are, we can look at the risk.

Assessing the risk of a usability flaw is an area I’d love to learn more about (suggestions welcome) but I strongly suspect it gets deep into cognitive science and stuff. I am only aware of one (1) industry that routinely, conistently and conscientiously evaluates usability when designing their systems, and has been doing so for years, and that’s aircraft manufacturing. The FAA has been on this for decades. Take a look at Human factors considerations in the design and evaluation of flight deck displays and controls : version 2.0. Surely a title only Government can love.

To sum up, systems have a ton of vulnerabilities of various flavors, including security, functional, usability (and we can throw in a bunch of others too while we’re at it: policy, privacy, scalability, performance, …) but we always, always, always need to assess the risk. There’s an attitude of “fix all the things” when it comes to security vulnerabilities, but as you know, the best way to ensure 100% secure software is never to release it.

It’s risky business.