Threat Modeling Retrospective

In 1999 I participated in a task force effort to address the growing realization that something had to be done about the security of Microsoft’s products — and of course, the challenge was hardly unique to one company. As a leading platform technology provider, Microsoft had the resources as well as motivation to make a sustained effort, and to my knowledge was the sole pioneer tackling security at such a large scale.

Twenty years later awareness of software security is widespread, with vulnerability reports routinely appearing in mainstream media. Yet in no way, shape, or form, does anyone imagine we are anywhere near turning the corner and reliably producing very secure software systems.

That early work brought security threat modeling from an academic concept into application by the commercial software industry. The technique that came to be know as STRIDE(*) was first described in The threats to our products, and disseminated in a company-wide publication to spearhead the new effort to understand the security of Microsoft products and begin the work of mitigation and eventually proactively shipping more security releases.

(*) STRIDE is an acronym for: Spoofing identity, Tampering data, Repudiation (denial of responsibility), Information disclosure (data breach), Denial of Service (a.k.a. DoS), and Elevation of privilege.

Despite demonstrably raising the security bar, threat modeling is hardly commonplace in commercial software development today. Why is that?

I would be the first to admit that, as originally envisioned, threat modeling with data flow diagrams and an enumeration of potential threats (STRIDE) is a major undertaking for large software systems. In principle, threat modeling should capture everything going on within the system, anticipating every possible type of threat, to identify a prioritized list of needed mitigations.

I believe that we came up with a solid method that could be disseminated and applied within the various product groups at scale. The company-wide roll-out took a few years, but in the end arguably achieved a quantum increase in the security of products. Much room for improvement did, and still, remains.

The major criticism of threat modeling boils down to it being heavyweight: substantial training is required to practice the method effectively, capturing complex data flows accurately is time-consuming, and then considerable experience is needed to see and effectively prioritize the myriad potential threats identified. Very large projects tend to snowball — however, this may be inevitable as the security analysis of large systems is inherently complex.

Researchers at CMU’s Security Engineering Institute surveyed 12 varieties of threat modeling that have been described. Their conclusion offers a mix of pros and cons, with the best method depending on a number factors — to which I would add the overhead of training required to use new methods.

Unsurprisingly, more thorough and consistent results entail greater expenditures of effort. Without studying (much less actually trying out) these other methods of threat modeling, I think it is safe to say that they represent different points on a spectrum trading off various aspects of the full-blown Microsoft style threat enumeration for more tractable approaches. For example, “Persona non grata” uses profiles of potential bad-guy attackers: analysis derives from anticipating what they would do given their defined goals and skills. Embodying potential attacks in theoretical avatars is a good way to imagine potential attacks, but success vitally depends on the analyst’s ability to “think like an attacker” and of course the accuracy with which the goals of actual malicious actors can actually be anticipated.

In summary, my take would be that just as physical exercise is essential to maintain good health, there is no one form that is superior: whatever actually gets done is always a great choice, and undoubtedly better than not doing anything. Furthermore, over time, regular practice proactively identifying security risks builds experience that makes the process easier while gaining better and better results. And, inevitably, to the extent that efforts fall short the bad guys will be sure to teach whatever lessons remain to be learned.

Everyone serious about software should make security a priority, beginning with awareness and mitigation starting from the design stage. Threat modeling is not necessarily the only way to accomplish this end, but it’s a well established approach based on solid fundamentals. (I’m working on a book that describes one industry-proven alternative that effectively applies the same security and privacy principles to software designs.)