Turning tables with attackers; from fixing vulnerabilities into fixing weaknesses

Should we fix vulnerabilities or weaknesses?

In my experience, the people working as security researchers and testers face a difficult choice. Should they only report security vulnerabilities, or should they push for getting also weaknesses fixed? In this blog, I share my thinking on this topic.

TL;DR

It is often said that an attacker has an advantage over a defender, as the attacker needs only to find a single vulnerability, but the defender must find them all. However, we may turn the table around by starting to fix weaknesses rather than vulnerabilities. It often takes a lot of effort to determine if a weakness is a vulnerability or not. By using that effort to fix the weaknesses, we make the system more secure.

Vulnerability vs. weakness

Common Weakness Enumeration FAQ defines vulnerability and weakness (https://cwe.mitre.org/about/faq.html):

Weaknesses are errors that can lead to vulnerabilities. A software vulnerability, such as those enumerated on the Common Vulnerabilities and Exposures (CVE®) List, is a mistake in software that can be directly used by a hacker to gain access to a system or network…

So, vulnerability is a flaw that can be used to compromise a system, while weakness is a potential vulnerability. So, as a security tester, should we report the weaknesses we find, or is it our job to analyze the weaknesses and only report the vulnerabilities?

A quick survey on the Internet finds backing for both opinions. Some sources especially mention that security testing should report vulnerabilities, but some say that also weaknesses, threats, and risks should be reported.

Below we use the term triage for the activity to determine if a weakness is a vulnerability.

Cost of vulnerability testing

Figure 1 illustrates the cost of a weakness found by testing: cost of finding C_find, cost of triaging C_triage, and cost of fixing C_fix. The total cost is C = C_find + C_triage + C_fix. You can avoid the cost to fix C_fix, when your triage shows that the weakness cannot be exploited, and you leave the weakness as is. You save the cost to triage C_triage if you skip the triage and just fix the weakness.

Figure 1: Cost breakdown into cost to find, cost to triage, and cost to fix.

The cost to fix C_fix is not just the cost of making the code change, but also the cost to create and deploy the fixed version. Typically this cost is low when the product is being developed and high when the product is already used by customers. The cost should also include the potential impact of the fix for disturbing the other functionality, which can be a significant risk.

Figure 2 shows the situation when the cost to triage C_triage is much bigger than the cost to fix C_fix. This may be, e.g. as the product is in the (early) development phase, so fixing is smooth with new development versions rolling out constantly. As you can see, leaving out triage and fixing a weakness is a smaller cost than triaging the weakness and leaving it unfixed. In this scenario, it is inefficient to spend time triaging weaknesses.

Figure 2: Cost to triage is bigger than cost to fix.

Figure 3 shows the case where the cost to fix C_fix is bigger than the cost to triage C_triage. Triaging issues increase efficiency if it leads to avoiding fixes. However, if most of the weaknesses turn out to be vulnerabilities, then triaging does not reduce cost.

Figure 3: Cost to fix is bigger than cost to triage.

So, the decision of whether to fix weaknesses or vulnerabilities seems to be dependent on 1) the relative cost of triage and fix, and 2) the ratio of weaknesses turning out to be vulnerabilities.

Cost of vulnerabilities

The cost of security testing should be compared to the expected cost of security breaches resulting from the vulnerabilities left unfixed. You can find more information e.g. from IBM Cost of Data Breaches report 2020 (https://www.ibm.com/security/digital-assets/cost-data-breach-report/).

Practical considerations and advice

Now that we have covered some basics, I want to follow-up on the nature of weaknesses and vulnerabilities, the related costs, and discuss some practical considerations and advice.

Threat analysis

A threat analysis is required to know if a weakness is a vulnerability. Threat analysis comes in various forms, but the relevant part is where the paths for an attacker to compromise the system are identified. If the weakness lies in an attack path, then it is a vulnerability. Also, future attack paths should be considered, a weakness may transform into a vulnerability when combined with yet unknown or yet unimplemented weakness elsewhere in the system.

If you are security testing a system, there is a good chance that the threat analysis is already performed. However, it may lack the details to know if a particular weakness constitutes a vulnerability. This is for a good reason, as such detailed threat analysis would be complex and it is pretty much impossible to enumerate all possible attack paths beforehand.

So, in practice, you must build bottom-up from the weaknesses towards a scenario where the system is compromised to find out if it is a vulnerability. This may be a lot of work and may require in-depth understanding of the system. It is also fairly easy to overlook some aspects and come up with a wrong diagnosis. The world is full of examples of vulnerabilities initially dismissed as mere weaknesses, see e.g. “Heartbleed security flaw may not be as dangerous as thought” from The Verge (https://www.theverge.com/2014/4/11/5604300/heartbleed-may-not-leak-private-ssl-keys-after-all):

Apr 11, 2014, 10:09am EDT … the content distribution network CloudFlare has announced Heartbleed may not allow access to those private keys after all. In two weeks of testing, the company has been unable to successfully access private keys with Heartbleed, suggesting the attack may not be possible at all…

Update April 11, 12:08PM EST: Updated to acknowledge that private keys have been revealed by Apache servers on first request.

Update April 11, 9:39PM EST: Cloudflare now states that the Heartbleed bug has been successfully used to retrieve SSL keys, despite its earlier claim.

Performing threat analysis increases the cost to triage C_triage, so the situation may resemble the situation in Figure 2, and you should consider fixing weaknesses without triage.

Avoiding or delaying changes

Sometimes it is not easy to make fixes, or it may be almost impossible. Perhaps, the product is already in production and fixing requires manual updates. The product may also be certified, and making fixes would require expensive recertification.

In this case, you may want to investigate if a found weakness can be left as is. The situation is similar to Figure 3, where the cost of fixing C_fix is bigger than the cost to triage C_triage. However, if a new version of the product will be produced at some point, you should put the findings into the backlog and fix them from the development version.

Worst case scenario is that the system you are testing has both big C_triage and C_fix. The system is complex, fragile, and lacks relevant documentation. You do not know if a weakness represents a vulnerability and you do not know if fixing it causes problems elsewhere. In this case security testing is probably not the right thing to do, but the system needs threat analysis and refactoring first.

Defence-in-depth

Defence-in-depth principle means that we do not rely on a single defense mechanism to block attacks, but we have several independent security mechanisms in place. E.g. see SANS Whitepaper “Defense In Depth” (https://www.sans.org/reading-room/whitepapers/basics/paper/525):

Defense in depth is the concept of protecting a computer network with a series of defensive mechanisms such that if one mechanism fails, another will already be in place to thwart an attack.

Defence-in-depth principle promotes fixing weaknesses. It is not required to prove that the weakness can be exploited, as an eliminated weakness may be a hurdle that stops an attacker that has compromised the other defenses.

Triage as risk assessment

Triaging a weakness does not only help to decide whether to fix it or not but also gives insight into the security of the system. The insight allows to update the threat analysis and helps to make the system more secure.

This alone may be a good enough reason to perform triage.

Prioritizing fixes

When there are too many weaknesses to fix, it may make sense to start from vulnerabilities, and thus the triage is required. Not all weaknesses are equally difficult to fix, which should be taken into account in prioritization. When a weakness is cheap to fix (C_fix is low), then it should be fixed first.

Fixing for quality

Perhaps weaknesses should be fixed also to improve the quality of the system. Could a system riddled with weaknesses promote unsafe practices? An analogy can be taken from article by World Bank on neighborhood safety “Cleaner streets may be safer streets” (https://www.worldbank.org/en/news/feature/2015/06/09/por-que-las-calles-mas-limpias-pueden-ser-mas-seguras):

“A disorderly space with garbage gives the impression that people are not taking care of that place,” explains John Morton, World Bank environmental expert. Therefore, if a neighborhood tolerates disorder, whether that means throwing trash on the ground or dirty streets, it could — hypothetically — produce an environment more conducive to crime. For criminals, this type of disorder suggests that crimes will not be reported or controlled, in other words, that no one is in charge. “It is a symptom and also part of the cause; the neighborhood seems more defenseless,” says Morton.

Similarly, it can be argued that even without malicious intent a developer working with a system filled with weaknesses is less likely to use safe development practices. Seeing flaws all around promotes a “don’t care” attitude and proliferation of weakness through the copying of existing bad patterns of problem-solving.

A system that has a lot of issues is hard to (security) review, manually or automatically, as the unfixed weaknesses clutter the view and hide the vulnerabilities. In contrast, a system with no apparent weaknesses sets the bar high for new features to match the level of prudence. A security analysis produces a limited number of findings, which can be properly triaged and/or fixed.

Turning tables with attackers

It is often said that an attacker has an advantage over a defender as the attacker needs only to find a single vulnerability, but the defender must find them all. This is described in “Writing Secure Code, Second Edition” like this (https://flylib.com/books/en/1.287.1.17/1/):

The Attacker s Advantage and the Defender s Dilemma … Principle #1: The defender must defend all points; the attacker can choose the weakest point.

However, if we find and fix weaknesses, we can turn the table around. When the cost of finding a vulnerability is much greater than finding and fixing a weakness, we as the defender can have the advantage. The attacker needs to find a vulnerability, but the defender needs only to find weaknesses!

Acknowledgements

This work is done in the project SECREDAS (Product Security for Cross Domain Reliable Dependable Automated Systems) funded by ECSEL-JU (Electronic Component Systems for European Leadership Joint Undertaking) of the European Union’s Horizon 2020 research and innovation programme under grant agreement nr. 783119, and by Business Finland.

More…

There is a continuation blog “Reduce vulnerabilities by improving security requirements”, where I discuss how to reduce vulnerabilities when writing security requirements.