Security Tools : Why more findings are bad for your security program
More Results or Better Results?
A security tool should help advance a security program. Any good tool makes work easier, not harder. So when I look at a potential security tool or service I ask, will this tool decrease a manual security effort and help us be more efficient? If the human cost required to operate a tool or validate the tool’s output is high, then the tool has done little to help build a scalable security program.
Accuracy is the most important
Security vendors love to try and sell the “silver bullet” of security. This single solution will “find all your vulnerabilities” or “secure all the data”.
When building a security program I need to design for massive scale. The answer is not more humans and more alerts, the answer is scale through highly accurate automation. How does a vendor solution fit into this approach? I value the accuracy of their product above nearly all other things.
The reason for this is actually pretty obvious when you look at the impact of the results.
- True Positives → These are valid issues and these deliver immediate value and a direct security benefit
- False Positives → These generate more work for my team, decrease trust in the tool and can result in loss of trust from partner teams
- False Negatives → Acceptable, I’ll find other solutions or approaches to accurately identify these remaining issues
Quantity or Quality?
Lets imagine two security products, A and B, listed in the graph above. We capture their results in a head to head competition of identifying security issues in the product’s target space. In this bakeoff, I’d go with product B. The true positives might be less, meaning this approach won’t catch all of the valid issues, but product B is decreasing my work while providing value with minimal additional human effort for validation. This allows the team to focus on other items that require human inspection or to find the next tool/service/approach that can address the residual false negatives.
Product A on the other hand actually increases the work required from my security team because the team will have to inspect and validate all of the results. Alternatively, you could skip validation and throw all the results to the responsible team to fix the issue, but you’re just destroying the credibility of the security team since over half the results are wrong.
What about the False Negatives?
With product B we will be missing valid issues. That’s fine. We shouldn’t strive for a silver bullet at the expense of accuracy. Instead, a highly accurate solution is one piece of a larger solution and other approaches or tools will be identified to address the residual issues.
There are of course scenarios where this model doesn’t hold and you will accept any number of false positives in search of the true positives. However, I’d argue the vast majority of security programs will benefit from product B rather than product A.
Is More Better?
Quantity is not better than quality. To build a scalable security program you must find solutions that reduce work, not increase it. Find the right solution for a subset of the problem space. Ensure it can scale and be accurate with the findings and then move on to the next challenge. This approach allows continued progress without overloading your security team to hand hold security tools that are supposed to be “helping”.
Like this blog? Follow me on medium, twitter, or signup to receive new posts via email.
