Finding More Bugs Won’t Fix AppSec
In the 2018 Verizon Data Breach Investigations Report (DBIR), web applications were the number one source of successful attack breaches, accounting for one in every five breaches. However, we believe this is a drastic under-representation based on how they qualify incidents*.
You’d assume this clear and worrisome trend would be ringing alarm bells for the industry and the authors of the report. You’d expect recommendations along the lines of:
“We need to seriously rethink how we’ve been approaching how we secure our applications.”
Or:
“We need innovation in appsec or this trend will get much worse in the future.”
But, nothing. No recommendations were made, and very little commentary was even paid to this disturbing trend. This lack of focus was similar in previous reports — but it’s not just the DBIR. Industry reports from SANS show that despite 30–40% of all breaches are coming through the application, appsec is underinvested in. Investment and focus aren’t going to where the breaches are happening and this is both puzzling and concerning.
Why aren’t we pushing hard to evolve and improve how we defend applications?
From my experience, it seems to be a bit of pack mentality (i.e. “no one else is investing in appsec so we don’t have to either”). However, the bigger challenge is that historically, security teams have had a very hard time collaborating on security with their application developer counterparts. To make matters worse, the app dev groups are adopting cloud architectures and DevOps processes, adding additional hurdles to embedding security into application development.
How have we approached application security in the past and how can we make it more effective while also making it app developer, cloud, and DevOps friendly?
The problem of bug pile-on and prioritization
A major focus of application security has been to generate lists of bugs and vulnerabilities and pass them to the developers to fix. Long-time application security leader, Jeremiah Grossman dug into this in his latest blog post, “All these vulnerabilities, rarely matter.” Grossman explains that Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) tool vendors have been successful winning deals when they generate longer lists of bugs than their competitors.
Pen testing is similar where, in general, the more bugs they identify in their reports, the better. As Jeremiah points out, while longer lists of bugs may help a vendor win a contract, longer bug lists may actually be counterproductive to improving application security. The core problem in appsec is not that we aren’t finding enough bugs to fix. The problem is about prioritization — and there hasn’t been a good answer for how to address it.
Think about it from the developers’ point of view. Application security teams are identifying and assigning you a never ending list of bugs to fix. So is everyone else. The QA team finds performance issues. The customer support team creates tickets reported from customers. The product manager is pinging you for updates on the bugs she assigned to you last week. And so on.
The point is, security flaws are just a subset of bugs they have in their backlog.
But the appsec person thinks, “These vulnerabilities might be exploited at any time! Why aren’t we urgently fixing this! These developers must need more security training!”
The customer support representative who continues to get angry emails from a frustrated customers running into a bug stopping them from critical business functionality feels similarly: “Our paying customers are upset and we know the problem! Why hasn’t this been fixed yet?”
Same with QA. And on and on.
Here’s how a developer might respond, likely only to themselves, to the appsec team’s list of bugs: “You didn’t have to buy a tool or pay a pen tester to tell me I have bugs in my code! I have tons of bugs in my backlog, and most of them are hindering important business functionality. Why should I care about these bugs compared to those ones? These are all just theoretical risks right? 80% of these aren’t even exploitable and how do we know anyone is even attacking us in the first place?”
The challenge for appsec is prioritization, not the total number of bugs. So how do we help get the right bugs prioritized?
Here’s a conversation I regularly find myself having with security teams.
Me: Is your goal really to find all the bugs, fix them all, and do that all before they go into production?
Security: [After thinking for a second] Well, no, of course not.
Me: Ok then you have to assume you have vulnerabilities in your production code at all times correct?
Security: Yes.
Me: Then why are you not obsessed with getting visibility into where your attackers are attempting to breach your code so that you can know if one of those vulnerabilities is being breached or someone is attempting to do so?
Security: “Because I didn’t think to do that,” or “because I didn’t think that’s possible,” or “because I don’t know how to do that.” (Note: it’s never because they say that’s not something they don’t want or need.)
Me: This will inherently change the conversation with your developers around application security and how to think about bug prioritization because — with this visibility — you have a real (not theoretical) threat on your hands that everyone can see and justify.
The value in seeing the “how and now”
If you can see all your attack traffic and understand what parts of your app it’s focused on, then you can start having data to build an informed sense of the probability of bug exploitation. The real attack is not a list of theoretical vulnerabilities nor is it something that you’ve paid someone to find in the code. It’s an actual attacker attempting to breach the code the developer wrote right now.
We’ve found real-time attack monitoring grabs developers’ attention.
They start proactively reviewing the attack data and pulling it into the DevOps toolchains they are already using to monitor other aspects of their apps. When they’re focused on understanding how their application is being misused and attacked, it naturally facilitates the discussion between infosec and app dev managers about what parts of the code base you should be proactively hardening (e.g. the ones with the highest risk which have most critical bugs and are highly targeted by attackers).
We’re stronger together
We should always do our best to find and solve as many vulnerabilities as we can, but we have to assume we can’t find them all. Given that assumption, visibility into what attackers are attempting against our systems becomes the cornerstone of how we build our defensive strategy and prioritize our very limited development time.
Don’t agree with this approach? Excellent! I actively want to engage in discussion on the topic because, as an industry, we’re losing the security battle against our attackers at the app layer. As Grossman notes, “If we really want to push forward our collective understanding of application security and increase the value of our work, we need to completely change the way we think.”
Whatever solutions we come up with, to be successful, they will have to come from security and app development teams working together. Your tech teams own the app. If they’re not bought in on an approach and solution, nothing will change.
DEMAND MORE FROM YOUR WAF
Demand more from your WAF
Companies like Duo Security, Chef, and more trust Signal Sciences to protect their most critical web apps, APIs, and microservices. Find out why.
* as noted in appendix H from the 2018 DBIR: “This year, like last, we removed several thousand (23,244 but who’s counting) incidents where web applications were compromised with a secondary motive. In other words, they are compromised to aid and abet in the attack of another victim. They are legitimate incidents but are light on actionable details such as the variety(ies) of hacking used to gain control of the asset. In addition to these concerns, we also cannot confirm if they were organizational breaches. So rather than analyze them as part of the main dataset, we call them out here.” 23k incidents compared to the base of 53,000 incidents and 2,216 confirmed data breaches shows just how many web app incidents and breaches occur regardless of their motive. This confirms what most appsec professionals know to be true that the web app is often the critical first phase of an attack chain even if it’s not the end point to where the attacker is exfiltrating data.
