Shifting Security “Down,” Early
Making plans and conceiving projects involves many agreeable sentiments. Those with the strength to be nothing but contrivers of plans all their lives would be happy people. But one must occasionally have a rest from this activity by carrying a plan into execution–and then comes anger and sobriety. — Friedrich Nietzsche
According to Ponemon’s Cost of a Data Breach 2023 report, DevSecOps adoption led to the highest cost savings for a data breach compared to any other practice or trait in a security program. While the value of “shifting left” with security is undeniable, the reality is that many teams end up hitting a wall very early on. At Google Cloud Next ’23 a few months ago, I had a chance to meet with security leaders who had experienced a successful DevSecOps adoption and those who were stuck or on the verge of abandoning their plans. Most of them faced the same initial challenges:
- A barrage of high-severity vulnerability alerts, most of which were false positives.
- The conflation of “severity” with actual risk at all levels of the organization.
- Other bottlenecks that could potentially worsen development and security instead of improving them.
So, what was the difference that helped those who succeeded? Despite setbacks, they almost always had an executive sponsor committed to a long-term plan. At first, that may seem to go against the ideological valence of “cultural change” that pictures different groups of engineers independently breaking down barriers and becoming friends (although that does actually happen). They still need to have a high-level leader committed to getting to a certain end-state, willing to provide air cover, able to understand initial setbacks, and, of course, able to relate to the business what the eventual cost-benefit of DevSecOps will be.
So, what’s the opposite image of that type of leader? One who is:
- Counting on seeing immediate value from shifting left
- Does not realize that “adding security” can, at least initially, significantly slow down all of the incredible speed of a modern CI/CD pipeline
- Frames the implementation as “bottom-up,” but in a way that merely shirks commitment and leaves engineers confused about their priority. Is it speed? Frugality? Is it creating a bunch of new tools instead of buying them?
The neat thing was that in cases where a leader realized they were falling into this pattern, there was still a chance to recover and learn in ways that helped them anticipate even more significant difficulties. The ones who recovered would immediately start to plan how to mitigate the burden on developers and security teams. Besides merely “shifting left,” they described what Richard Seroter generally refers to as “shifting down” in The Modernization Imperative. That article offers a high-level strategy to maximize developer efficiency and innovation. I wanted to say more about security at a practical level, at the very early stages of adding static testing to software development.
From the Start
Beginning is the time for taking the most delicate care that the balances are correct. — Princess Irulan, Frank Herbert’s Dune
The beginning hurts the most for the companies I’ve met who have attempted to integrate SAST, SCA, and other capabilities. Initially, there’s excitement about the potential to catch vulnerabilities much earlier. If we manage to shift left, there is a sense that we automatically win from the get-go. It is too easy to take for granted the value realized in this graph:
But then, reality hits. The slick, 15-minute-or-less build to deploy now gets mired in warnings that your team can’t possibly address in time. Security, it turns out, requires deliberation and applied knowledge of your application and your business that doesn’t match the speed of the rest of the development process. If you look at OWASP’s DevSecOps Maturity Model, you can start to anticipate many of these issues looking at the less mature stages:
- Early on, you don’t break the build on failed tests–because there will be too many failures whose validity is unknown.
- You enable software component analysis (SCA) and static analysis (SAST) starting from the server side, without any configuration unique to your application, and then slowly add breadth (to the client side) and depth (with better configuration).
- You start at high-intensity testing and progressively turn tests off as you discover they are less critical.
TL;DR, it’s going to be very noisy and unreliable at first. Even with the wisdom that this initial stage is only temporary, it can lead to alert fatigue, burnout, and, ironically, a decline in security posture and the creation of additional project risks.
With the End in Mind
The importance of having a security vision can’t be overstated here: everyone, from the engineer wrangling tests to the C-level describing risks to the board, should have a general sense of how their efforts, over time, are going to lead to much less painful, more valuable state in the future:
But this is almost impossible if we ignore the developers’ time and cognitive load, i.e., if the security risk mitigation leads people to overlook the growth risk they might inadvertently create. Imagine this conversation:
“So I see we added static testing, and that’s generated a lot of vulnerability notifications. It also delayed our latest release by a week; even minor iterations now take hours instead of minutes. What did we gain for it, though?”
— “Well, we gained security. We would otherwise be shipping software with lots of vulnerabilities. Right?”
“Yeah, but how bad were the vulnerabilities?”
— “Oh, pretty bad. 10% were CRITICAL, 15% were HIGH, and even the MEDIUM ones could be chained together, so overall, an orange-meets-crimson red, if we used a color wheel.”
“Okay, well, the delays cost us approximately $2 gazillion. How bad would it have been if all those CRITICALS got exploited?”
— “Oh, I mean, any of them was likely or highly likely; look, it said so in the dashboard with CVSS details. Just one of them would probably cost us about, I don’t know, take a wild guess–$10 bazillion?”
“How do you determine likelihood? Can we see all the possible scenarios, what the impact would be, and some justification for why or why not it could happen?”
— “Impossible to really know. We go based on the score in the dashboard. There are too many issues for us to spend time getting into an academic debate about everyone. So even if it hurts a little, we trust in the process.”
In the past half-decade or so, the thinking pattern in that conversation is now facing fresh criticism. (It met challenges before, but the solutions for prioritizing risk from 12 years ago never penetrated modern dev teams, who are now the ones playing catchup.) The challenges are becoming more apparent, especially in the past couple of years, from the organization maintaining the CVSS specification stressing that severity cannot be mistaken as risk to more data-driven descriptions of likelihood from EPSS and CISA KEV to Google Cloud adding VEX statements.
But What are Developers Seeing?
Despite the advances in helping organizations think about and measure likelihood, exploitability, and so forth, you have to ask what the people who are actually writing the software tend to encounter when they go to a trusted source like OWASP:
Generally, identifying whether the likelihood [of a potential risk] is low, medium, or high is sufficient. — OWASP
This is precisely where the gap exists.
Does [OWASP’s] phrase, stating “low, medium, or high is sufficient,” need to be taken on faith? Considering the critical nature of the decisions such methods will guide, we argue that it should not. — Douglas Hubbard and Richard Seiersen, How to Measure Anything in Cybersecurity Risk
The reality on the ground (despite the years of many security blogs crying out in protest against it) is that dev teams are still sorting and patching purely based on uncritical, faith-based severity scores, with almost no attention to context-specific likelihood or impact.
Regarding risk prioritization, part of “shifting down” is not reducible to tooling. However, there are great products addressing parts of this issue, such as Silk Security, which has a configurable risk model based on the asset and environment, and Mondoo, which can tell a developer very early on in their IDE if a new line of code would violate compliance necessary to do business.
Shifting down occurs very early when setting requirements: it means picking goals, frameworks, and measurements that stem the need to “invent” prioritization well after getting inundated with tons of noise. Avoid cognitive overload by thinking about what can cause it and mitigating it.
Shifting Down on False Positive Analysis
We suffer more in imagination than in reality. — Seneca
I am going to skip the litany about false positives in general. Last year, I was a guest on a podcast with Anton Chuvakin, Tim Peacock, and Steve McGhee in SRE Wisdom for Security — Alerting, where we unpack this problem in detail. Those lessons, in part, address what Michele Chiburka recently described in Fear and Loathing in Security Dashboards–that appeals to fear often backfire–and we have the evidence.
So, what’s the practical issue with SAST and SCA? The pain point is that with many open-source and free SAST and SCA tools that teams use, there will be a ton of false positives. In a (somewhat superficial) effort to “improve security early on,” we have suddenly created an authentic and significant project risk. The content of these alerts, when it isn’t perplexing even to senior developers, still takes time to analyze.
My first question to a colleague at a large multinational company planning to implement SAST earlier in their dev pipeline was, “Who exactly does the false positive analysis?” The DevOps lead exclaimed, “Exactly!” They subtly meant that we were suddenly creating a new, vast, likely underappreciated burden, whether it was developers or security engineers.
How do we shift down here?
- An upstream that introduces fewer issues to begin with. Developers should not be hounding down every dependency or worrying that maintainers might neglect security testing. Using Assured OSS, for example, a developer can pull in packages in which Google finds and fixes vulnerabilities, including AI/ML projects such as TensorFlow.
- Software component analysis that tracks the dependencies of dependencies and focuses on alerting on what part of the package we use, not just the version. Endor Labs has an excellent breakdown of why SCAs tend to be so noisy and a way to reduce them to a much more manageable level in Google Cloud.
- SAST tools integrate easily into the build process with minimal friction. Plenty of great SAST tools exist, but the integration piece alone can significantly burden a dev team. GitLab has a SAST capability using other scanners, from Kubesec to Semgrep, that supports many different languages, can take in rules from multiple sources, and can automatically resolve issues that are not relevant. You can add that as the initial step to a build process going to Google Cloud.
Those are just a few practical examples, but here is the guiding idea: shifting security down is a matter of reducing risk–not just security risk, but also growth risk. We need to ask: is there something we are asking developers to spend time on related to security, where we don’t know the actual value in terms of the risk we’re mitigating? Is there some aspect of testing that is producing too much noise? Have we considered more managed tools, products, or practices that can reduce it, and do we have people with the leverage to see this step as part of our vision for what we want security to look like? Find me in the Innovator’s Hive at Google Cloud Next in Las Vegas this year to share your stories with me and figure out other ways to shift down.
