Seven Hard Realities of Application Security For Non-Developers — Part 2

Long the domain of ex-developers, application security as its own discipline is maturing and beginning to gain interest from traditional information security practitioners and information security leaders. These more recent converts to the app sec world and security leaders with these new responsibilities are often lacking application development backgrounds, but still are enthusiastic about wanting to expand their experience and influence into the shiny new appsec space.

The current thought leadership often omits or glosses over the complications and pitfalls that can make the journey to a successful application security program perilous. Not only do vendors drive much of the thought leadership in application security with mixed results but there are many other factors that can determine the level of success for any app sec devotee outside of simply the tools they buy and their level of enthusiasm.

In the first post, we covered the first three of the seven hard appsec realities. These 3 included the following:

#1 — App Security Success Is In The Ability To Adapt To Dev Culture, Not The Other Way Around

#2 — App Security Is More Than Scans And OWASP Top 10

#3 –The App Sec Team’s Dev Skills Largely Determine The Division of Work Needed With Devs

This is the second post of a two part series.

The author has been successful as a VP level CISO and director level leader in standing up new application security programs within multiple large companies. Each appsec program needed to support and complement an existing and mature application development organization. Though there are some hard messages, these posts come from a place of love for application security.

I’m intentionally attempting to be transparent to other CISOs and senior cyber practitioners in order to help change the balance of thought leadership around application security. Readers need to know the full story so we can change from discussions trying to sell products and services to a more broad conversation that is more inclusive of the range of obstacles, issues, and techniques that can help, rather than impede, an appsec program’s success.

In this second post, we’ll work to understand the last 4 hard realities.

#4 — App Security Tools Are Less Mature And Comprehensive Than Expected

#5 — Not All AppSec Defects Are The Same Or Even Are AppSec Defects

#6 — DevSecOps Is A Very Limiting Misnomer

#7 — Measurable Results Are What Matter At The App Security Program Level

Let’s jump right in….

App Security Tools Aren’t Very Mature

Application security is a fairly new discipline. There are some app sec tools that have been around for years, but the majority are fairly new. When we think of the maturity of app sec tools, the time that a given tool has been in the market isn’t the right metric. For the purpose of this post, “mature” can be defined as easy to install, highly automated toolsets without a lot of false positives. If we are using that definition across the entire appsec tool landscape, app security tools aren’t very mature at all .

First of all, these tools aren’t plug and play in terms of installation. They can’t be, even when the marketing brochure says otherwise. Like most enterprise applications, you probably aren’t going to simply double click a file, finish the install, and be ready to get right to work except for the smallest and most manual of app sec tools. Some coding will be required to stand up the tools and adapt them to your specific environment. Additional coding will likely required to automate app sec tools into your build process. That’s a fair amount of code that has to be written right up front.

If your organization’s application security team isn’t able to code what’s necessary for the installation, the team really only has the following three choices:

• rely on available bandwidth from the dev team to write the code
• pay for additional services to perform and customize the installation
• budget for and oversee a managed service provider who, in order to be successful, will still likely need to steal cycles from your dev team to understand the details of the build process during installation

Complicating things is that there are multiple types of application security scans:

• Static application security testing(SAST): White box scans focused on the code’s structure
• Dynamic application security testing (DAST): Black box scans that introduce faults to test code paths
• Interactive application security testing (IAST): Software instrumentation that finds code vulnerabilities
• Runtime application self protection (RASP): Instrumentation that can detect vulnerabilities in production runtime environments
• Dependency scanning to find vulnerabilities in third party libraries and licensing

If you are planning on more than one type of scan, your installation dependencies might multiply this time commitment by anywhere from 2x to 4x. We’ll get more into why additional installations may be required later in this post.

Once the team has the tools installed, your devs will want their code scanned as quickly as possible within the build process. They don’t want to be slowed down by any sort of security gate. The fatal flaw of manual scans in an otherwise automated build process is that scans can only occur when someone (as in “a person”) is available and willing to kick off the scans. This fatal flaw remains true regardless of whoever is performing the scans.

While this may work fine initially, eventually manual scans won’t be able to keep up. The more devs writing code needing to be scanned, the sooner manual scanning will become a blocker, rather than a value add to developers. Once devs don’t see value, their level of support may quickly wane.

From a senior cyber security leadership perspective, if code scanning is being performed on a part time or ad hoc basis, you probably don’t have an app sec program. It’s more like an occasional activity. When I led a commercial software development team at a major software company, the scans were usually run manually by an external partner just before we released. The defect count was usually in the 10s of thousands with the majority being false positives. The scan results would arrive in the last week before shipping the product. The results were always too ineffective and too late to make actionable.

Ideally, application security teams will want to have scans automated into the build process. Some vendors provide a build agent for doing this. Build agents can’t be deployed in an automated way and so will need to be coded into the build pipeline (see the paragraph above on installation).

Many app sec vendors do not provide a build agent for their product. If your vendor doesn’t provide one, you’ll either have to write your own or be limited to manual scans.

Historically, my app sec teams have found success by matching the dev team’s velocity. The best and most effective approach that has worked for my teams is to gain buy-in by enabling devs. Unfortunately, this may mean not moving forward with app sec products without build agents including a number of very popular and otherwise good appsec vendors.

So, now that app sec scans have been built into the build pipeline, you are probably thinking that you are ready to begin scanning. Assuming that you (or your app sec managed service provider) is scanning legacy code, be prepared for your very first appsec scans to take 48 hours or more to finish. Both you and your dev team will likely say, “Wait. What?!?” in unison when that happens.

Your vendor’s marketing materials probably didn’t include that nor did any of the blog posts or books that you may have read. You see, source code should ideally managed in a way that is optimized for rapid scanning. This may be very easy for startups. But, if you have lots of legacy code that hasn’t been historically scanned, the code is likely not optimized.

Managing source code is an area that an effective app sec team should be negotiating with the dev teams they support. Entire books have been written on this topic but suffice it to say that there is more to securing source code than configuring the security controls on GitHub. You’ll need to become conversant in methods to manage source code. Again, you can attempt to rely on the devs, but this will consume even more of their cycles.

Another complicating factor around appsec tools is that most tools only do one aspect of scanning well. I wish that I had better news for you but it is what it is. This view comes from someone that has tried a fair number of them and, unlike contractors, actually had to rely on their results long term. There are product suites out there that do SAST or DAST really well but the other product(s) they offer are barely above beta.

My current application security teams perform SAST, DAST, and dependency scans. We’ve tried most of the combined suites but decided for the short- to mid-term to focus on the tools that are best in class. This means that we had to deploy three separate solutions. Like us, you’ll probably need more than one tool set to really bring high quality application security scanning results. We remain hopeful that, at some point, we’ll be able to combine at least SAST and DAST into one solution without sacrificing quality of results.

So, now all the various tools are (finally) installed that you need to build a successful app sec program but there are still some significant process issues for which you’ll likely need to plan, prepare, negotiate, and agree upon. There are some very fundamental questions — who will scan (app sec? devs? QA? Devops?), Who will interpret results? What will be scanned? When will scans take place and where within the code base?

While an appsec team can never go wrong with focusing on the OWASP Top 10, there are a lot of other vulnerabilities and security controls needed to have a comprehensive appsec program. These decisions are all part of establishing appsec as a program.

Now, now let’s look at interpreting the scan results….

Not All AppSec Defects Are The Same Or Even AppSec Defects

In general, software code can have all sorts of defects from new functional defects to regressions to performance. Only a subset might be security defects. Your tools might be security focused but the scan results will likely expose a number of operational or quality defects that are not security bugs at all.

Definitions are important. One of the most fundamental tasks that you’ll have in standing up a new application security function is clearly defining what is meant by “security defect”. From a process perspective, you’ll also need to ensure that the defect intake on the part of the dev team is a common process for all types of defects. This way, your organization doesn’t have one way for dealing with defects found with security tools and a completely process for defects found by the QA or test team.

Modern app sec scanning tools classify the vulnerabilities they find in different ways — critical/serious/medium/low/informational, perhaps high/medium/low, or myriad other combinations. Results are organized according to the a taxonomy (or classification) defined by the vendor.

No one ever talks about the complexities of taxonomy related to application security, especially multiple toolsets. Few, if any, of the taxonomies are the same. You’ll need to be sure that particular vendor’s reporting taxonomy matches the objectives, controls landscape, and alignment of your organization.

Interpretation of scan results requires some level of expert judgment. For instance, an organization’s working taxonomy should account for the impacts of any existing compensating controls. Think of a scan as a risk assessment with a very granular view. Your scans might well unearth vulnerabilities that, in a vacuum or absent other data, might be critical or high. That said, the organization may have strong compensating controls in place that change or offset the appsec tool classification of the defect.

A SQL injection may not be accessible to exploit because of strong network controls. That doesn’t mean that the defect shouldn’t be remediated. It does mean that perhaps the “high” classification should be reconsidered and the defect prioritized relative to its actual level of risk. Others may disagree but prioritization and credibility are two key facets of a successful application security team.

Different tools might report the same defect or have different classifications for the same defect. The defect could be a false positive. You’ll have lots of false positives and will need a process to identify and eliminate them. The defects might be functional defects and not security defects. You’ll need to ensure that a process exists for the test/QA team to pick up responsibility for these software defects. My team built an appsec database that allows the team to identify these scenarios, de-dupe defect copies, and deconflict taxonomy differences. The database saves countless hours of manual work.

Lastly, someone will need to prioritize the fixes. Again, this may be the appsec team, the devs, or a third party. Any or all work, but the effort will need to coordinated and resourced regardless of where it lands.

Prioritizing security defects will be the subject of a future blog post. That said, 1000 valid defects might mean 995 code changes or might mean 4 with one code fix remediating 996 of the 1000. There are a ton of scenarios that will need to be accounted for in the prioritization. You’ll need to coordinate and integrate with the dev process as well. If the devs disagree with the prioritization, they simply won’t code the changes that are needed. Collaboration is a key skill in app security.

DevSecOps Is A Misnomer

DevOps, the mix of application development and infrastructure operations, has rightfully grown into a legitimate business discipline within the enterprise. As influence and funding for devops has grown, security practitioners have started to make an argument that security needs “to be part of” devops and has started trying to formalize the term, devsecops, for the following reasons:

- At a conceptual level: Teams need to build security into their DevOps culture to ensure secure software

- On the practical level: IT security must play an integrated role in the full life cycle of your apps such as automating some security gates to keep the DevOps workflow from slowing down rather than security being isolated to a specific team in the final stage of development.

Perhaps this effort at recognition as DevSecOps reflects a lack of influence or a lack of funding on the part of infosec. It could be that software dev culture has been seemingly impermeable in some organizations and this is the “hail mary” attempt at being successful rather than successfully negotiating as any other exec would do. I don’t know.

Here is what I do know.

The security discipline provides two key functions within IT or development organizations

• Operational capability building and monitoring
• Security governance and reporting

While I highlighted the first hard reality in this series as “app security success is in the ability to adapt to dev culture, not the other way around”, we need to interpret how to operationalize application security in a way that makes the organization and its business outcomes most successful.

I often half-joke that if we need to add additional functions to devops because they are important, it would make more sense to add performance (“devperfops”?) or scalability (“devscaleops”?) than security. Security is far less likely to be forgotten than either of these two equally important capabilities.

Back to being serious….

DevOps is a business process. Nothing more, nothing less. Infosec needs to have a common engagement, capability building approach, and monitoring visibility across all business processes from accounting to HR to IT. DevOps is not an exception or any different.

Perhaps, the list of previous hard realities that I’ve highlighted has made the process more difficult but resolving these is a function of leadership, negotiations, and team building. If these aren’t present in or part of a successful app security strategy from outside of devops, why should a leader believe that they’ll be more successful from inside? Successful devops requires coordination, role definition, and controls capability internally. Each of these domains will require the same level of leadership, negotiations, and team building.

Building security controls into the devops process is no different than building security controls into any other business process. If the leadership isn’t in place and aligned, the outcome will almost always be something short of success.

The second and more compelling rationale for not using DevSecOps is security’s governance function. It’s taken a long time for executives to understand that there is a inherent conflict of interest when the CISO reports to a traditional infrastructure focused CIO in many organizations.

History has borne out that teams can’t effectively govern themselves beyond the short term. By creating a devsecops function based on one small aspect of application security (i.e. building automated controls the build process), an organization will be diluting the governance function across business processes.

If execs don’t understand a process, they are assuredly not going to understand any nuance behind the process. I’m not sure that there is value or advantage in pushing the nuance of a conceptual construct like devsecops that negates both the enterprise role of security as well as key aspects of security’s governance function. By doing this we’ve effectively put security squarely back into the IT box.

But, even with the negative perceptions and tangible impacts to core infosec functions, devsecops does bring a shiny new cool name that sounds great on social media. So, there is that.

Measurable Results Are What Matter At The App Security Program Level

Application security is an investment. Like any large security investment, you’ll need to demonstrate that the new capability is providing value. This doesn’t have to be quantitative, qualitative value is just fine. You can turn application security into a business advantage.

If you can’t measure what’s being done within your appsec program, it’s likely not adding value.

Hopefully, these two posts can help prospective app sec practitioners understand those challenges up front in their planning and preparation so they aren’t later surprised during execution.

Good hunting!

For more insights into how cyber leaders can best enable the business and build rock solid cyber programs, please follow me on Twitter at @opinionatedsec1

You can also find more of my previous content at the “CISO & Cyber Leaders” publication on Medium: https://medium.com/ciso-cyber-leaders