Getting Rugged DevOps Right

Two Perspectives

Jack, an accomplished application security pro, tells me, “The developers won’t talk to us. It’s like we speak a different language. They are releasing new builds so fast, how could they check each one for security vulnerabilities? We can’t move as fast as they do.”

Then in the next moment, Diane, a DevOps pro, let’s me know, “Our current security team’s tools and practices can’t keep up with the pace we have now established. If it doesn’t fit, it doesn’t get done here.”

I have heard this little ditty ‘bout Jack & Diane so often in the past two years, it could be a hit record. I imagine you have heard it too.

The Slow Lane Won’t Lead to Rugged DevOps

Let’s visit Jack’s world, first.

When visiting a well-known bank last month, they shared insight on their current application security practice. They have 10 full-time people employed (offshore where personnel costs are lower) to analyze a set of 100 applications. The bank has over 2000 applications, but the security practice can’t scale to access their entire portfolio. In 2.5 years, they had only been able to cover 5% of the application portfolio.

The application security team employs an application security tool originally built for waterfall-centric development timelines. As much as they want it and need it to move faster, it is not really for “continuous” velocity.

Each scan of an application can require anywhere from four hours to two days to get an assessment report. The reports average about 30,000 potential defects which then require further analysis. That analysis may take up to two weeks in order to sift through thousands of false-positives and false-negatives. While this security practice is critical to the bank, it cannot keep pace with development practices that are aiming to go much faster in order for the bank to remain competitive in their industry.

The Continuous Gap

Sound familiar? It should. This scenario is playing out all across the DevOps and Continuous Delivery landscape. It is reflective of what I call “The Continuous Gap”.

In Diane’s world, DevOps teams pursue “shift left” strategies in an effort to build quality in from the beginning. Shifting left enables them to bake in the right code, components, configurations, and practices from the beginning. When done right, velocity increases dramatically, while operating costs are minimized, and unplanned/unscheduled work is reduced. Investments can then shift more in favor of innovation over upkeep and maintenance.

The Old-School Rub

Jack’s world of traditional application security practices often wait to check for security vulnerabilities at testing or release stages of the SDLC. There are two reasons for this: (1) some security tests like static or dynamic analysis can take hours or days to perform and (2) security teams cannot embed themselves earlier in the SDLC (due to political, organizational, or operational obstacles) without dramatically slowing development in a high velocity world.

If you are using old-school application security tools, you will continue to get value from them. At the same time, expect continued blocking if trying to get more deeply entrenched in development. New velocity requirements call for new, complementary approaches to application security.

The New-School Shifts Left

A number of newer application security solutions and practices been introduced recently that are development and developer-centric. For example, imagine if build managers could analyze every open source component used in every build for known security vulnerabilities.

What if, developers could access information inside their IDE that would inform them if a component they were planning to use in an application was known to be vulnerable? What if developers had access to spell-check like alerts of security issues as they were creating their code?

Furthermore, imagine if information was available that not only told the developer of a known security defect, but directed them to an alternative safer version of that component to use. Components would be automatically vetted against corporate security policies at the instant they were chosen by a developer. When 90% of an application is composed of open source and third party components today, automatic analysis early in the lifecycle save millions of dollars. This how Rugged DevOps rolls. This is Diane’s world.

But Diane’s world (and your’s for that matter) are not just about increasing velocity. All organizations need to consider operational costs to both detect and remediate security vulnerabilities in their applications. We all realize that the further along we are in the application development lifecycle the more costly fixes become. Feel free to apply your own cost figures to the chart below, but you should also note that these calculations do not include breach cost, help desk calls, maintenance, time-to-market, reputation or stock price.

Rugged DevOps: Shifting Gears…Quickly

Shifting from old-school to new school approaches doesn’t take long. On a recent visit to one of the largest insurance companies in North America, they shared their tale of transition. Of five major applications that run their business, one in particular had 40,000 different files that needed to be analyzed. Their old-school approach involved a lot of manual code investigation and required about two-and-a-half weeks to complete the analysis of that large application. The new approach that used automated analysis of security vulnerabilities took two minutes — and produced identical results.

A government tax office wanting to shift security analysis of applications from hours (outside of development — Jack’s world) to seconds (inside of development — Diane’s world) was able to complete the first pass analysis of their existing portfolio of 600 applications within two working days. The security team was not only faster at identifying known security vulnerabilities in applications, but could also quickly identify alternative open source and third-party software components that were safer to use. The security team was not simply discovering issues, but the new class of Rugged DevOps solutions were aiding in guided remediation.

Questions to Ask About That Rugged DevOps Solution

There are several companies and open source projects that now offer application security solutions that work at extremely high velocity. When exploring solutions to match the velocity of your Rugged DevOps practices, consider asking the following questions:

1. How long does it take to complete the analysis of a small, medium, and large application?
2. Do developers use these tools, or are they only meant for security personnel? Do you have members of development teams that act as positive references for your tool?
3. Does the analysis require further investigation of the findings, or does it pinpoint the root cause of the issue?
4. Does the solution only identify vulnerabilities or does it help guide remediation of issues?
5. What is the balance of false positives to positive alerts generated by the solution?
6. How many people are employed to support the solution in environments similar to ours?