When Dev, Sec, and Ops play nice together, they can build excellent software

Yes, it’s possible to build high-quality software products at the speed the market demands with security built in. But doing the security component effectively requires automated testing tools and policies. While the human element will always be necessary in some areas, manual everything won’t cut it.

That’s the key takeaway from a recent survey by the SANS Analyst Program. It concluded that while it takes a significant, ongoing investment to bring together the three teams involved in building software products — development, security, and operations (DevSecOps) — “the benefits are well documented.”

The survey, titled “SANS 2022 DevSecOps Survey: Creating a Culture to Significantly Improve Your Organization’s Security Posture,” relied on responses from 341 security professionals in organizations ranging from fewer than 1,000 to more than 50,000 employees, and from a variety of industries and locations globally, although 80% were in the Western Hemisphere. The survey was sponsored by the Synopsys Software Integrity Group (Disclosure: I write for Synopsys).

Why should you care? For the same reasons you care that your vehicle is built with quality parts and safety features. Your safety is at stake. Software is embedded in your life — even if you don’t create it, you rely on it. It’s behind every app you use, every “smart” device in your home, just about every transaction you conduct, the operation of the business where you work, and the school you attend.

It’s in your car, it controls the traffic lights on the road, and those flashing signs that tell you how long it’s going to take you to go the next 10 miles. It’s the brains behind the utilities you rely on for heat, water, and light. And more.

But if that software contains vulnerabilities that criminal hackers can exploit, not only can it undermine all the conveniences software provides, it can also hurt you in multiple ways — financial, personal privacy, and physical.

Indeed, it doesn’t really matter how cool and edgy a product purports to be if it doesn’t work as intended. It doesn’t matter how many bells and whistles it has if it’s not secure and ends up exposing your personal and financial information to the world.

Which is why it’s so important that those three teams work well together. It might seem that would happen naturally given that they are mutually dependent in reaching a common goal — the creation of quality products that are safe, secure, and irresistible to customers. But the reality is that they aren’t always as cozy and cooperative as the DevSecOps abbreviation might imply.

Teammate tension

Because even with mutual dependency, there is a natural tension between Sec and DevOps — one that has been dissected at security conferences for more than a decade. The major pressure on the security team is to make the software in a product as bulletproof as possible by testing it at every stage of the software development life cycle (SDLC). The major pressure on developers and operations teams is for speed — get a product to the market before the competition does.

Developers have responded to that push for speed — deployments have increased exponentially over the past decade. As Rod Musser, product manager within the Synopsys Software Integrity Group, put it in a recent edition of the Synopsys video blog AppSec Decoded, “Organizations used to deploy applications once a year or once a quarter. Now they deploy new apps and updates several times a day or even several times an hour.”

So understandably, developers don’t want anything to slow them down, and for years the perception has been that security testing does just that — creating “friction” in the SDLC.

But security teams have been working just as hard to keep up with the need for speed, through automation. As the SANS survey found, the only way to keep pace with an exponential increase in the pace of development is to automate both testing and policy. James Rabon, senior product manager with the Synopsys Software Integrity Group, noted in a presentation on the survey that, “Automation is king, and the only way forward for DevSecOps.”

The good news is that automation is available. Even better news is that 83.3% of survey respondents said they have “build automation,” and the percentage of respondents reporting that they consider “automated test coverage” to be a key performance indicator (KPI) jumped from 28.4% to 45.1% in a single year.

Automated testing tools can conduct static and dynamic application security testing that, respectively, expose defects as code is being written and as it’s being run. Another tool, software composition analysis, helps developers find and fix known vulnerabilities and potential licensing conflicts in open source software components.

Right tool, right time

A newer tool, application security orchestration and correlation, can be configured to do the right test at the right time at any point within the SDLC, depending on the needs and priorities of an organization.

And policy-as-code, lets the security team create digital guardrails that, among other things, prevents developers from getting overwhelmed with notifications about trivial defects.

All that helps eliminate the friction that can slow development. Indeed, finding and fixing defects early and throughout the SDLC is both much cheaper and much faster than doing it at the end. A widely cited finding by Ponemon Institute Research is that the cost of fixing a quality or security defect during coding costs about $80. It rises to $240 if it has to be done during the build, to $960 during the quality assurance and security phase, and to a staggering $7,600 during production.

That ought to eliminate any doubt that, as the survey notes, embedding security into DevOps pays tangible dividends.

Of course, there is always room for improvement, and the survey yielded a number of recommendations to help DevSecOps function more efficiently and effectively.

• Cloud benefits and risks: SANS says cloud-managed services generally provide improved security and financial benefits that DevSecOps teams should explore. And Thomas Madden, managing consultant with the Synopsys Software Integrity Group, said a trend toward organizations using multiple cloud providers amounts to “not keeping their eggs in one basket. It provides the benefit of increased availability should one provider be negatively impacted with an outage, which has occurred several times over the past few years.”

But the SANS report also notes that as organizations move toward using multiple cloud hosting providers “the work of securing each cloud environment increases exponentially. Organizations should consider using or increasing their adoption of commercial or open source CSPM [cloud security posture management] software to ensure that each cloud infrastructure is secure.”

• Go agnostic with tools: An organization’s testing policy should be able to work seamlessly with different tools and vendors. “Different security tools are required to find modern software weaknesses and vulnerabilities,” Rabon said. “Policy should be clear to anyone writing software the enterprise requires to be fixed and when, regardless of the tool required to find the security issue.”
• Limit language barriers: There are dozens of programming languages out there, and the SANS survey found that the riskiest are those used for web application development. So while it’s important to integrate testing tools like static and dynamic analysis into the SDLC, that works “only if the programming languages being used are supported by the tools.” Otherwise it has to be done manually, which is much too slow.
• Shared responsibility: One of the major mantras in software development is: “Security is everyone’s responsibility.” According to SANS, that mantra can become reality by “leveraging documentation, training, and the socialization of DevSecOps best practices.”

Madden added that shared responsibility “removes the finger-pointing that often is associated with a more siloed approach, so teams are more likely to assist one another in learning how to identify weaknesses in their respective applications. Longer term, it leads to more efficient development via a cohesive team structure.”

• Evaluate, evaluate: It’s not enough simply to measure performance if you’re not measuring the right things. For example, tracking the number of open (as in, not fixed) security vulnerabilities is good. But it’s much better to track how many of those rank as trivial, severe, or critical. That way, the DevSecOps team can focus on what really needs improving.

Finally, the SANS survey recommends three key ways to make rapid DevSecOps advancements. “(They) should consider implementing immutable infrastructure provisioning, blameless retrospectives, and chaos engineering to accelerate learning while improving confidentiality, integrity, and availability.”

What do those mean? Madden offers a brief explainer.

• “Immutable infrastructure provisioning is straightforward, consistent, standards-based information technology infrastructure that has been tested and retested and can be easily provisioned or deployed by an organization’s cloud team.”
• “Blameless retrospectives is an ongoing learning and improvement process. Errors, when they occur, are used to improve the overall development process, not to point fingers and assign blame, which is grossly ineffective.”
• “Chaos engineering is an ongoing testing process where errors and faults are intentionally introduced to expose faults within a system or application and validate that system’s resilience or expose its flaws so they can be fixed.”

Those, the SANS report concludes, can help organizations “focus on the path to DevSecOps excellence.”