Want Rugged DevOps? Team Up Your Release and Security Engineers
What do release engineers and security engineers have in common, and what does that have to do with DevOps? If you don’t have an immediate answer, you’re not alone. A ballroom full of people at the recent RSA Conference was preoccupied with this question as part of the daylong track titled “Rugged DevOps.”
If you’ve never heard of rugged DevOps, you’re not alone. It’s a relatively newer moniker for an emerging community of practitioners exploring the practices and tools needed to improve security for high-velocity software development and delivery. And it’s no wonder that RSA filled a conference room in an all-day session on the topic: It’s difficult to integrate security into the DevOps value stream in a way that doesn’t frustrate everyone, including the security professionals trying to keep users and data safe. And the fact that the security and threat landscapes are constantly evolving makes doing so in a sustainable way even more daunting.
So how do you ruggedize your own organization’s DevOps practices? The answer, it turns out, lies in getting your security engineers together with your release engineers.
A Tougher DevOps
The idea of rugged DevOps grew out of the rugged software movement, a group founded about four years ago with deep roots in the technology security community. Its Rugged Software Manifesto, put forward by Joshua Corman, David Rice, and Jeff Williams, attempts to bring light to what developers and software organizations often don’t adequately plan for: the fact that their software is likely to be used in ways they don’t expect or didn’t intend, and may be subject to active, unrelenting attacks by bad actors. Rugged DevOps challenges practitioners and organizations to face those realities and pledge to do something about it.
Rugged DevOps brings the lean thinking and manufacturing work of W. Edwards Deming that DevOps espouses to “ruggedizing” software. In the context of security, this means re-examining the supply chain of software you use to build your products on top of and working to build in security instead of trying to tack it on at the end in a last-ditch security review.
Why Release Engineers and Security Engineers Should Collaborate
Release engineers and security engineers have a lot in common. In both cases, their explanations of what they do and the value they provide typically range from mostly confusing to outright horrible. That’s due at least in part to the fact that the work they do is often opaque and steeped in esoterica. And while continuous delivery (CD) pipelines help with making this work more visible, it’s still often ignored or assumed to “just happen” in the software value stream.
Because security and release engineering often don’t have that direct cost-benefit connection, these teams can get budgeting-table scraps, both in terms of money and time in the delivery schedule. Then, when security breaches happen or releases get delayed, these failures get attributed to these teams. Suddenly, everyone cares a lot about the work that security and release engineers do, and those teams are elevated — to the front of the firing line.
Additionally, even though they don’t know it, they often care about the same things. At the RSA Conference, a security engineer relayed a story about a bank that used a particular Java library. Across all of its products, the developers used 60 different versions of this library. Of those, 57 had published security vulnerabilities. Horror stories such as this keep release engineers awake at night as much as they keep security engineers staring at that dark ceiling. One vulnerable library in your product is a security problem, but multiple versions of a vulnerable library in your product is a release engineering problem, which is yet another reason why security and release engineers make not-so-strange bedfellows.
Why should your release engineers help to usher security folks to the DevOps party? Because one of the best ways to introduce security work is by tactically integrating it into the software delivery value stream, usually at critical points in your continuous delivery pipeline. Your release engineers should be working on a CD pipeline so they’ll have good ideas about where and how to get security into the mix to start making your DevOps rugged.
3 Ways to Ruggedize DevOps
Here are three concrete steps can you take to ruggedize your DevOps and start actively integrating security into your CD pipeline:
- Introduce your release and security engineers. Chances are, these two kindred spirits aren’t aware of each other in your company. Introduce the two teams over a lunch or informal coffee, so they can trade war stories. The overlap between those stories is likely to surprise both sides.
- Ask the two engineering groups to research your software supply chain. One of the core tenets of rugged DevOps is that you should know where your software components come from and reduce those components to a minimum number in your software supply chain: the group of well-known, high-quality vendors with which you work. A good first step here is to have your release and security engineers look at your flagship products and determine which libraries they use. This type of excavation is where release engineers excel and security engineers know how to make sifting through the results more actionable. That might mean addressing an incompatible licensing issue or updating to a newer version of a library that has had a critical security vulnerability fixed. Finding out what code is lurking in there is likely to be a revelation, and the exercise is highly valuable to boot.
- Create a project that involves other teams: After you’ve started to get a handle on your software supply chain, have your release and security engineers start a project that has a clear, known benefit but requires help from developers or operations. One of my favorite projects I do with clients is to set up a continuous integration environment with compiler warnings cranked up and set to fail the build. I’ve run this exercise with teams in the past, and when developers sat down to look at those warnings, they discovered that more than half were legitimate bugs (usually in the form of logic-corner cases). Of those, about one quarter were security-related. In addition to providing obvious value, getting to a build process that has zero-warnings has been surprisingly satisfying for the developers, security, QA, and release engineers alike.
Make Those Introductions
Rugged DevOps is about everyone involved in the software delivery value stream taking responsibility for security-critical aspects of the product. Our colleagues and customers place their trust in us. The stakes are admittedly high, and the task is daunting.
Getting your release engineers acquainted with your security engineers to understand and tame your software supply chain is a great place to start. Getting them working on high-value projects is one of the simplest and most effective ways to start providing a strong foundation for rugged DevOps. It also paves the way to involving the rest of your development team in helping to improve your own software’s robustness while bringing lasting progress in the security arena to your DevOps strategy.