Rugged DevOps: Less Capture the Flag, More Collaboration

At the recent DEVNEXUS conference in Atlanta, I caught up with Chris Corriere to talk about his experiences in the realm of Rugged DevOps. We discussed automation, culture and collaboration, and which thought leaders he is following. Chris also shared insights on the upcoming DevOps Days Atlanta conference where he serves on the organizing committee.

Chris Corriere: Hey, I’m Chris Corriere. I’m a DevOps Engineer AutoTrader and I write for DevOps.com.

Derek Weeks: Today we’re going to talk about Rugged DevOps. It’s a subject that’s gaining a lot of traction in the community but not a lot of people are really familiar with what it is.

DW: What’s your definition of Rugged DevOps?

CC: Rugged DevOps to me is definitely manifested in the security sector but it’s a shifting quality and awareness left in doing the best we can with the tools we have, the constraints we’re given, and in the environment we’re in. It’s about being more experimental, understanding that we’re going to encounter failure, and learning to adapt accordingly. We’re very much having to make tough decisions in real time. If you’re running anything in production these days, you need to do that as safely and as consistently as you can.

DW: Is Rugged DevOps only about security?

CC: No. Security is really about mitigating risks. I think a lot of this ties into situational awareness about understanding your company’s risk. Is it user data? Is it credit card transactions or do you have a Russian botnet running in your infrastructure that you’re not aware of? What’s the asset you need to protect? Are you trying to protect an asset that is not there?

You can’t have security come in and become suffocating where it’s not needed. In other situations you can’t leave things wide open that really do need to be protected because you’re more concerned with moving quickly than moving safely.

________________________________

“You can’t leave things wide open that really do need to be protected because you’re more concerned with moving quickly than moving safely.”

________________________________

At the end of the day when you’re mitigating risk, it doesn’t matter if it’s customer data or a broken build or a bad user experience, these are all risks we need to mitigate. That’s really what Rugged DevOps. It is about is tying that value back to the user. It’s knowing what problems we trying to solve and understanding what we doing effectively (or not). It can be locked down as tight as you please, but if you cannot use the tool to solve the problem you picked it up for, it might as well not be running.

DW: Security teams have been one of the last parts of the organization to come into DevOps practices. What’s been your experience with engaging security teams at Auto Trader?

CC: That’s a good question. I’ve had experience with more than just security teams at Auto Trader. I’ve been involved in security a few places and they’ve approached me first when they found out I was into the DevOps stuff. They often come looking for assistance. They want to know: How do we get this baked in? How do we make this a priority?

It really ties back into multi-factor authentication. It’s a good way to frame it up where you want developers and operations (those people who aren’t permanently rooted in security) to be aware of these things, to have automated scans, to be aware of cross-site scripting, and to understand how to remediate those kinds of issues.

We’ve been implementing open source tools earlier in the development lifecycle to give us quicker feedback loops in order to be able to triage these things. Security applauds that but they can’t trust us too much with it. They’ve got to verify that we’re doing a good job with this, right? It’s a give and take where I’ve seen security come and offer automated solutions to DevOps teams and those solutions get adopted. The DevOps teams then take that and really start to run with it. Then security wanted to slow down a little bit because they were concerned about the quality of the vulnerability remediation. There is some back and forth there that has to be anticipated.

________________________________

“We’ve been implementing open source tools earlier in the development lifecycle to give us quicker feedback loops.”

________________________________

For security to trust you you’ve got to trust security to a large extent. If you’re putting more automation into the left side of your process and doing things before they hit production should be making your job easier. One of the side effects of that security may start going through some systems with a finer tooth comb because they’re not doing as much fire fighting. That may result in another ask from security. Understanding that this isn’t a declaration of war — we are simply continuing with an incremental progression toward further improvements. Interactions are cyclic and will happen again.

We need to help security understand our pipeline, determine what’s going to be a good fit, and keep that communication level high so the trust is there. In these symbiotic relationships, cooperation is dependent on that trust. If that trust goes away you end up in something more competitive.

There’s always room to drop back to where it’s more commensal and knowing when the kitchen’s gotten too hot. At those times, everybody needs to take a cool down lap and come back to it at the beginning of the next month. It goes back to that situation awareness, knowing where you’re at right now before you try to get to where you’re going.

DW: We talked earlier today about red teams, blue teams, purple teams. What are they and can you describe some of the practices you’ve used to end up with more purple teams?

CC: Yeah. Red teams take an attacking, reactive posture. Red security teams are worried about internal compromise, and when they find gray areas — for example, things that haven’t been fully signed off and authorized — it can result in unplanned work cropping up for the DevOps team. Generally that kind of unplanned work is introduced because the “Red” team felt they were not able to accomplish their job another way.

Then the blue team side of the equation is really trying to stomp all that out. Blue teams keep things standard and locked down. They make sure everything’s approved in triplicate before it gets implemented, which can be rigid and very frustrating for developers.

Purple teams allow that experimentation in earlier in the development lifecycle where they’ve got a little bit more leash. They’re allowed to experiment and stand up new things and make those things work before they getting it approved. Again, it’s about being transparent. Teams first want to understand what change was introduced and what the advantages of it are. They may also acknowledge that you need more than one solution to some problems, and that one tool might not fit everybody. If they’re using tools to automate that’s good. We can’t box them all into one necessarily. That’s where you see this go from blue and red to more of a purple. There’s more communication. There’s more conversation. There’s less capture the flag, more teamwork.

________________________________

“There’s more communication. There’s more conversation. There’s less capture the flag, more teamwork.”

________________________________

DW: You’ve been involved in the conversations around Rugged DevOps for awhile. Who are some of the other thought leaders you learn from?

CC: Jamesha Fisher out of San Francisco is one. She’s done some neat security stuff and is starting to look more into the vulnerabilities around some of the automation tools. Where are those vulnerable and where can they be leveraged against us? Automation is great until it automatically starts doing things we didn’t want. Another aspect of that is really trying to think more like an attacker; we need to think about: how would I break into this thing instead of just thinking how do I protect it.

Georgia Weidman is popular in a lot of security circles and I’ve missed her workshop once. I’d like to catch it. She’s got a book out on pen testing which covers a lot of operations and concepts from an attack perspective. I would say that can be a fault of some Rugged DevOps practices; you’re so quick to get the thing working and provide business value that you don’t realize where you’ve left something vulnerable. Sometimes you’ve got questions about why that thing you work with frequently is always locked down. Perhaps you find you can’t use it the way you want to. At those moments, it’s time to read through Georgia’s book. You’ll get some information as to why that thing is not open the way you want and where someone might use it for reasons might not expect..

________________________________

“You’re so quick to get the thing working to provide business value that you don’t realize where you’ve left something vulnerable.”

________________________________

DW: As we wrap up I know you’re on the organizing committee for DevOpsDays Atlanta. Do you want to give us an insight on the dates of when that’s coming up and call for papers, et cetera or we can help you share that?

CC: Our CFP is open through the first of March. Our event is going to be in April. It’s the 26th and 27th which is a Tuesday and Wednesday. Monday, the 25th of April, we have Jeff Sussna coming in to do a full day workshop on continuous design. He’s going to be giving the keynote on our first day. We’re excited to have Jeff in town and excited to have him as a keynote. John Willis is going to be giving our keynote on the second day.

We’re looking for proposals from anything in progress, people learning in the field, or trying to drop that barrier of entry. We’ve got a lot of bright people in the space. We’ve got a lot of people learning as they go. We need to be more vocal about that learning process and how we find our way through it. There’s not always this instantaneous vision that strikes us. We understand how to convert a shop into continuous delivery every night. There are definitely steps in between that. We need to have more conversations about those step in our industry. Yeah, anywhere in the stack, any interesting projects going on, we’d love to hear from you if you’ve got something.

DW: Awesome. Thanks, Chris.

CW: Thank you.

If you are interested in learning more about this subject, I invite you to download Amy DeMartine’s Forrester research paper, “The 7 Habits of Rugged DevOps”.

As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called rugged DevOps.”