How to DevSecOps — Part 2: the diagnosis

It this part, I will outline the context, natural strength and constraints of DevOps when it comes to security. The aim is to help you to create a strategy base that you can build on by incorporating the specifics of your organisation. I will focus on the security of the product resulting from the development process, instead of the security of the process itself.

Common misconceptions

Before getting into the diagnosis, I’d like to highlight an approach that I often see derailing conversations about DevSecOps.

Frequently, the complexity in doing secure DevOps is framed based on previous security approaches. The classic problem statements are often: we cannot do a pentest for each release if we release multiple times a day. How can we be sure that people are doing the right thing by themselves, if there are no external security gates?

These questions completely derail the analysis and consequently the strategy. They either mask the actual complexities, or in some cases miss the issues entirely. I believe we need to have a fresh look at the topic to find out what the real crux of the issue is.

Pentesting

I’d argue that not being able to do pentests before features are published is not a problem. In applications, detecting obvious problems (unpatched vulnerabilities, obvious injections) that can be easily automated by attackers, is by definition easy to automate. The remaining problems are the exotic things specific to your product where pentesters can really add value. Again by definition these are exotic and hard(er) to find. Leaving these in production and eventually testing them will not open you up to a huge risk. How many times did a pentest find an issue that a previous one did not, but that nobody exploited in the meantime? If you decouple pentesting from releases by switching to periodic pentests and regular baseline scans, you will be back to similar risk levels. There are better solutions than this, but there is no good reason to stick to doing a pentest for each release.

Gatekeeping

For security gatekeeping, I would again break down the problem. The purpose of having security gates was to:

1. catch mistakes stemming from ignorance/lack of specialised knowledge in security;
2. hopefully detect malicious changes; and to
3. comply with security/governance frameworks that intended #1 and #2 to be allegedly useful.

#1 is a valid concern and one of the main things we have to address in our strategy. I’d argue that #2 has never really worked well and we would have a better shot in DevOps if changes are reviewed often by peers with more contextual understanding and who are therefore better positioned to catch such attempts. #3 should change (because it is a reflection of the established way of dealing with #1 and #2); however, it needs to be addressed here as it is a constraint for now.

Diagnosis

So what is at the heart of the challenge?

Security is difficult because it requires specialised, fast changing knowledge. Actual feedback is rare (for instance getting hacked compared to say getting user feedback) and outcomes are hard to measure (e.g. security compared to conversion). One could argue that even adequate requirements are hard to define. This relates to #1, which security gates were trying to address with some “quality assurance”.

Additionally, security is tightly coupled and issues in unrelated components can have a catastrophic interplay. This makes security complex, but it is in fact similar to issues in software architecture. Most security issues that you need to catch can be done based on local observation. For the exceptions, I do not think anybody who had to debug async code or complex microservices would think this is different from development and especially operations.

A constraint of DevOps is its decentralisation, so creating overviews and reports of everything will not be obvious. Call me a radical, but I’d say this is not necessarily a problem. That said, as we discussed earlier, this will have to be accepted as a constraint in most cases. This is another thing historically addressed with security gates, #3 to be specific.

Advantages

There are several specifics in DevOps, Agile or some of their practices that will make it easier to accomplish certain things important for security. These are leverages you should include in your strategy and everyday thinking when faced with security challenges. Here is a list of the ones I found most useful and some practical ways they can be utilised for security purposes.

Frequent changes mean that fast fixing of bugs, patching and incident response will naturally become easier. A fast pace usually also means high coverage of automated tests — developers won’t want to manually test so many changes. Additionally, you can leverage tests for your security tooling or add security relevant ones to make security part of general product quality.

Joint ownership of code is also a key practice that you can use to your advantage. You can be sure that changes will be looked at by multiple people and worked on by the entire team over time. This means that teams will become better at catching bugs and bad patterns.

You don’t need each developer to know how to harden XML parsers against XXE, you need one of them to have it in their passive memory, “Wait a sec, I remember I read this article about some funky exploits related to XML, lemme check it out before we merge this”.

Reproducibility is key in DevOps pipelines, each change in the product should be traceable to a change in the code. It should be clear what went into each specific version. This transparency will help a lot in incidents, vulnerability management or getting the right audit trails without unnecessary extra effort.

Infrastructure as code itself is a way to achieve easier reproducibility and shared ownership. It is clear how one can use it to their advantage in security. This allows for all infrastructure changes to be scrutinised as carefully as software changes — you get the audit trails, which creates different venues for assessing security and greater resilience (such as in recovery). You do not need years of bespoke manual changes to create a functioning environment, but a mere few minutes of running scripts.

Autonomy, responsibility pushed to the product teams makes them more careful and engaged if you picked the right people. This means that instead of expecting security be the responsibility of the security team and caring less about it themselves, they actually take charge of making sure their product is safe.

Next up

In the next part, I’ll outline an approach for using these advantages to cover the key areas of security in DevOps. The aim isn’t to get to the same level of security where we were before Agile and DevOps, but to surpass it. By now the data is in, companies that managed to nail DevSecOps have done this consistently.