How to DevSecOps — part 1: the frame

When you find yourself over the phone, explaining something on a beach, it is a pretty good sign that you are better off typing it out at once. So here we go, my take on how to DevSecOps. I have been working on building security into the development lifecycle for quite a few years (it used to be called Agile Security back in those days) both for large companies and FinTech startups.

I’m writing this from the perspective of a security person to security people. Most of this is probably obvious to you if you are coming from operations or development, this is actually Sec catching up to Dev and Ops.

In this first post, I’ll be exploring some of the key tenets of DevOps and how that changes the way one should think about security to be able to successfully integrate it.

Disclaimer

In most of the companies I have worked with, defending against nation states is not part of the security model. So please keep that in mind — if that is the space you are working in, good luck to you my friend, most of my experience is not directly reusable in your context.

That said, even if those are the guys and gals you are defending against, it is likely you will find more success catching them based on tactics and techniques they are using than anything you put into you development lifecycle. I don’t have strong evidence for this though, so please prove me wrong!

Why DevOps

If you are also one of the old-timers like myself, the most important thing you have to understand is the change in mindset.

The key idea behind DevOps is that, in uncertain environments like innovation, nobody knows what they are doing and most of your great ideas are wrong and potentially harmful.

Hence, the goal of DevOps is to make experimentation as efficient as possible. This way, you can get data or feedback on your ideas and assumptions before they do a lot of harm.

This is when the motto: “In theory, theory and practice are the same. In practice they are not” comes back full circle.

Rule #1: Do not mandate changes/controls. If you do you are harming the organisation by distorting the data and value driven exploration that was supposed to be the key benefit DevOps. So even if you could, you shouldn’t!

Theoretical issues, inflated impacts, referring to best practices, the reputational damage trump card: I’m guilty as charged

To be honest, we security professionals have some soul searching to do here: few of us try to get data on the effectiveness of our recommendations and especially go back to see if not following those recommendations resulted in actual losses. I believe this type of decision making was previously already not great (Cormac Herley explained it much better why), but it is directly incompatible with DevOps.

From the organisation’s perspective, justifications like these create a precedent, an exception where assumptions cannot be challenged and outcomes don’t need to be verified.

You must be able to sell whatever control, hardening etc. you would like to see implemented. It is just another feature that fights for the bandwidth and it has to win on its own merits. If it gets rejected or deprioritised, you should check if your calculation of its importance was perhaps incorrect. If you still believe in your feature, you need to find better arguments and most of all, more convincing data to get it higher on the backlog.

One often overlooked cost that security measures will have is the drag they create. It is not hard to see how adding cost to each iteration does not help in DevOps, a system meant to reduce exactly that cost. Spotting drag and cost of complexity is not easy and I’d argue that security people do not have a good track record optimising for this. We could use dissenting opinion even if that will, at times, mean a tradeoff as legit security features will need more explanation as well.

Rule #2: All decision making and processes have to be contained within the team: no more substantive security approvals.

Ehm…, but doesn’t that mean a lot of unsafe nonsense being published? It does. Potentially. Luckily nothing unsafe ever passed a change advisory board!

Let’s start with the obvious: if creating quality gates and requiring approvals would have worked to keep things secure we would have seen it by now.

For the last 20 years we simplified products into models and CIA ratings, mapped organisational behaviour into processes, created controls for these abstractions and measured deviations expecting that this will be reflective of actual security outcomes. What if we tried measuring those instead and did whatever worked the best? This is the type of question DevOps was designed to answer. And answer it did.

This is not a radical change, the only thing we do is taking away the special status of security. Teams have the autonomy to make all kinds of product decisions why would security be an exception here? If I had to guess those decisions will potentially and especially likely will have bigger impact. A crappy page load or bad design can easily create a huge hit in conversion and just as a good idea similarly results in a lift.

Yes, but developers know how to deal with product problems, but how would they know all the security things?!

The thing is DevOps teams have full ownership of their products. They don’t want to build unsafe products. In fact when they are unsure they will ask you willingly.

It’s true that security is specialised knowledge, maybe developers don’t know that they should be unsure of and ask your support in. Good luck, it is an exciting problem for you to solve! You don’t get to make it their problem though.

Besides, it is good for you! If people don’t have to come to you for blessing it makes you sharper. You have to be able to prioritise things to make sure you can sleep well at night. Being under pressure will keep you honest because you won’t have time to chase nonsense risks and unnecessary improvements. You will have to focus on the important things.

Your goal should be to get rid of your jobs such that, you can rest assured that everybody is making the right decisions related to security without any involvement from your side.

Rule #3: Look at mitigating risks, and creating transparency and not adding tools. Automation is important in DevSecOps but it is doubtful you will get a lot of value anything off the shelf.

This problem is not exactly new in security, since the beginning of time people tried to solve hard problem adding some magical tool or appliance. The fact that DevSecOps is often pitched from an automation direction does make make it even worse, forcing these blinders on, so it should be up there in the top 3.

This is a type of query I quite often get: which SAST tool should I buy? which CVE scanner is the best? what are the essential DevSecOps tools? what risks can you mitigate with this new cloud compliance tool?

That is approaching the problem backwards. Yes, you will need tools, but the question is which solves the specific problem you have? Most of them will not do anything useful unless you spend quite some time tailoring it to your needs. If you have to tailor it to your needs, why not understand your needs first and then figure out which is the tool for it?

To be able to pitch controls and security enhancements, you will have to understand where your risks are. Developers will rightfully push back on technical details and you will have to be able to be their sparring partner on a technical level if you want anything to be implemented. There is no way around it; you will have deeply familiar with all these topics and if that is the case why not start with the root cause questions when you are addressing them.

Tool providers are glad to jump on the DevSecOps bandwagon, shift left and update the rest of the buzzwords.

You don’t need an integrated SAST or DAST tool, you need to be reasonably confident that no vulnerabilities are introduced that are relevant to your application.

You don’t need a CVE scanner, you need to make sure you are able to identify and mitigate within hours the few publicly disclosed vulnerabilities that will be important for you.

Opening up these questions will help you to get solid return on investment on the purchases you make, and forces you to ask hard questions if these tools are even doing a good job at addressing your problems. It is clear that false positives change the cost part the equation but in my experience it is often overlooked that false negatives directly reduce the expected return. If a scanner only finds 30% of vulnerabilities that are actually important for you, you just chopped off a large chunk of expected return.

Does it mean you SAST is bad and CVE scanners are useless? No, but they are not very obviously a good investment either. Testing them, especially their limitations, will also help you understand the actual problem you are facing alternatives you might have.