DevOps and security: from trenches to command centers

Shared values, if you’re doing it right, doom and gloom if you’re not.

Previously, system administrators frequently were responsible for security: applying patches, building firewalls, enforcing security practices. Work often involved a lot of exciting manual labor, unique tooling, and esoteric techniques.

Rare, esoteric techniques

Here come the DevOps

As time passed, both security operations and maintenance operations have evolved. DevOps movement emerged as an attempt to build the bridge between people who write code, people who maintain the infrastructure to run it, and people who make the business decisions. These changes have put emphasis on the new set of techniques and values. These techniques and values can either be beneficial or problematic for the security posture.

New exciting techniques

Through the years, in one way or another, I’ve been responsible for building, running, and enhancing product infrastructures for companies I worked in, or for their clients. Somehow, many things emerging into the landscape from DevOps seem pretty familiar if looked at from the right angle. Working for a cryptographic engineering company Cossack Labs, where demands towards the development security are slightly higher than the regular “hey, we’ve got payment processing data and potentially millions in regulatory fines”, I’ve had an opportunity to rethink many of ideas behind building secure processes in a modern way. Somehow, many DevOps concepts perfectly fit into the picture.

This post is an inquiry into mental models that make DevOps beneficial for security, instead of being detrimental.

Looking at DevOps through its values, DevOps is all about business performance through technical innovation:

1. Improving value delivery to customer: delivering continuously, reliably, predictably, via automated scenarios.
2. Improving change management and value creation: using continuous integration and testing everything, programmatic infrastructure, testable user experience.
3. Improving failure recovery: preventing incidents, managing incidents, understanding root causes, repeat.

In the end, business is able to charge clients faster and more frequently: just because more value gets delivered faster, with increasing consistency. All of this is achieved by using certain processes and tools. When used blindly, it brings a lot of damage.

Driving blindly

Security is an invisible showstopper for all the aforementioned values if done wrong:

1. Real-world impact: Customer data leaks and service disruptions ruin any value you aim to deliver faster. Lawsuits and regulator fines will not just affect the developers, in many cases they will rule the affected companies out of business.
2. More causes for damaging impact: Quicker changes, done while impairing security and reliability, will introduce more leaks, more service disruptions, and indirect damage as well: your systems will be used to damage others.
3. Even fixing things quickly is a threat: Failure recovery, which puts “time to get back in production” first, without security considerations, is a frequent path to introduce security inconsistencies into a system.

At the same time, security-conscious DevOps could significantly increase the security posture:

1. Improve value reliability: Optimize for reliability, repeatability and predictability of value delivery, and the speed will organically follow. Use programmatic and consistent security controls.
2. Improve value verification: Optimize change management for deterministic state of any code where state includes code’s security status verified automatically.
3. Prepare for the bad times: Aggregate metrics for incident response and backup controls for threat mitigation.

Whether your company has a security policy and security-aware developers and operators, or you’re just looking forward to get started, DevOps tooling can bring a lot of process/infrastructure safety from day zero, then help evolve it into consistent security posture.

Transforming DevOps practices for security

Even though tooling is just a part of DevOps phenomena, it’s a rather important part of it.

From viewpoint of traditional secure development practices, many DevOps approaches are not just “compatible” — they’re perfectly fitting. Let’s take a look.

Even though a bit chaotic, DevOps methodologies can be put to a good use

Test everything!

Functional testing as Dynamic Application Security testing: First of all, you want to know that security controls within your application actually work.

Testing security functions in functional tests and running specific non-functional tests against known vulnerabilities allows ensuring that security controls do work as expected on every iteration of continuous integration.

Things to try:

• BDD-security suite as testing framework for functional security testing, infrastructure security testing, and application security testing. Integrates with Jenkins!
• Gauntlt, a number of Ruby hooks to security tools to integrate into your CI infrastructure.
• OWASP Zap and OWASP Zapper (Jenkins plugin): automate attack proxy to test some of the attacks.
• Mittn, F-Secure’s security testing tooling for CI.

Source code analysis as Static Application Security testing: it’s worth investing effort into making sure that detectable vulnerabilities get detected automatically.

Reducing vulnerability risk by scanning sources with static code analysis tools / static application security tests for possibly vulnerable code during CI iterations.

Things to try: OWASP list of some SAST tools, a list of classic tools, many modern ones emerging daily.

Blocking or parallel? Obviously, when developing for security, security tests should be blocking, period.

Collection of audit and operational data

Metrics DevOps’ use to monitor and tune development and production environments are an important context for audit logs relevant to security. Being gathered together, in a secure way, can make intrusion detection and incident response stellar.

Automatic configurations 101

Over the course of years, I’ve had several occasions to have automatic configurations in building firewalls and access control, and while some of the stuff was taxing to operations, it’s the simplest way to have safe defaults, consistent firewalls, and some confidence in your access control system.

Moreover, positive security for something like application firewall means enumerating all possible scenarios of moving around HTTP endpoint graphs with parameters and stuff… easy to make manual mistakes, easy for insiders to hide a channel to leak data out.

Generating firewall rules programatically, if done right, is a great nerve savior.

Orchestrated automatic configurations

Orchestrating multiple automatic configuration tools is where magic starts: imagine you know the role and address of every node in the system. Now imagine you can whitelist access based on role model, and basically have positive modelled iptables configuration instantly drawing out significant portion of threats.

Know thy limits

Investing into automation shouldn’t be religious. Preventing attacks and managing ongoing incidents is still a decision-making process of a human being under stress, pretty much like the OODA loop in the military methodology.

Things that can be automated, should be. The rest should be orchestrated in a way that humans, who should actually make important decisions, will not only have all the relevant metrics and audit logs, but means to act quickly, efficiently and across vast infrastructures of today as well.

Getting serious and boring

Well-orchestrated DevOps environment

After your infrastructural tools have been adjusted, you’re already far ahead of many development and production environments. However, whatever amount of labor you’ve put into it, it’s worthless if there still is a simple way to mount an attack against your infrastructure. Security is asymmetric game: you have to defend against every threat, while attackers have to find just one weakness to succeed.

And so, security policy and procedures are still necessary. Having built parts of necessary infrastructure, next steps will be simpler: understand risk model and security objectives, audit current state, define gaps in processes, techniques, then come up with development plan and then, finally, go back to engineering, but with all-encompassing construction plan.

Conclusion

If DevOps practitioner sees that DevOps is about enhancing business value and business value includes business continuity and security — suddenly, tooling falls into place and helps adjacent operational needs.

If DevOps practitioner is just excited by tools and fancy lingo — it’s a wreck in the making. And if you add security into equation, the wreck might be more than just inefficient development and maintenance, but all the typical scary tales coming true.

_______________________________________________________________

This rant is based off a talk I gave at DevOps Saturday meetup, you may want to go through the slides for some reason, too.