Security at Speed: Modern Security and DevOps
John “Four” Flynn, Chief Information Security Officer
Industry folklore is full of stories where security slowed things down — extensive compliance checklists, code audits, and detection systems have all been blamed for delaying the speed of business and product innovation. However, the recent WannaCry worm reminded organizations what’s possible when they don’t implement security fast enough.
Taking a DevOps mindset to security can turn speed into a security enabler, not only through frequent technical improvements, but more importantly, through organizational process changes than make it possible for teams to act quickly.
Patching is hard, but not because it’s technically complex. On the contrary, patching is simple. It’s the organizational challenges that continue to trip up security teams.
Culture Designed for Change
The overall objective of DevOps is to support rapid, frequent, and reliable software releases. Yet, several common characteristics of companies and technical environments can make it difficult to quickly introduce new implementations:
- A company itself could have an aversion to change. As companies grow older and larger they tend to build up a culture resistant to change.
- A technical environment is mission critical so the fear of breaking it could outweigh the risk of improving it.
- A legacy system could be abandoned and leaving it alone is assumed to be safer than updating it since no one could get things running again if any changes led to serious damage.
At Uber, our engineering organization was designed to manage the risk of frequent releases, even at large scale with constantly evolving business and customer needs. In practice this means supporting both growth and frequent changes while minimizing service disruption. Several components of the DevOps model can help engineering teams reduce risk without slowing down the rate of innovation:
- Automated testing analyzes potential problems before changes pushed into production
- Peer reviews mean even more experts look at new code written by other engineers
- Phased roll outs test the stability of changes in a small area before being rolled out broadly
- Known safe states can be quickly returned if something goes wrong
These organizational patterns make it safer for engineers to deploy technical changes through a set of validations and fallback plans as a natural part of how the system works.
Modern Security is DevOps
Much of what we do in security requires introducing change into our environment, including patching and hardening systems, fixing security bugs in our codebase, or pushing updates to a firewall. The approach DevOps engineers take to reduce the risks of rapid change can help inform security professionals on how to safely introduce ongoing improvements.
To successfully adopt DevOps principles, security teams can start by identifying places in their organization where introducing change is difficult. These are the areas where a DevOps approach can be most useful. The objective here is to not only be competent at introducing change during an emergency, but quickly accept ongoing change as a natural and inherent condition of your environment.
Applying these principles to something like patching can help security teams better manage the risk associated with technical change. For example, configuration management tools give IT and security teams more control over the deployment of software updates. They can schedule peer reviews of new code, controlled rollouts to test success before expanding deployment, and facilitate a roll back changes if problems occur. Together, these organizational processes help create an environment and culture where technical change is accepted and valuable security updates can be deployed quickly.
Support from the Business
For many companies, changing culture to orient around higher velocity can be a challenge. This is why it’s important that even the highest levels of leadership understand the need for speed in accomplishing security goals. Long before the WannaCry attacks hit the press, we discussed the rising risk of ransomware with Uber’s board of directors.
Ransomware, along with DDoS attacks, is a modern business risk for many companies because it can lead to service outages. Yet, while the increase in ransomware threats is a worrisome trend, the solutions are not new. Doing the basics well, and as quickly as possible, is still the most effective strategy.
Approaching the Board with security concerns as a component to their overall risk framework for Uber, led to a productive discussion about prioritizing organizational challenges, not just technical ones. Being nimble, resilient, and capable of leaning into high velocity change are challenges — and important characteristics — of modern security teams operating at speed.
Securing the Future
There has been a lot of discussion about the need for businesses to move quickly to avoid disruption. Today, we think there is reason to orient your security strategy around speed. The threat landscape is changing more rapidly than ever. Every company should be asking whether their organizational environments allow security teams to keep pace.
