Threat Vectors and Software Security: Securing DevOps
Part one of this series on software security explored the role of the organization in establishing best practices around security. The bottom line: Employees can have the greatest impact on the prevention of exposure to vulnerabilities. Now we’ll go a layer deeper and look at DevOps.
One leading issue with security examines how many security professionals you need to designate to that role. According to GitHub, “security professionals are outnumbered 500 to one by developers.” Given this scale, exploring security around DevOps comes with high ROI.
What is DevOps?
DevOps combines software development (Dev) with IT operations (Ops). It focuses attention on people and culture to deliver high quality applications at a faster pace. At its core, DevOps shortens the development lifecycle by having the people who build your application deploy it. Another term some might use is DevSecOps. This is the same notion as DevOps but with emphasis on a security-first mindset.
DevOps enables speed and proficiency. It allows you to code, build, and deploy your application at a rapid rate to get features (or fixes) out to your organization or customers faster. By integrating security into your DevOps pipeline, you ensure the faster turn-around doesn’t come at the expense of increased exposure and vulnerability. For instance, automated tests and static code analysis are more repeatable and reliable than their manual counterparts. And, having these systems in place helps to ingrain security as part of your culture. It’s not just about writing code, but about writing secure code.
DevOps Impact on Security
Now that we have a good understanding of what DevOps is and the role it plays, we can dissect its components. We’ll start first with some of the mechanics, like building, deploying and automating. This is how your code goes from the developer’s machine to production. Then, we’ll look at process elements, like handling and rotating keys. This addresses potential vulnerabilities that could expose sensitive data. Lastly, we’ll look at monitoring, auditing and backups. This enables a holistic feedback loop to ensure the machine is working as designed.
By incorporating these elements into your DevOps process, you’ll rest easy knowing your software is being deployed in a safe and secure manor.
Before modern CI tools became prevalent, developed software was packaged up and uploaded to a production environment. The update process was manual and often nuanced, with only a handful of people with sufficient knowledge to perform the task. Then, people began to write scripts to automate some of these process steps. And they manually triggered those scripts to facilitate the operation.
Times have changed. We have modern tools that will detect newly committed code and run automatic processes for that code such as compiling, building, and checking dependencies. This removes the human element from the process and improves repeatability and reliability. Consequently, the process itself is both more efficient and more secure.
Automating builds can significantly improve quality as well. By including unit tests, static code analysis and packaging in the automation, you ensure that critical steps aren’t forgotten. And by performing these tasks on every commit, the source of an issue presents itself as fast as possible. This has the added benefit of accountability since we know which individual committed that code.
Once you build the application, you need to deploy it out to your environments. The major security risk here is access to those infrastructure components. If developers have access to production machines, there’s an increased security risk, whether accidental or otherwise. This is particularly true when they have permission to make changes (“write access”).
This is where automated deployments come in. Using the same tools for automating builds, you can incorporate scripts for sending the packaged code out to the relevant environment. In this setup, the developers have access to trigger a deployment, but they themselves cannot access the environment directly. Instead, you configure the tool with the appropriate access and it deploys your application, never deviating from its one and only purpose. Tools like GitLab, TeamCity and Bamboo are able to perform these types of operations. They can be configured to perform operations that are very custom to a particular environment or application.
Infrastructure as Code
The infrastructure that modern applications run on is ever-growing in size and complexity. Managing that is a difficult enough task in and of itself. And it only complicates the matter of testing and rolling out new software. The people that have access to infrastructure components have a thorough understanding of its design and dependencies. And anyone with access who doesn’t have that knowledge is a vulnerability.
This is where IAC tools like AWS CloudFormation and Terraform come into play. These tools enable developers to “code” the environment. Then, that environment can be reliably duplicated for other purposes. For example, a developer could use AWS CloudFormation to set up an environment for development. Then, with the click of a button, they can stand-up identical environments for testing, staging and production. For new features that necessitate adjustments, developers make the change one time in CloudFormation. All environments are kept in sync. Use of IAC tools ensures continuity between environments and limits tinkering of individual ones.
With IAC in place, you can grant developers more limited roles without impeding their responsibilities. They can click a button to issue a reliable deployment without having any access to the underlying infrastructure.
Encrypting Security Keys
Another benefit of automating builds/deployments/infrastructure is the ability to encrypt and hide access keys. Access keys are the means by which software communicates with other services. There might be multiple keys to account for the many services with which modern applications communicate. Examples include internal components like application databases and external services like merchant services. If keys become exposed, then malevolent individuals can gain access to sensitive data and information.
Storing access keys in a secure location reduces exposure to this threat vector. For example, AWS provides a service called Secrets Manager. Developers don’t have to know or care what the actual access key is. Rather, they reference the secret by name in the deployment script and the automated process handles retrieving the key and using it. If people don’t know the access key, they can’t accidentally mishandle it.
Rotating Security Keys
Expanding on secure key storage, automated key rotation brings an added layer of security. Key rotation is the process of expiring a key and replacing it with a new one. This ensures that if a key does somehow get compromised, it’s useless to the acquiring party. It’s a good idea to run key rotation on a scheduled basis. But you can also configure it to be engaged manually. This comes in handy if you have reason to believe the key has been exposed.
Studies show that people behave better when being watched. Developers are more likely to walk the straightened line when they know they’re being tracked and monitored.
Audit trails are a key component of effective DevOps. In fact, auditing in general is a common theme throughout all of security. An audit trail identifies who is doing what and when. If a security incident occurs, you can quickly and easily identify the specific individual and hold them accountable for the incident.
There are a variety of tools available to facilitate auditing as part of DevOps. AWS, for instance, provides CloudTrail, which documents everything that’s happening within an account. It captures everything, from provisioning machines to changing permissions to accessing secrets. And it provides a clear history that can be looked back on.
Audit logs should be protected from deletion or manipulation. Store them in a secure place, accessible by only a few privileged individuals.
Security Metrics & Alarms
With the environment up and running, the next step is assurances. This is accomplished with metrics and alarms. Key metrics help you know that an application is running smoothly and securely. Alarms notify appropriate parties when key metrics exceed the expected or acceptable conditions. Consider a metric to monitor inbound requests for an Internet-facing application. Admins could be alerted to unexpected spikes in inbound requests, which could be a sign of a targeted attack. Having an alarm in place enables real-time awareness of potential threats.
Backing Everything Up
The last step is crucial — backing up the application and storing it in a separate location. For databases, this can be done in real-time. In the event something does happen to an application or its data, you should find comfort knowing there’s a way to recover.
The key pieces described above facilitate a secure and compliant DevOps process. While it requires effort to set this up, the automation of it brings exponential ROI over the application lifecycle. And the cost of a DevOps process is nothing when compared with the potential misuse and mishandling of sensitive information.
In the next segment, we’ll look at best practices in securing the application’s environment.
Originally published at https://www.gavant.com on February 3, 2021.