Critical controls for a DevSecOps practice — Part 1

This is the first post of a two-part series on DevSecOps practice.

Almost everyone is familiar with Fast and Furious — You’d better build that app fast, or your boss will be furious. We imagine ourselves building a street racing car, with one mission — be the first to cross the finish line, safely.

Building an app is akin to building a race car. Some may say app development and cybersecurity are two separate missions, or one main mission with a side quest. *Ding ding* mindset change time — it’s one and the same. It’s not enough to build an app fast; the product must be robust and safe too! And this is the mindset of DevSecOps.

What is DevSecOps?

DevSecOps is cybersecurity done DevOps-style. It leverages the DevOps culture of ownership, automation principles, continuous delivery, and a tight feedback loop. The culture helps to build an approach that emphasises security, without compromising on the speed of development and operations in application delivery. DevSecOps ensures that cybersecurity adds value to the application delivery process, instead of being an impediment.

As part of the team delivering Singapore’s national digital identity, I’ve had the privilege to embark on an adventure to bring DevSecOps practices to life. We sailed the choppy waters of implementation, relished the sweet taste of victory from each feature launch, and now eagerly await to set foot on “Commissioning Island”.

Successful DevSecOps requires culture reformation. A major differentiator from existing paradigms is the “shift-left” philosophy. “Shift-left”, when applied in the context of DevSecOps, moves away from after-the-fact compliance audits to an upfront, intentional effort to consider cybersecurity at every stage of the development lifecycle. For “shift-left” to work, cybersecurity has to be the norm, and not an afterthought, for every team member.

DevSecOps encompasses the entire development stack and the whole lifecycle of application delivery. It comprises every layer of the infrastructure and application stack; every configuration and line of code. Each layer undergoes a unique development and operational lifecycle, which needs to be secured at every stage.

Why DevSecOps?

DevOps is the nitrous oxide to an already solid engine. While it does give a speed boost, it should be used strategically because it has drawbacks.

DevSecOps recognises the immense speed potential of the car (fast app development), including the nitrous oxide (DevOps), and slaps on better brakes (cybersecurity) for better control of the ride. Sure, brakes technically slow the car down (as many perceive cybersecurity to do so), but it enables the driver to navigate dangerous bends (cybersecurity risks) or go full speed ahead on the straights (accelerated IT processes) to be the first to finish safely.

How to #DevSecOps?

Here are 3 must-haves in a DevSecOps practice to help you accomplish your Fast & Furious mission (The rest will be in part 2!). These controls should be considered based on the totality of the full-stack and whole lifecycle of your development and operations activities.

1. Build a security-conscious culture

Risks: Human error, ignorance, lack of communication, miscommunication, insider threat. An analogy would be authorities failing to tell you to look out for falling boulders on the road you’re driving through, or that you’ve missed the “falling rocks” sign.

Controls:

• Conduct relevant & relatable cybersecurity training based on the team member’s role. For example, developers should be trained to fiercely validate inputs, and operators to strategically deploy monitoring and response controls. The last thing we want is for the team to doze off at the wheel because training was irrelevant and boring.
• Establish guard rails. Guard rails reduce friction and encourage people to do things the right way. For instance, all changes made to the production environment must go through the pipeline. Manual access to the production environment is disabled unless absolutely necessary. A manual quality gate, controlled by a change control committee, reviews and approves changes to be made to production, while changes in other environments can go through automated quality gates. A risk-based approach should support decision-making (e.g. changes to security components require greater scrutiny than aesthetic changes). This is especially useful in mitigating insider collusion and reducing unnecessary overheads.
• Appoint a Security Champion. This role bridges the divide between the DevOps folks and the existing cybersecurity team. The security champion remains plugged into the development heartbeat as well as the security requirements from the organisation, to properly communicate security requirements and influence development, especially in its earliest stages.

2. Visibility and sense-making

Risks: Lack of visibility of the happenings at the moment that could lead to a deadly, sprawling mess. This is akin to driving with a fogged-up windscreen.

Controls:

• Inventorise everything. How will you know if a secret key was compromised if you never knew or forgot about its existence? How can you identify the vulnerable part of your application based on a recently reported vulnerability in a popular open-source library? If you have always hated lists, now’s the time to embrace them.
• Establish security metrics throughout the development lifecycle. Every stage will require different security metrics. For example, during the build phase, you want to monitor results from security unit tests, code scans, dependency checks, and configuration tests, before deciding to break a build or let it pass through. This is also known as a quality gate attuned for security requirements. To reach a common understanding of what’s considered “acceptable”, we have to communicate the risks and impact on whether a build should be let through.

3. Key management

Risks: Leaked keys or secrets — collectively termed as keys (e.g. API keys, database credentials, SSH credentials, identity and access tokens like OAuth and OIDC tokens, data encryption keys), that could lead to unauthorised access, abused resources, excessive bills, leaked data, and even a complete shutdown of your business.

Controls:

• Call your keys, don’t embed them. Avoid embedding keys into scripts, code, or configuration, and definitely not in some custom obfuscated form. Have you ever seen an Amazon Web Service (AWS) access key embedded in some code on a public GitHub page? Don’t repeat that mistake. Security through obscurity is a no-go. Instead, store the keys in a vault, then draw out the keys into temporary memory to perform the necessary actions when needed. This applies to all sorts of keys, from developer keys to API keys.
• Vault your keys. Manage access controls to your keys, and protect them with encryption at rest. This prevents a sprawling mess of keys that could lead to the abuse of leaked keys. Also, you’d be better able to manage key lifecycles when they are centralised.
• Rotate your keys frequently. · Key rotation is essential to the overall security of your keys, as it reduces the exposure of the data it protects if the key was stolen. However, traditional IT made key rotation really painful and expensive. Such activities had to be done during maintenance windows and you’d have to suspend most of your operations just to change keys. As a result, those keys were hardly ever changed. With modern automation methods, keys can be rotated seamlessly and affordably.

Cloud Service Providers offer a suite of services to manage keys and secrets. Using AWS services as an example, AWS CloudHSM can act as the root of trust in the Cloud to secure cryptographic operations. AWS Key Management Service can manage keys for data encryption. AWS Secrets Manager manages application access secrets. These can be integrated easily with existing cloud workloads and are typically being used to manage keys or secrets for native cloud services.

Conclusion

DevSecOps isn’t a buzz word. It is a practice, involving all of your people, processes, and technology, that helps you deliver products and services that meet your business objectives.

Discipline and rigour are required for a successful DevSecOps. Stick it out and you’ll see tangible benefits, e.g. more secure products, and quicker, accurate responses to incidents.

Keen to incorporate DevSecOps in your product development? Stay tuned for more pointers on the DevSecOps practice in the second post of this two-part series coming soon!