A practical guide to scaling Application Security so Developers can focus on what they’re great at while shipping products more securely.
By: James Chiappetta
Disclaimer: The opinions stated here are my own, not necessarily those of my employer.
The role of Developers and DevOps Engineers is to create the code and instrumentation needed to release a product or service. It is important that they have what they need in their workflow to release code securely. How do you make sure this actually happens without constant obstruction?
This post will help provide a high level overview on how to successfully embed security in the daily development workflow and the Continuous Build and Continuous Deployment (CI/CD) system. This way Developers can focus on what they’re great at while shipping products and applications more securely.
First Things First
In my first post, How to Kick Start your AppSec Program with Ease, I illustrate how a foundation for scaling a security program happens through 1) creating a culture of education, empowerment, and accountability and 2) getting product owners and engineers to think like their adversaries while they are designing. While this is foundational, it doesn’t actually address the pieces needed to achieve scale.
Trust in the Developer is important but trust in the systems that support them is equally as important.
To achieve Application Security scale:
- Set up security guardrails — Define universal product and application security guardrails.
- Make security free for the masses — Create a central, validated, repository of libraries and services that perform critical security functions.
- Automate the checks and balances — Inject Security Tools in the CI/CD system to automate the detection of drift from the guardrails.
- Calculate the value — Measure the effectiveness and cost savings in practice.
Step 1: Set Security Guardrails
Create the universal set of security requirements — that all products and applications must adhere to across products and applications within an organization. You will need to set these guardrails with the product and engineering functions within the organization. The buy in from the onset will provide the support for execution of what comes next.
The content of these guardrails is completely up to you and the needs of your organization. It is common to include things OWASP top ten, which would include Authentication and Input Validation.
Step 2: Create Centralized Libraries
Create a central set of security libraries and services — that solve for the adherence of the guardrails. Your organization can centralize the creation and management of these libraries and services under a security development team. They would be responsible for acquiring tooling from third parties, choose a distributed development approach, or any combination of structures.
It’s a crucially important point that the libraries themselves are centrally available, validated, and ideally mandated by the guardrails. This is where Developers and DevOps Engineers can “get security for free” by including those libraries in their projects. Once these are created then the checks and balances with security tooling in the CI/CD system will provide the confidence that everything is in place before anything is released.
This approach puts development ownership in the hands of subject matter experts in security and allows non-security development teams to focus on their core competencies and delivery of value without the burden of supporting security technology. Learning through failure is great in many cases, but risking security breaches is not the wisest!
- There may already be built solutions that provide the security libraries and services needed for your technology stack. For instance, Helmet provides a number of HTTP headers for securing a Express.js app.
- If you have a Developer or DevOps Engineer who has deep subject matter expertise in a technology, you may want to ask for their partnership in creating anything custom for the organization.
Step 3: CI/CD Process Injection
The following are some of the key tools that will set the groundwork for embedding security in the CD/CD processes. Use the visual representation at the end of this post to better see how it all fits together.
Build Time (CI) —
Auto identify high risk changes — 4 eye rule for high risk changes. Use automation (like a Herald rule) on certain repositories to auto add an approval for anything that changes high risk code or infrastructure.
Perform Static Analysis Security Testing (SAST) —
- Use a security focused tool to catch various security issues within the code. The site analysis-tools.dev and OWASP both offer different options you can filter on. This will dramatically help the automation of detecting common vulnerabilities. If you are interested in a more technical read on SAST at scale, check out this post.
- Infrastructure as Code (IaC)/Policy as Code Analysis will help detect when you have either cloud native or systems based policy drift. Low hanging fruit issues such as public AWS S3 bucket policies or systems or any broad access Security Groups.
- Use custom linting to check for very easy to spot issues, e.g. the use of deprecated functions, missing an import or use of a critical security function, like authentication.
Perform Third Party Dependency scans — Developers will often turn to prebuilt SDKs and Open Source Libraries so they don’t have to reinvent the wheel. Using the same methodology, security tooling can come from already built tools! These dependencies and their flaws will need to be factored into the vulnerability management processes and creating standard language/feedback. The OWASP Dependency Track tool could help here.
Sign Build Artifacts — In order to detect tampering, sign your build artifacts (files, images, containers, etc.) before they are released to the build system. With Docker, you can use Notary to achieve this.
Deploy Time (CD) —
Perform Dynamic Application Security Testing (DAST) — Use a tool that will scan the application or service in a running state to find common security flaws. Check out this OWASP page with the various commercial and free tools that can be leveraged.
Integration testing — Ensure critical controls work (e.g authentication, authorization, password reset, session invalidation, output encoding, etc.) through integration testing and test driven development practices.
- Prioritize which tools outlined here make the most for your situation. Do those first. Priority should be based on effort, time, and desire of the product and engineering team.
- If you are a part of an organization that runs 24/7/365 you will need to ensure that whatever pieces you put into the CI/CD System are supported around the clock, and a break-glass is added for critical changes. Logging and monitoring are vital here.
- Security tooling needs to be highly performant in order to not increase the build/deploy processes beyond an acceptable threshold and monitored for drift beyond acceptable SLAs.
Feedback loops & Reporting — It’s crucial to get feedback from the tooling to the Developer both expeditiously and accurately. You will also need to factor in what breaks/blocks builds or deployments and the amount of time it will take to run security scans. Feedback from the tools should go into a ticket/work tracking system so notable flaws can be accounted for and addressed. You should also consider how each flaw or issue maps back to the guardrails and incorporate a “how to” for addressing the issue.
The accuracy from the libraries and tooling is vital. Whether the feedback is sent at build or deploy time, one of the biggest areas of concern is false positives and issues reported that are not actually representative of the true severity of an issue. When it comes to tuning, focus on “High Fidelity” and “High Risk” checks first. The rest can come later in an iterative approach.
Trust — If Developers and security folks alike can’t trust the libraries and the tooling to run smoothly and report accurate issues then it will be back to the drawing board. You will need to take the process and tooling improvement suggestion and turn it into action. Improvements will be needed and planned for. This is how you build and keep trust in the system.
Pro Tip: Developers are your partners and together you need to manage the workflow. Create a pilot group for testing various tools and feedback/reporting methodologies. We need to bridge the cultural divide!
Step 4: Measure the Value
Business enablement & revenue protection — Establishing a set of universal set of security requirements and set of security libraries or services to meet these requirements will serve as one of the key value propositions. Once built an organization can embed checks in the CI/CD system and provide feedback to Developers quickly. This should translate to more features, released quickly, and with security built in.
Less work for Developers (reduce the security tax)
- Leading KPI — Quantify how much time it would take each Developer to meet each requirement independently. The calculation of time saved and vulnerabilities avoided can be done on an ongoing basis.
Higher bar for Attackers
- Lagging KPI — Create a dashboard that shows the number of attacks prevented that map to the various security requirements. This will show the Developers that they are helping to protect the business!
- Create a set of guardrails for security to create automated detection and reporting.
- Create a set of libraries and/or services to solve/meet security requirements. Have a security focused Development build and maintain these.
- Let Developers focus on what they are good at and “get security for free”
- Deploy tools in the CI/CD system that get Developers, DevOps, and Security folks the feedback to make good decisions in a timely fashion.
- Track security flaws (blocker or non-blockers) as tickets with common methods for addressing them.
- Build a dashboard to track team time saved, vulnerabilities avoided, and attacks prevented.
Words of Wisdom
Buckle up for the road ahead. The tooling outlined here is not trivial to implement and maintain at scale. It will take a well executed strategy in order to deploy it and get it to scale.
Not every organization has one build and release system. Prioritize based on usage and/or what software is funneling through each. I write more on effective AppSec Program management here, if you are interested.
Want to know how to build a killer pentesting program? Check out my post How to Prevent Security Breaches with AppSec Pentesting.
A Visual Representation
Contributions and Thanks
A special thanks to you, the reader. While this post is on the longer side, I hope you benefited from it in some way. I want everyone to be successful at this. While these posts aren’t a silver bullet, I hope they get you started.
Please do follow my page if you enjoy this and my other posts. More to come!