Fast and Fortified: How to Build for Security and Speed
Is it within reach to bolster security while increasing developer productivity? Carta’s product security team believes the answer is “yes”. Today we’ll share our thoughts on the relationship between security and developers, what we’re accomplishing at Carta, and a framework for building trust.
Product security teams have the challenge of living at the intersection of security and software development. Our mission is to increase application-level security and be good partners to development teams. As product security engineers, our primary duty is to “Find, Fix, Fortify”: find security vulnerabilities, ensure timely fixes, and fortify systems to prevent future issues. At the same time, Carta Engineering is an environment that values bias to action and moving quickly. So our goal boils down to helping developers efficiently build secure products that our customers love.
Once we implemented a new security scanning tool suite within our code repositories, our security engineers and developers were tasked with a lot of work that was slow, manual effort. While CI/CD integrations prevented new vulnerabilities from being introduced, there was real friction in resolving pre-existing issues and new CVE’s. In order to get vulnerabilities fixed, we had to review findings, triage them, create Jira tasks, identify owners, send Slack messages, and monitor issue activity. We concluded that there must be a better way.
Our solution is to pursue automation not just for security, but also for development teams. Developers are our “customers,” and just as they work diligently to create products that delight Carta’s customers, so do we aim to build tools that make their development experience better. In doing so, we build strong, collaborative relationships between security engineers and developers. We invest in cultivating trust with developers, which leads to better security and earning more trust from our customers.
Automation built for security and developers
Carta’s product security team kicked off our “Automate to Accelerate” (A2A) initiative to deliver a streamlined experience for vulnerability management to developers at Carta.
Our three-pronged approach:
- Fine-tuned prioritization to raise only important issues and set appropriate fix timelines
- Helpful notifications to prevent issues from slipping through the cracks
- Leading with empathy to cultivate productive relationships
Fine-tuned prioritization
Appropriate prioritization of vulnerabilities is key; it builds credibility with developers and ensures the bugs that matter now are being addressed immediately. When new alerts are raised by our security tools, a Slack message goes to the security team, where they can verify its legitimacy. If we accept it, this feature automatically creates a Jira issue with the appropriate owner, severity categorization, and due date. Having an easy way for security engineers to review issues first has drastically minimized the number of false positives being assigned to development teams.
But even with this improvement, the prioritization issue is not completely resolved. For example, our generic software composition analysis (SCA) tool raises alerts for every installed package that has a CVE. Security engineers would get bogged down by this if there weren’t another layer of filtering. This is where the Exploit Prediction Scoring System (EPSS) comes into play. EPSS leverages machine learning to assign a numerical value to each CVE. This “score” represents the likelihood of a vulnerability’s exploitation. A2A automation prioritizes OSS vulnerabilities that are more likely to be exploited.
Helpful notifications
Carta’s developers live in Slack and we build where they live. But many of us have experienced annoying Slack bots that barrage us with messages that add no value and merely distract us from our work. This is our anti-goal. For our Slack notifications to provide real value to Carta, they must achieve two outcomes:
- Developers are aware of the existence, priority, and expectations related to vulnerabilities they own
- Security bug fixes gain traction and are resolved faster
And there should be the least amount of noise possible along the way.
With this in mind, A2A provides a simple system of three possible automated notifications in the lifecycle of a vulnerability:
- Notify — When a security issue is created or newly assigned, there is an initial notification to the vulnerability owner. This alerts the development team to the issue and gets the ball rolling on planning for a fix.
- Remind — Next, we send reminders based on the bug’s severity and deadline. For example, an issue with a seven-day deadline will trigger a reminder three days before it is due to be fixed. A vulnerability with a thirty-day timeframe will remind the owner one week before it needs to be resolved.
- Escalate — Finally, in the event that an issue becomes past-due, an escalation Slack message is sent which tags security engineers and leadership. Every one of these messages contain context like the summary, priority, status, due date, and link to the relevant Jira issue.
These notifications are helpful because they are minimal, actionable, and timely.
Correctly identifying vulnerability owners is critical in ensuring notifications are helpful. That’s why we built an internal web application to map out a taxonomy for Carta Engineering. It houses information about teams, code repositories, vulnerability owners, and the relations between them. We use this to ensure we assign issues to the right people.
Leading with empathy
Empathy leads to trust. Trust builds productive relationships. Productive relationships result in impactful action. Leading with empathy was one of our key principles when designing our A2A initiative.
First, we incorporated developer feedback during the earliest stages of building A2A. Next we included key developers to nail the messaging, ensuring our All-Hands presentations and executive meetings told the story of vulnerability management from the development team’s perspective. We focused on how A2A would improve the development experience at Carta, and not merely on how it would impact our security posture.
Not every bug requires a developer to “drop everything and fix it now”. For this reason, we make it easy for development teams to request deadline extensions in Slack. The security team promptly reviews each of these to see if the risk is tolerable. This means that we only escalate when truly necessary.
Is it working?
We grade our success with A2A by considering both vulnerability engagement metrics and developer satisfaction.
One such metric we track is how long an issue sits on a Jira board before anyone starts working on it. We call this “staleness.” Ideally, a plan is put in place quickly, even if the resolution is scheduled for weeks later based on priority and impact. This ensures that we are thoughtfully balancing product security with product development.
How have we done so far? Since rolling out A2A, Security issues have sat in the “To Do” bucket for one-third as much time. Put another way, “staleness” is 3x less now!
Additionally, we have seen a significant increase in how many vulnerabilities get resolved on-time.
Finally, we asked developers about their experience with vulnerability management at Carta:
- Did A2A make addressing security issues more efficient or time-consuming?
- What feedback and/or feature requests would you like to see integrated into the process of fixing vulnerabilities?
The results are promising. So far there have been positive reviews of A2A, and there is no indication that the vulnerability management experience has become worse for developers. This is a huge “win” since security improvements usually come at the expense of developer productivity.
We’re not done yet
If our A2A vision is to “build a car”, then what we have today is a “bicycle”. Our roadmap to ship a “motorcycle” looks like this:
- Reachability analysis for OSS vulnerabilities to prioritize the most exploitable vulnerabilities
- AI-generated remediation recommendations and explanations of security issues to assist developers with designing fixes
- Executive reporting and dashboards providing metrics to ensure accountability
- Support for additional security scanner integrations to expand A2A’s reach and impact
Investing in trust
We here on Carta’s product security team believe it is possible and beneficial to build for both security and velocity. We can lead with empathy while not compromising on foundational security principles. Trust will be achieved when we work to understand developers’ challenges, demonstrate motivation to help them build secure products efficiently, and leverage technology and creativity to deliver a solution with measurable results. Building trust with development teams leads to more secure products and increased trust from our customers. Trust is the fuel that Carta runs on; our security team’s core goal is congruent with that truth.
