Introducing Vulnreport
Trust is our number one priority at Salesforce. We do a lot of testing to support the security of our services, so that we always keep our guard up. A big part of that is something called penetration testing. In this post, we’ll tell you a little bit about penetration testing, and introduce you to a new tool we’re open sourcing, called Vulnreport, to make the process of penetration testing easier for companies everywhere.
What is Vulnreport?
Vulnreport is a “penetration test management and automation platform”. That’s a bit of a mouthful; what it means is that this software takes the tedious work out of running and reporting on security penetration testing. It doesn’t replace security testers; it just augments their abilities, and makes them more efficient.
But, let’s back up for a minute. In case you’re not familiar, a penetration test (or “pentest” for short) is an exercise where you attempt to gain forceful control over your own software system from the outside, with minimal insider knowledge — the same way that a malicious hacker would. Why? In order to find (and fix) potential holes in your own defenses.
Pentesting is part of our multi-layered approach to protecting our customers’ data at Salesforce; not only do we continually test our own platform and services, but we also run pentests against partner code that runs on and connects to our platform — the packages available on the Salesforce AppExchange. This review process assesses the security posture of partner offerings to ensure that applications published on the Salesforce AppExchange follow industry best practices for security, and promotes trust. This is critical to maintain the highest level of security, in every interaction that our customers have with Salesforce.
Pentesting is hard work, and it requires a very specialized skill set. Because of this, it’s increasingly hard to find talented individuals who can execute non-trivial pentesting, and do it well. So, when you have people performing pentesting, it’s very important to optimize every minute of their time; in particular, you don’t want them having to do a lot of busywork. There’s no reason that a security engineer, who you need to do really high-value research work, should spend a lot of time formatting documentation.
So, at its most basic, Vulnreport acts as platform to run the operations of your pentest. It gets the busywork done fast, so your engineers can focus on the hard stuff.
Augmenting Security Engineers
So what kind of “tedious work” does a pentest usually entail? Regardless of the outcome, there’s always a lot of follow-up work needed for pentests. It usually involves: filing bug reports, opening cases, writing and formatting documents with some boilerplate text, emailing collaborators, etc. None of this is rocket science, but it tends to be very time-consuming (and boring) work.
For a real-world example: we’ve done thousands of pentests on apps released via the Salesforce AppExchange. When we give those reports, we try to make it very actionable; for any vulnerability we might uncover, we like to educate people, not just correct the vulnerability. We usually say: “Here’s why this is bad. Here’s how to fix this instance, but more importantly, here’s where to learn about best practices, so you can prevent similar issues in the future”.
That means that each one of these vulnerabilities might have 3–5 paragraphs of text. It might involve filing reports and sending emails, or logging bugs against the developers who created the software. It’s extremely time consuming — not individually, perhaps, but certainly in aggregate.
So the flow of Vulnreport is pretty simple, and built around exactly how pentesters work: as you’re performing a test, whenever you encounter a vulnerability (say, Stored XSS), you just click to select it from a list, and enter a little data about it:
At the end of a test, you select “pass” or “fail”, and it will generate a nice report. Boom. (OK, there’s a bit more to it than that, but not that much; you can find all the gory details on our documentation page.)
If you’ve been a part of Salesforce’s partner ecosystem and gone through the AppExchange Security Review process in the past couple of years, you’ve likely seen what the generated report looks like. The most important thing to us is that although our engineers are only inputting the technical vulnerability data, we’ve already spent time once, up-front, to write explanations for each class of vulnerability (“Vulntype” in Vulnreport). That means that when a report is generated, it will pull that data together to explain what XSS means:
Integration into Workflow
For reporting, the system supports what we call “linked objects”. This allows you to perform any type of external action (reporting, sending results emails, filing bugs, toggling record states, etc. — the possibilities are practically limitless), by implementing a simple interface. For us, that means updating records in our internal ticket-tracking system, and sending emails. For you, that might mean logging a bug in JIRA, posting to a channel on Slack, or opening a case in your service desk app, or … well, really, anything.
As an example, I implemented a sample connector to post information in a case in Salesforce via the API (you can see it in our Github repo, here). Doing so took me under an hour! (If you have ideas for other connectors you’d like to create and contribute, we’d love to include them.)
Sharing Vulnreport
At Salesforce, Vulnreport is now deployed across the Trust team. While v.1 ran the AppExchange security review process, in its current iteration Vulnreport is also used to manage our vendor assessment program, infrastructure security program, M&A security audits, and Open Source Software security approvals. This system has saved us a TON of time — we estimate it to be about 1 full year of a security engineer’s full-time work. (Not to mention probably raising their quality of life, too! Nobody likes doing boring, repetitive tasks.)
As we began talking about it to our partners and other security consulting groups we work with, we heard the same thing over and over again: “Hey, this would be useful for us too!” They could have built something similar, of course, but we happened to do so first.
And, since we’d already spent the time to create it, it only made sense to contribute it as an open source project, so everyone can get the benefit. In fact, among the security consulting groups we’ve shared it with, a couple are already planning implementations within the next few weeks. And, we’ll be showing some demos about Vulnreport at Black Hat USA’s Arsenal this week, to unveil it to the masses.
Testing is hard, and pentesting doubly so. While it’s noble to want to “automate everything”, it turns out that the hard work here still requires a lot of human intelligence! That’s why our strategy is tester augmentation, not just automation (as my coworker Josh Meier wrote so eloquently in his recent post).
That’s the goal of Vulnreport: to be a force multiplier, and make sure that the humans are doing the really hard stuff, while we automate away the repetitive work. This is the same model we’ve used with other automation the Product Security team has open sourced, like Providence. It’s worked well for us, and now we’re giving it back to the community, because our ultimate goal is better security for all.
