Quintupling Optimizely’s Security Bug Bounty

In February 2015, we announced our security bug bounty program to reward anyone who finds a security vulnerability in our products. The program’s first 14 months have been a great success, finding over 85 legitimate security bugs. We’ve paid out over $19,000 to independent researchers who reported these bugs.

What’s Hot

The ROI on our bug bounty program is outstanding, much higher than the cost per bug found by security consultants or automated security scanners. 5% of bugs we’ve rewarded have been High severity and 20% are Medium severity; these are exploitable bugs with real-world impact on our customers. After the duplicate and poor quality reports are screened out, the reports have enough information for developers to reproduce and understand the problem. Reports are imported to Engineering’s issue tracking system at the click of a button.

What’s Not

The biggest cost of the program is not the rewards themselves, but the time spent triaging bugs. Reports vary in quality and a significant volume of poor quality reports come through. We use Cobalt’s triage service to help filter out duplicate & poor quality reports, but we still invest significant time reviewing and reproducing reports. Of the 457 reports we’ve received, only 18.6% have been validated and rewarded. Many of the other 372 reports still required an engineer to spend time reviewing the report. Furthermore, the reports tend to be Low severity, resulting in more time spent than we’d like focusing on Low severity bugs.

Optimizing Incentives

We believe this means all low-hanging High-severity fruit have been found by researchers who have taken a look at our site. Our last High severity bug was reported five months ago. To encourage the advanced effort needed by l337 researchers to find High severity bugs deeper in our code, a stronger incentive is needed. Today we’re increasing the High severity bounty 5x and going forward we’ll pay a minimum of $5000 for a High severity bug.

Most reports that we reward are Low severity. These bugs tend to point out logic errors that are bad hygiene but are not exploitable on their own. We still appreciate these reports, but the goal of the program is to reward security bug findings that are exploitable and have real impact on our customers. We’re reducing the Low severity payout by 50% to $50 to further encourage a focus on Medium and High severity bugs.

Like these challenges? We’re hiring!