Taking a Bug Bounty Program Public
If you’re following along from our first post on “Bootstrapping a Bug Bounty”, you’re already caught up on where we came from. If you’re interested, give it a look but here I will be focused on the process of transitioning from a private self-managed program to a public and managed program with HackerOne in this post. This is all based on my experience leading the program with the awesome team over at Casper in 2018.
Going Public
After six months into our private bug bounty program at Casper, once the teams were comfortable resolving reports, they had developed practices and processes that were iterated upon; the reports had started to become a predictable trickle of similar issues. We had made progress but we had no illusions that everything was now totally secure. We felt that we needed more visibility and to attract more attention to the program. This spurred discussions about transitioning from a private program(where researchers are invited by HackerOne on an individual basis) to a public program, where anyone with a HackerOne account can join the program and file reports.
Casper was still a small team with just one newly dedicated security engineer and a group of passionate developers. We had come far in our journey but HackerOne warned us that public programs were a different beast, with more reports, more noise and more time commitment. As with starting the program, we had to set some success criteria to reach for. In this case we aimed for an increase in verified reports to 20 a month. At this point we were lucky to receive around 5 or so verified issues(reports that are applicable and not duplicates). This was a hard metric to project from our vantage point but worth setting a goal to see what we could reach.
Getting Managed
We approached taking the program public with some caution and explored our options. Going public meant more reports from folks that might not otherwise ever see our program. In doing so, we discussed also moving to a ‘managed’ bug bounty. This involved gaining access to HackerOne’s team of own triage engineers to handle the initial report validation and communication. Our team would step in once a report was validated and put into our separate queue for review. This would relieve us the burden of initial response but with more reports than before it would place a greater burden on the development teams.
We worked with HackerOne and our internal AppSec guild to setup a plan to go public and do it without drowning our team in reports by going the managed program route. It is important to note that this does come with additional costs that, as always, must be weighed against engineering time spent and organizational gain. As any endeavor of this magnitude requires, it entailed some planning and discussions up and down the organization.
Get Ready, Stay Ready
To prepare, HackerOne would steadily invite more and more researchers to our program over the course of two months to make sure we would have time to adjust to working with their triage team. This would give us additional time to modify our internal processes in order to handle the increase in reports. Ensuring that the outside triage team understood our SLAs, process and response style was important to setting expectations upfront. This went pretty well and with few growing pains. To handle the expected increase in reports we increased our triage meeting frequency and to handle additional communications we added more swim lanes in our tracker such as ‘HackerOne Verified’ and ‘Needs Internal Response’ to accommodate the transition from interacting directly with researchers to interacting with HackerOne’s triage team.
In order to take the program public we did some research on other public bounty programs to make sure that our severity ratings followed industry standards(CVSS) and that our bounty payouts were comparable to others in our space. This was important to bring our program up to the standards set by the companies in our community and attract the right researchers.
Flip the Switch!
At this point we felt confident in taking the program public and letting the whole Internet have a shot at us. What happened next did surprise us. Over the next week we saw a 325% increase in reports that sustained a high volume for an entire month! We saw 115+ reports in that first two week. This was beyond our expectations and threw the team into overdrive in order to be able to handle the volume of reports. Even with the HackerOne triage team handling verification and initial communication, we were triaging so many tickets it overwhelmed the teams. This was not a good look. We quickly moved to create an ad-hoc war room with the leads to assess issues based on their CVSS score and prioritize them against our existing commitments.
There was negative feedback at the next all hands and angry team leads at the triage meetings. Sprint backlogs had swelled with tickets and we had to ruthlessly prioritize our feature work versus our security work. In hindsight, I should have prepared the teams by asking them to budget at least half of their sprint to security and communicated more thoroughly than I did what to expect for teams.
Post Public
Things did eventually settle down after a few weeks and the frequency of reports came back down to a reasonable volume of just a handful of valid reports a week. After the dust had settled and we took a look back, switching to the managed program with the HackerOne triage team saved our team an immeasurable amount of time spend reproducing reports and communication ‘back and forth’ that allowed us to focus on other work.
In the following week we ran a post-mortem on the experience and came away with some important lessons on communication and timing. We should not have launched the program so close to a busy planning period, in which many were already very busy wrapping up the current quarter and planning for the next. Communication on the rollout of the program needed to be broader, repeated and integrated into the coming planning process with larger estimates. Even if this was a ‘one time’ launch, this information was valuable in understanding requirements for projects with a large rollout and cross-organizational impact.
Don’t Look Back in Anger
If you are thinking of taking the plunge, as we did, make sure that your teams are on the same page and understand the impact of the work that is likely to come up. For project leads, ask your teams if they understand what the impact of this change will be for them and how are they preparing for it. Make sure teams have blocked off time in their sprints and stakeholders understand the impact of such a change to the team’s schedule. Lastly, make sure that information is widely communicated. Once you’ve gone public, make sure to celebrate the win! This is a major change that will likely have a big impact on your organization. Make sure you recognize the work of everyone involved, show off and explain the wins to the company. I wish you the best on your journey and thanks again to all of the wonderful folks at Casper and HackerOne that helped make this work possible.
