Bug Bounty Launch Lessons

@magoo and @hypatiadotca

You’re thinking of launching a security bug bounty program where you pay researchers cold hard cash to report their security bugs directly to you.

Great!

But before you write your shiny announcement blog post and collect the precious retweets, let’s do some thinking and build a launch plan so you don’t drown yourself.

You Get What You Ask For

The root cause of many bounty problems is launching too fast. Bug bounties aren’t going to be the right move for every organization when there’s a lot of work that could be spent on the fundamentals. While you should always have security in mind developing software, bounty programs require particular ways of prioritizing work and allocating staff that may not be right for your team right now. Let’s walk through each burden a bug bounty program will bring, and then you’ll be ready.

Before the Bounty

Before you even consider a program, your engineering house should be in reasonable order. A bounty program will always give you more engineering work. It is not a development framework, firewall, network appliance, or other magic security product. It is a program of “exercise and vegetables” that will succeed with care and attention.

Here are the attributes we’ve seen make for successful programs:

• Reality Check: Do you already have extensive security debt — known vulnerabilities and creaky infrastructure?
• Triage: Are bugs centralized, tagged, prioritized, and do they generally find an ultimate owner? Are security bugs reasonably prioritized versus their owner’s other work?
• Culture: Is there an appetite from your engineers to fix security bugs? Do you have leadership buy-in to prioritize externally reported issues, and executive air cover for the PR noise that will come out of a bounty program?
• Backstop: Who will be fixing bugs with no clear owner, or project manage systemic issues requiring substantial engineering effort?

If you aren’t already treating security issues like you would treat scale issues, you risk creating a new problem. Some teams have used bounties as an instigator to fix these issues, but that’s your own call — it can be a risky move. Having the above items in place is your best chance for success.

Starting Slow

Don’t make the mistake of launching a bounty program in one fell swoop. You do not want public commitments and press haunting you while running headlong into signal versus noise issues and internal engineering drag.

Instead, design your launch properly.

Minimize the scope so that you’re only receiving reports for areas you are confident are more robust, have lots of room to mitigate issues, or are under active development and able to make changes with the most agility. Grow the scope as you become confident until your program has broad coverage.

Start with lower bounty amounts to keep things lower-stakes. Increase the amounts as you become confident, until you’re competing with going rates in your industry.

Invite high quality researchers in a private program so you’re not suddenly following up with a horde of mixed quality researchers. As your spikes become lulls, start inviting more researchers until the program is public.

Having a project manager onboard to attach milestones to your program rollout can be very helpful, for example “Full scope within 3 months” or “$1k bounties by EOY”. It’s all a matter of your own pace.

Running Smoothly

The bulk of the work of running a bounty program can be broken down into the following parts: triage, engineering fixes, and public communications.

Incoming bugs will need initial triage for signal. Submissions will include product misunderstandings, disagreements on risk trade offs and best practices, and your standard OWASP Top Ten bugs. Reputation systems can be an important tool to keep noise under control, and give the folks running triage a good metric for how much to dig into a particular bug.

Having security and non-security engineers collaborate on first level triage can be a great way to promote awareness and empathy towards the security mission. As your security team grows and specializes, you’ll likely move from an all-hands-on-deck rotation to dedicated triage, with higher level triage going to engineers who are closer to mitigation.

Contracting can also make sense for initial triage. Budget 2–5 hours per week for first level triage for a startup with a small attack surface, or work out a per-bug fee with a reputable vendor.

Be wary of burnout when it comes to triage, particularly if one person ends up handling most or all of it. However you handle triage — collaborating with non-security engineers, sharing a rotation, or contracting it out, it becomes easier to involve others if your culture can appreciate a nasty bug. The folks interfacing with researchers will spend a lot less time crafting the perfect “sorry but not a bug” message if they know that their company has their back.

Engineering fixes for bounty bugs will range from changing one line in a config to substantial coordination between multiple teams, outside vendors, or upstream open source projects. Treat this like you would any systemic problem within your engineering organization and have project management capability in place to shepherd the complex bugs.

When it comes to public communication, don’t let your submission threads turn into typical internet debate. Some researchers will permanently disagree on the severity of bugs, submit low quality bugs, or threaten hostile blog posts. It’s important to expect this and involve level headed teammates who are well practiced in empathy. Try to be payout-lenient and encourage good research (even when off the mark), but have clear policies for what constitutes a bug and what you’ll pay out for when you find research that is way off the mark. If there’s a conflict over the severity of a bug and you’re certain it’s low risk… disclose the details. With the specifics out in public, misrepresentation can’t happen.

Bounties Forever!

Once you’re paying public bounties under a wide scope, you should have plans to track the interaction of your bounty program with the engineering team on a regular basis.

Here’s some basic questions to ask regularly:

• Budget: How much are we paying weekly / quarterly / annually?
• Risk: What are our top five bugs in severity this quarter?
• Regression: What issues keep recurring? What tools or practices can we deploy to prevent them in the future?
• Hiring: Which researchers should we recruit?
• Atrophy: What bugs took too long to fix?
• Happiness: Are researchers happy?

We should all aspire for a million dollar bounty. It should be so hard to score a bug in a product that we’ll someday be willing to put a million dollars behind it.

Someday, your security program could be described by expensive bugs.

Good luck.

@magoo

I’m a security guy, former Director at Facebook, Coinbase, and currently a HackerOne founder / advisor and consultant for a handful of startups. Incident Response and security team building is generally my thing, but I’m mostly all over the place.

@hypatiadotca

I’m a security engineer at Slack. Prior to Slack, I worked at Salesforce.com, Microsoft, and Symantec. I care a lot about building sustainable, healthy security cultures that help developers ship code with confidence.