One Year With a Private Bug Bounty Program at FINN.no
Over the years, FINN.no has been doing a lot of different security assessments: from the classical one test per release to regular on-site review and testing by security professionals, and more extensive bi-yearly tests.
Still, last year we discovered that the average lifetime of vulnerabilities found in production was higher than expected. The average lifetime was several years, and the outliers had been in production for a decade! We realized that the way we had done security testing did not keep up with all the changes in FINN.
The “release test” made sense back in the day when we had few releases per year, but now we are pushing changes to production well over 1500 times a week, and the concept of a release test or bi-yearly tests makes little sense. Also, a lot of the vulnerabilities had survived previous security assessments, and that is probably not for lack of skills in the penetration testers, but proof that sufficiently large enough applications are hard to test with limited time and personnel.
With that in mind, we realized that we need more continuous testing with many eyes on the target, preferably with diverse skill-sets. And one way to do that is to launch a bug bounty program.
What is a bug bounty program?
You can think of bug bounty programs as crowd-sourced security testing, where people can report vulnerabilities and get paid for their findings based on the impact of the vulnerability. We have been running a private program on the well-known platform HackerOne for a year now, and we are happy with how effective this program has been. In terms of vulnerabilities found, we have gone from 15 per year to 15 per month!
The apparent reason for this difference in discovered vulnerabilities is that a bug bounty program is not limited by time and the number of people testing, as opposed to classical security assessments. In our program, we have many eyes on the target, and they are free to look for flaws on our site whenever they like. The result of that is a steady flow of new reports every month. We received 221 reports, and we rewarded 129 of these with $55k divided among 31 hackers. Each peak in the graph corresponds to when we invited a new batch of hackers, or when we have extended the scope of what the hackers can attack.
One key difference with the bug bounty program is that we do not have any guarantee that specific parts of the site are being tested, nor do we control when the site is tested. Still, it is possible to create incentives for hackers to focus on specific parts. Some programs run special promotions with extra bonuses for certain types of flaws to incentivize. We have yet to do this, but we want to create some way for us to communicate changes to hackers easily. Remember, with thousands of deployments a week; there is a big chance of some changes introducing vulnerabilities. One of the most critical findings in our program resulted from a one-line configuration change — and not new complex code. That flaw tells us that all changes, both big or small, are worth investigating.
Why is the program private?
A typical path to launching a public bug bounty program is to start a private program first, then graduate to a public program when you are ready. We do not have any plans of going public any time soon, as we are happy with the number of reports and the overall quality of the reports. By quality, we mean the number of valid reports. It is no fun for hackers nor us to close a report as not valid. We all want the number of valid reports to be as high as possible, since then we do not spend time on unnecessary reports and hackers get paid for their work. With public programs, anybody can submit reports, and therefore you will get more noise in your program. To back this statement up, I have looked at some data from other programs. Shopify runs a popular public bug bounty program on HackerOne, and each month they publish statistics from their program on Twitter. I have also received data from Visma’s private and public program (Shout out to Joakim! for the data). In the graph below, you can see the closed reports state statistics, and only reports in the resolved state are valid and given a reward.
Based on these numbers, we can see that the private programs are getting a much higher share of valid reports and that the public programs are getting high portions of not applicable and informative reports. Meaning reports that are not accepted or just closed as informational for various reasons. The high share of valid reports is one reason we are staying private for now, as it works well for the hackers and us: we spend most of our time dealing with valid findings, and the hackers are more likely to get a payout if they submit reports to our program. Note that while we have a higher share of valid reports, big public programs like Shopify get more reports per month than we get per year and just in September they paid more than we have done in one year. This means that it is hard to compare the effects. We may have much faster response times and a higher likelihood of bounty payouts, but Shopify is probably getting way more testing coverage.
We do like the dual model that Visma has put in place, where new teams/services are first onboarded in the private program before they graduate to the public program when they are mature enough to handle it. With their public program, they can publicly disclose reports on HackerOne.com, and that is good for transparency and cool for hackers to showcase their findings. We also do private disclosures in our program so that the participants can look at each other’s reports and learn from them. Private disclosure also helps with transparency inside the program, as the participants can see that they are being treated fairly regarding bounty payouts. By enabling private disclosures, we have also had several hackers discover new vulnerabilities based on information in old reports, and come up with new bypasses for already “fixed” flaws. So private disclosures is a must if you are running a private program, we all win something on it.
How do you know if you are ready for a bug bounty program?
That question is worthy of its own blog post, and to get some tips we can refer you to the great blog post by Leif Dreizler about how they run their program at Segment, as it is the definitive guide on how to start and manage a program.
How do we keep hackers happy?
Based on the severity from low, medium, high and critical, we pay up to $150, $300, $1000 and $3000, respectively. The gap between medium and above is large, and that is because we want to reward higher impact reports appropriately, and also compete with other programs for the talent. We cannot compete directly with large programs like Shopify on bounty payouts, as they pay up to over 10x as much for critical findings. Still, we pay more than other big tech companies like Spotify(not to be confused with Shopify) which has high and critical payouts set to $700 and $2000.
Besides focusing on the payouts, there are a lot of other things we can do to keep hackers happy. Many hackers experience slow triage times, and also a very long time to bounty payout, and that can be frustrating. We have heard stories about reports not being triaged in days to months! We strive to triage the reports as quickly as possible and pay the bounty on triage after an impact assessment. For common bug types, this process is quick, as we piggyback on previous similar reports, example: reflected XSS triages in seconds, while some business logic error bug depends on the impact of that specific flaw, which we need more time to determine. Data from our program also show this: simple bug reports that are easy to verify, like XSS and CSRF has an average triage time of 4 and 6 hours respectively, and vulnerabilities that are harder to verify, like HTTP Request Smuggling and Business logic flaws averages 27 hours and 19 hours respectively.
For all reports, our median triage time is about 45 minutes, and over 80% is triaged within one hour, and based on feedback from our program’s hackers, we can safely say that our triaging times satisfy and motivate. We have had many positive comments on our response times, and some even say that is one of the reasons they like submitting reports to us.
If you want to join our program, or chat about bug bounty programs, please send an email to emil.vaagland at finn dot no.
