Acknowledgments: Special thanks to Sock and Samara LeMerle from Code4rena, Oliver Hörr from Hats Finance, Konstantin Andriotis from Hexens, Bogdan Stanciu from Runtime Verification for the review and great feedback which contributed to this article. Also many thanks to Dominic Bruetsch from ChainSecurity, Gökhan Er from IOSG Ventures for their feedback and contributions.
Security Life Cycle Shift Left: Why We Need Community Security
If we consider the blockchain ecosystem as a supercomputer program system, the complexity level of the current system has far exceeded that of five years ago. The complexity of the infrastructure and the intricacy of smart contract logic at the application layer have significantly advanced, meanwhile the interactions between smart contract contracts become much more frequent. The blockchain system now hosts a significant amount of assets, leading to increased discussions in the blockchain security community about security throughout the development lifecycle and more importantly, most of the code is open source for both good and bad actors.
Previously, we posted what the entire security lifecycle of a blockchain program looks like: from program pre-deployment to post-deployment, there are tests before an audit, third-party audits, monitoring and updating audits in the whole circle. Almost all the work was finished by conventional security firms, but now we are witnessing an interesting trend that more and more projects and protocols are also willing to work with community driven security platforms mainly because they want to have more eyeballs reviewing their codebase.
Security is a dynamic arm race game between attackers and defenders. As Bruce Schneier said “Security is a process, not a product. It’s a way of thinking about the world, and it has to permeate all aspects of the software development process.” In the world of Web3, where almost everything is open source and transparent, like a dark forest, a protocol that aims to survive in the long term inevitably has an eternal need for security. Until now, most of the popular web3 products have some degree of financial attributes, and we all know that trust is the most important asset in the finance space. User trust can only be gained once.
Hence, we are frequently saying shift security left — — checking for security issues earlier and letting them reveal in the development process. In this sense, game theory and contest model driven crowdsourcing security could be a final guard of the game, and even more, it could be complementary to the conventional security in the whole program life cycle in improving overall security levels.
The Landscape and Market Size
In the Web2 field of security, community driven services like bug bounties or vulnerability rewards programs are not new things. Many big technology companies like Facebook, Google, Microsoft, etc., deploy bounty programs for their in-house security teams. In 2012, the third-party platform HackerOne was founded by several hackers after their reports of hundreds of flaws were ignored by those big companies. Now they are the top player in the Web2 Bug Bounty Platform market with a total paid out bounty of $230 million (in 2021, 2022 paid out $150 million), 65,000 reported vulnerabilities in 2022, 1 million registered users/hackers and 1000+ clients rely on their service.
In Web3, community driven security service not only has bug bounties, but also competitive audits. According to Hats Finance, in 2022, there was $50 million total paid out to hackers on bug bounties and competitive audit platforms. The average platform service fee rate ranges from 10% ~ 30, so very roughly estimated that the current market size ranges from $5 million to $15 million, which shows the growth potential.
Another interesting signal is in regards to competitive audit platforms: More and more clients are willing to directly pay for community-oriented security solutions before they launch their protocols on the blockchain (Code4rena has finished 200+ audits, Sherlock has finished 80+) .
A key difference between bug bounty platforms and competitive audit platforms like Code4rena is the stage in the development lifecycle in which they come into play. Bug bounties generally occur after a project has been deployed, employing a pay-to-find business model. Competitive audit platforms like Code4rena and Sherlock typically go to work prior to a project’s launch, charging a flat fee as a means of guaranteeing audit rewards for security researchers. The most notable example so far is Opensea, which, before launching their new product Seaport, chose to set up a $1M prize pool with Code4rena over a conventional security firm. Their rationale was that they wanted to utilize the thousands of security researchers available in Code4rena’s community-strengthened model to review the code, with data backing up this approach’s efficacy.
Currently, there are 57 auditing companies in the market, with top 10 companies in total generating more than $100 million annual revenue. In today’s increasingly saturated conventional security auditing market with intense competition for talents, technical tools, business clients and marketing and brandings, could community driven services bring additional value to the whole security market? I believe bug bounty and competitive audit platforms have their own unique position in the security life cycle and a huge potential to grow the market.
Bug Bounty vs. Competitive Audits
Bug bounty and competitive audits have different positions in the security life cycle. Bug bounty usually positions in the program post-deployment stage and competitive audits is in the pre-deployment stage that could be better prepared for the conventional audit or double review to find what other audits can’t find before officially launching the protocol.
Conventional Audit vs Audit Competition
Although in the sense that clients have a fixed budget for security auditing, the traditional methods and community methods will have some competition, I wouldn’t think the audit competition will be a direct replacement to traditional auditing (or vice versa) but I can see the potential that they can work as a hybrid model to complement each other. Both approaches have their own advantages and disadvantages.
Conventional audits are a very mature business, the good quality firms or teams’ strongest points are their expertises, discipline, structure, methodologies to provide a comprehensive security analysis. But, in the bull market, we also heard the complaints from projects that they have to wait for a long time which will significantly affect their project progression plan, meanwhile the number of eyeballs to review the code is limited.
In comparison, competitive audits use a much more intensive contest model with strong economic incentive to attract more security researchers to review the code because there is already evidence that shows the more auditors, more issues will be found. However, it is important to acknowledge that this emerging decentralized model is still in its infancy and faces practical challenges. These challenges include effectively organizing the work, resolving disputes, establishing a fair rewards structure, and implementing effective governance for project leaders etc. These areas require further refinement and improvement in the future.
Enjoy the Power of the Wisdom of Crowd
In the vertical of crowdsourcing security services, Immunefi is the most well-known platform that focuses on the bug bounties. In competitive audits, Code4rena is the most reputable platform.
We also can see some new comers in providing community driven security, for example Hexens recently published their new vision to build a web3 security platform — Remedy which also proved that top security experts also have seen some shortcomings of the existing security life cycle and want to reshape it. Also, Spearbit recently launched Cantina mainly to build a highly-efficient aggregator for web3 security services.
Some notable players in competitive audits:
Code4rena (Completed 230+ audits)
There are three types of roles in Code4rena:
- Wardens audit the codes. Every individual from hardcore security engineers to green developers trying to get more experience can register as a Warden to take part in the open auditing contest.
- Judges are usually the most elite engineers in the C4 community. They decide the severity, validity, and quality of findings and rate the performance of wardens.
- Sponsors are the projects such as Opensea, Blur, ENS, Chainlink, etc put up prize pools to attract wardens to audit their project’s code. Projects can also choose from a range of security solutions to best suit their needs, such as invite-only audits for increased privacy.
Code4rena is the leader in this direction that having the combination of community-model + full-time team with a strong background in the security industry, increasing efficiency through modelling and processes. One of the most intriguing aspects of Code4rena’s culture is its emphasis on fostering collaboration and teamwork. Unlike bug bounty programs with the rule to reward only the first person to discover a bug, Code4rena recognizes and compensates all security researchers who report a valid bug in the contest. With the focus being on finding unique vulnerabilities, the Code4rena team believes this approach promotes a healthier level of competition among the contributors, motivating them to uncover everything that could be of value to projects. Additionally, the platform encourages participants to form ad hoc teams, enabling them to work together in their vulnerability-finding endeavors.
When we looked at the track record of the finished contest at Code4rena, the ratio of audited cases being hacked is good (over 154 contests ran by C4, there were 4 exploited but none of them were relevant to C4 audits). Here we put a link (https://rentry.co/audit-leaderboard) that roughly shows the data of conventional auditing firms’ track record (please notice that this data may not be accurate)
Sherlock (Completed 80+ audits)
Sherlock also has similar types of roles (auditors, judges, sponsors) mainly took part in the auditing life cycle. In addition, there is a unique role of Staker in depositing the money into an Insurance pool (current TVL $2.7m, cover $11.8m value) for a fixed term and receiving an APY in exchange for the risk of funds being used (up to 50%) to pay out an exploit at a covered protocol.
In Sherlock, the benefit allocation mechanism is a bit different. Compared to Code4rena, Sherlock has rules that let the Lead Senior Security Auditor and Lead Judge can take a fixed amount of money from the prize pool to properly compensate and incentivize dedicated senior auditors. Also, there is the system of selection and competition for the Leader roles.
Another unique point for Sherlock is about their insurance service. Investors/Stakers deposit their USDC in the insurance pool, and protocol customers can purchase the service. The insurance stakers’ revenue stream will come from: Premiums from protocol customers + Interest earned from depositing the fund into other DeFi pools (Aave, Compound etc.) + Sherlock token Incentive.
Hats Finance (Completed 8 audits, 31 bug bounties）
Hats Finance combines bug bounty and competitive audits model together and they are the first platform to bring products on-chain. Another point sets Hats apart is their token utilities and their mechanism to address disputes that commonly arise in crowdsourcing security.
To gain governance rights and additional utility, users can lock their tokens in a voting escrow. Stakers have the option to boost a single vault or distribute their boosting power among multiple vaults. This approach reduces protocol fees, increases liquidity mining incentives, and attracts more white hat hackers and liquidity providers to the vault. Furthermore, Hats is implementing a more decentralized governance process (expert committee and decentralized courts maybe involved) to safeguard the interests of security researchers. The community is currently designing an on-chain process that allows contributors to challenge the committee’s decisions in cases of conflicts of interest during bug bounties and audit competitions.
What Security Researchers Mostly Care About?
After reviewing different platforms and engaging in discussions with diverse security researchers and project founders, we are of the opinion that the main objective of these platforms is to establish a healthier, streamlined, and user-friendly communication channel between ethical hackers and projects. To gain a deeper understanding of the motivating factors, challenges, and overall benefits for security researchers, I have summarized the following table based on an academic survey conducted with 159 web2 bug hunters.
Also, some discussions in the community also show some concerns from the hackers about the current platform, for example: majority of security researchers (even some senior researchers) prioritize quantity over quality by aiming to discover as many issues as possible, time limitation is too short etc. Additionally, from the clients’ perspective many of them still don’t have enough knowledge about how competitive audit platforms work internally, for instance who will be allocated to lead which projects based on what governance process because the program leads (both auditor and judge) are the key for the quality and productivity.
Tooling for the Security Researchers?
With the popularity of LLMs like GPT, we heard a lot of people discussing whether AI can also replace human’s security audits. Based on the conversations with experienced security experts, they generally believe that GPT is difficult to directly replace human expertise. While language models may be able to detect some low-hanging fruit (easily discoverable issues), more complex tests such as static analysis and dynamic analysis (such as Formal Verification) still require a lot of human intelligence. These more advanced methods require experts to define the properties, expected status and testing domain in advance which is impossible to be directly finished by AI at this stage. For example, when doing a Formal Verification the most difficult part is to write a good specification that represents general ideas about a program’s execution (i.e., what the program should do).
However, we also have seen positive evidence that LLMs can make the audit process more efficient and effective. A recent academic paper shows LLMs have the potential to significantly improve the process of quickly and thoroughly examining a wide range of potential attacks on smart contracts. However, their performance entails a substantial fraction of false positives, and manual auditors are still necessary. Additionally, we have seen many cases show that security people are actively exploring the usage of AI and ML.
Another interesting perspective to consider is the ongoing arms race between hackers and auditors. While LLM tools can provide value to protectors, it raises the question of whether attackers can also leverage these tools. Currently at least the abundance of low-quality bug reports resulting from the misuse of GPT has become a headache for these platforms.
Security is about People
People tend to perceive the programme as a cold, mechanical, and logical thing. They believe that improving system security only requires enhancing the techniques. However, previously we lacked more consideration for security issues from the perspectives of economic incentives and human nature. In the dark forest of web3, in addition to improving the methodologies, skills and tools, we also need a healthy economic system to better incentivize the individuals to contribute their expertise in the life cycle of blockchain programs.
Currently, conventional security audit is a very matured business where brand reputation is the most valuable intangible asset for companies in this field. Over time, we believe that the impact of those good security firms will steadily improve (fully rely on the quality of the product that you provide to the clients). However, the traditional security auditing also faces with the challenges that the business model still heavily relies on talents’ manual work and make it difficult to scale in the short term because they need to balance the quality and quantity when they want to grow their business (we have witnessed some companies have struck in the troubles by too fastly growing the business while providing relative lower quality service and the brand value was negatively influenced). Hence, in the area of crowdsourcing security, I expect the platforms need to put a lot of effort to build a more robust structure in case of facing the similar problem we discussed above.
Will community-driven auditing platforms pose a threat to conventional auditing companies? I think they have some competition but they will coexist in a mutually beneficial and healthy competitive relationship. In the long term, the crowdsourcing security platform should enhance the decentralization of platform governance (as it currently relies on the democratic decision-making of a few elites), establish better processes, disciplines and frameworks, continuously build a community to address future human resource challenges (as the current bear market has led to talent overflow from auditing firms to these community-driven security platforms, but we are uncertain whether there will be sufficient talent when the bull market returns). These are the challenges that the community driven platforms will soon have to address, but the good signal is that we have already seen the clients are quite positive about the crowdsourcing security platforms such as Opensea and L2s.
References:
- Bug Bounties & Whitehats Panel — Devconnect 2022
- Audits: Conventional vs Community — DeFi Security Summit 2023
- https://github.com/code-423n4/org/discussions
- Productivity and Patterns of Activity in Bug Bounty Programs: Analysis of HackerOne and Google Vulnerability Research
- Bug Hunters’ Perspectives on the Challenges and Benefifits of the Bug Bounty Ecosystem
- Bounty Everything: Hackers and the Making of the Global Bug Marketplace
- The Power of Crowdsourcing Smart Contract Security for L2 Scaling Solutions
