Smart Contract Audit: What Security Service is Right for You?

After many good discussions and a spicy panel on conventional vs. community audits at the DeFi Security Summit, we at Arbitrary Execution have come to the conclusion that Web3 has a terminology problem with different security offerings. Viewing a protocol’s options as “traditional audits” or “community audits” is a false dichotomy. There are more options than just these two, and each option has distinct benefits and drawbacks. It’s not an apples-to-apples comparison. This blog post sets out to explain and compare the myriad of security options for project teams, so that you can make the best choices for your situation.

Traditional Audits

Traditional security audits in the context of this blog post are time-boxed security reviews performed by a security firm.

How it works

First, a firm typically scopes the engagement, where they determine the number of auditors and length of time to review your code. Auditors then spend the engagement identifying findings, assigning a severity, and compiling these findings into an audit report. After the report is delivered, a fix review can be part of the engagement, which is a round of review on the code changes to address the findings from the initial report.

Not all audits are created equal

There are dozens of security companies that offer “audits”, but that does not mean the experience, quality of deliverable, or depth of review is the same across firms. Broadly, we can slot firms into two categories: “volume” audit companies, and “depth” audit companies.

Imagine you’re a DeFi protocol with ~1500 lines of new functionality, you’re not a fork of something and a majority of the code is new. What might an audit from a “volume” or “depth” firm look like?

The “volume” audit is cheaper, and its duration is shorter. You have 1, maybe 2 auditors assigned to the engagement, which lasts a little longer than a week. You get a report at the end, but it is light on details. There are many “pass / fail” components in the report that seem to be generated by automated tooling. Finding descriptions and fix recommendations are hard to understand. Furthermore, very few findings were identified that are protocol-specific or high severity. The audit process is opaque to you, and there is little communication between you and the audit team during the engagement.

The “depth” audit costs more and has a longer duration. It is scoped in such a way that at least 2 auditors have enough time to work through the entirety of the codebase with manual review and some automated tooling. The report you receive is detailed, easy to understand, and contains detailed write-ups and fix recommendations for each finding. Because the team had time to build a strong mental model of the protocol, they were able to identify high severity and critical severity findings that an attacker could have leveraged to harm the protocol. There is constant communication between you and the audit team throughout the entire process.

Which of these options are “better”? These “audits” are so different, it’s not worth comparing the two. If you’re looking to check the box of “I got an audit”, the first option from a volume auditor might be fine. However, if you are concerned about the security of your protocol, we recommend a depth-focused audit.

Pros

• Working with a reputable security firm provides guarantees to the level of expertise and professionalism.
• Companies like AE take operational security seriously. You won’t see our auditors looking at your code in public places like an airplane or at a conference.
• Good traditional audit firms will staff auditors with the best experience for your project. If there is a doubt or gap in knowledge, assigned auditors can leverage the expertise of the rest of the company to answer questions or double-check their findings.

Cons

• The price for a depth audit can be large enough that it should be budgeted for.
• For larger projects, these engagements can take weeks. Fix reviews add even more time, so you need to plan in advance.
• Some firms are more difficult to book quickly than others.
• Audits are a point-in-time review, so the more your code changes after the report, the less relevant it becomes.

Choosing the right auditor

Picking an auditor can seem daunting, but it doesn’t have to be. As you approach traditional audit firms, ask the following questions to understand what you want out of an engagement:

• Are you just looking to check a box? (We hope not!)
• Are you concerned about the entirety of the codebase, or just certain components?
• Do you want guarantees around process and professionalism?
• Do you like the report format?

If you’re considering a security auditor for an upcoming engagement, check out our blog post on how to choose the best audit firm.

Solo Researcher Security Assessment

Another option for getting a review is contracting an independent researcher to review your protocol code. From Twitter observations, protocols are going this route for a quicker and sometimes more affordable review.

How it works

This engagement typically involves a single person reviewing the code, and either relaying findings in a report or sharing findings directly to the protocol team in the project repository or chat. Some solo researchers will go as far as fixing the findings themselves.

The biggest challenge with solo researchers is that the output is entirely dependent on the individual conducting the review. Some solo researchers do great work, but it is your responsibility to evaluate your choices.

Pros

• An individual researcher can sometimes move more quickly than a traditional audit firm or provide more flexibility from a scheduling standpoint.
• This can be a great option when you know a particular researcher with deep knowledge on a topic or protocol. For example, a solo auditor with previous experience with a specific DeFi protocol could be a great candidate to review your integration with the protocol.
• Individual researchers are generally the cheapest option and may be open to accepting tokens for early-stage projects.

Cons

• The review process and expertise is completely dependent on the individual.
• You do not have guarantees around the engagement and operational security without a contract.
• It is your responsibility to evaluate the researcher and set expectations around the work performed.
• One auditor means a single pair of eyes looking at the code.
• You’ll lose out on the creativity of a team and the fact that different researchers find bugs in different ways.

Audit Contests

Audit contests, also known as community audits, have the same goal of identifying and mitigating vulnerabilities, but go about it in a different manner. Contest platforms facilitate community review of a protocol.

How it works

First, you send your code to the platform for scoping, where they determine the contest duration and a start date. Once the contest begins, the code and engagement information is released to participants (this could be a private group, or the public) and people begin reviewing your code. Findings are submitted up until a submission deadline for the contest.

After the submission deadline, findings are reviewed by the platform team or a third party where finding validity is determined. Some platforms allow the protocol team to argue against findings; others do not.

Many contest platforms have FAQs that go far more in-depth on their inner workings. We suggest you check them out for additional information.

Public audit contests have some unique benefits, but they are not guaranteed for every contest.

Pros

• A large number of findings can be generated in a relatively short amount of time.
• With public contests, there is the potential to tap into a wide network of high-performing solo security researchers.
• For example, you might get researchers with deep knowledge of a particular protocol or contract pattern.

Cons

• You have no guarantees on 100% code coverage on review.
• Pricing for community audits has gone up and can be comparable in cost to traditional security audits.
• For public contests, you have no guarantees on community participation in the contest. There’s always the possibility that many of the top performers on the leaderboard might not show up for your contest.
• When researchers do devote time to the contest, you do not have guarantees around the amount of time they spend auditing. Is 40 researchers performing 1 hour of work the same, or better, than 1 researcher performing 40 hours of work?
• Low median payout per participant attracts less experienced, lower skilled researchers. High skilled researchers can command higher compensation and are less likely to participate.
• Platforms have different criteria for valid issues, so depending on the platform you choose, certain issues may get thrown out of the report.
• While contest platforms have measures in place to cut down on bogus submissions, we’ve observed that hundreds of findings have to be reviewed to narrow down to dozens of issues. This can lead to review fatigue and you run the risk of findings falling through the cracks.

Public vs Private Contests

Most of the pros and cons discussed above apply to public audit contests, where anyone registered on the platform can review your protocol code during the contest. Platforms also offer private contests, which are limited to a smaller, more trusted set of researchers, and “invite-only” contests, which are an even smaller subset. As the audience of researchers shrinks, the more audit contests start to resemble traditional security audits.

Security Retainers

Similar to an audit, a security retainer can take on different forms.

The first form is still audit-centric, where a client pays up-front for guaranteed audit slots. For example, Alice pays Bob’s company once a year for a guaranteed audit slot every 2 months. We won’t focus on this type of retainer in this blog post, as it shares the same pros/cons of traditional audit engagements.

The retainer style we’re more excited about at AE (and offer) is a holistic and continuous approach. This retainer is a bucket of hours or a number of tasks per month that can be used at different stages of the development life cycle. These retainers can include, but are not limited to, the following tasks:

• Design Reviews
• Changeset (PR) reviews during active development
• Focused reviews of existing components
• Fuzzing
• Monitoring, detection, and automated response

The big upside to a retainer relationship is that it covers areas that are often excluded from an audit, like unit testing, and are a great way to get your codebase audit-ready. It’s important to remember that this style of retainer is not a complete replacement for a time-boxed security review.

Pros

• Retainer customers get the benefit of having access to a diverse team of researchers on their side.
• Retainers are useful at any stage of the development lifecycle.
• These engagements can provide security improvements to areas that are typically excluded from an audit scope.
• Improvements to security posture are compounding, as the retainer team builds a deeper and deeper understanding of the project as the engagement progresses.
• Helpful during active development, as reviewing can happen continuously.
• Can be a lower up-front cost for security compared to an audit.

Cons

• Retainers are not a replacement for a comprehensive audit.
• Changeset reviews can catch many problems, but the context of these reviews can be limited.
• When only reviewing PRs you may not focus on certain areas of the codebase.

So what’s best?

As we’ve seen from the descriptions, each security offering has distinct benefits and drawbacks. Given infinite time and money, protocols would benefit from doing all the things, but we realize that advice is at odds with moving quickly and keeping costs low during development. The “right” option also depends on your team’s experience level and understanding of security. Depending on these factors, teams can choose to shoulder more of the security burden or accept looser guarantees around the offering experience.

Based on burden and offering guarantees, AE has arrived at three categories for offerings: do it yourself (DIY), do it with you (DIWY), and do it for you (DIFY).

Do it Yourself

If you are a development team with a deep understanding of security, you might already have an internal security team or be willing to put in more work evaluating offerings with looser guarantees around the experience. Do it yourself options include:

• Solo researcher security assessments
• Audit contests

As a general rule, the more time and effort you are willing to put into the DIY offerings, the more value you will get out of them. If your team is relatively inexperienced in Web3 security, or this is the first time you are engaging with external parties for security offerings, we do not recommend only choosing DIY offerings. Protocols that have experience with undergoing other security offerings, have engineers who have an understanding of Web3 security, and dedicate time to answer questions are the most likely to be able to extract the maximum amount of value out of the DIY offerings.

Do it With You

If you are a newer company that lacks firsthand experience with security offerings, you may not have the knowledge to determine whether or not you will get value out of DIY offerings. Do it With You offerings provide stronger guarantees around the experience and will require you to invest some of your own time into security (less time than DIY options). These options can include:

• Retainer (audit-centric, where you pay for X audits over some time period)
• Traditional Audits

DIWY options include traditional time-boxed security audits and retainer-style audits, which typically consist of paying for multiple audits over a period of time. For DIWY options, engagements might be scoped in such a way that they only focus on the most sensitive components of your project (for example, only the on-chain components). In this case, your team is responsible for securing the out-of-scope components. While this may keep costs lower than a DIFY option, audits are only able to secure the code in scope at a fixed point in time.

Do it For You

Perhaps due to team size, you don’t have the bandwidth to do deep security work leading up to your launch, but have the desire to secure all aspects of your project. In this case, higher-touch offerings are your best options, which could include more conservatively scoped engagements such as:

• Retainer (holistic / continuous)

DIFY retainer offerings are typically packaged as X hours per month, where one or more dedicated security professionals complete a variety of tasking.

You might be asking why aren’t traditional audits in the DIFY category? As we mentioned before, audits are only able to secure the code at a fixed point in time and don’t focus on all the components of a project or its development lifecycle. Furthermore, there’s typically a lack of follow-on questions/engagement once the audit is completed, unless you pay for another audit.

Combination Approach

Pairs of security offerings spanning the different categories can provide wide coverage for most teams. Consider the retainer + traditional audit route:

• The retainer partner works closely with you throughout the development process, catching bugs early, and improving all areas of the codebase including testing, CI, documentation, and monitoring. The retainer partner helps ensure this work is complete leading up to the traditional audit.
• When the time comes to get your top-to-bottom security review, the codebase is well-architected and clean. This lets the auditors come up to speed more quickly and focus on deeper issues, if any. Depending on the nature of your retainer relationship, an audit contest might better fit your needs at this stage.

Closing Thoughts

The point to reiterate is that no single security offering is necessarily the best — they are different. If your budget only permits a single choice, you need to weigh the benefits, drawbacks, and risks of each option, and select the category of offering that best aligns with your team and project.

AE does audits and retainers

Arbitrary Execution is a group of experienced security professionals providing high-quality smart contract audits that suit your needs. Our extensive background in Web2 security, along with our domain expertise in Web3 security, enables us to provide security services for smart contracts and off-chain code. We work closely with protocols like Milkomeda and Premia through retainers to bring security into every stage of their development lifecycle. We have performed audits for Aztec and Decent to secure their code. Check out our publications repo to see more of our work.