Our unique multi-layered approach to security audits

Audits Work, if Done Right

Smart Contracts are immutable and transparently visible on the blockchain. They also usually control a lot of value. These factors convert smart contracts into a security risk. It did not take long for this to become apparent. At least since the infamous 2016 DAO exploit, security audits have become a standard approach to reduce the risks involved.

However, a quick look at the Rekt leaderboard shows that not all audits seem to work. Plenty of audited projects have been exploited and are listed together with their audit provider. Even reputable auditing firms can be found on that list. Audits are only one part of the security puzzle and cannot guarantee that no vulnerabilities remain in the audited code. Therefore, inclusion in this list does not mean an audit firm does not provide good services. However, we do believe that keeping our two auditing brands, Oak Security and Solidified off this list, says something about the quality of our work, particularly, considering that we have audited hundreds of projects since 2017.

We attribute this to two main reasons.

Firstly, we have deliberately grown slowly, in line with the available talent, insisting to employ only the best security professionals. To do so, we have had to reject venture capital investors that wanted us to grow quickly. We have also had to reject clients due to capacity limits.

Secondly, we have developed what we believe to be a unique process to go hand in hand with our decision to employ only the top security researchers. Our process aims to leverage their talent in a way that reduces the remaining risks as much as possible. Let’s have a look at this process.

Multidisciplinary Team

We maintain a multidisciplinary pool of highly qualified auditors. We deliberately look for diversity in educational backgrounds. Whilst all of our auditors are, of course, well-versed in blockchain technology, they hold degrees in a variety of fields and specialize in subjects, such as computer science, mathematics, cryptography, or economics.

This allows us to staff projects according to their nature. For instance, a DeFi protocol with an economic model will have an auditor with a strong background in economics. We are convinced that auditing financial software requires deep knowledge of financial systems. Similarly, an audit of a ZK-rollup processor will have a cryptographer on the team.

Another principle we follow is to have an overall process (more on which below), but allow auditors to freely choose their methodology for finding vulnerabilities within that overall process. We believe it does not make sense to hire the brightest of the brightest and then tell them how to work. Our process, therefore, leaves a lot of room for auditors to apply their methodologies. We specifically staff projects striving to have a wide variety of methodologies applied in parallel. One member of an audit team might use a formal modeling approach whilst another auditor implements fuzz tests to look for certain edge cases that might break a protocol. At the end of each audit, there is a phase for knowledge transfer, in which auditors discuss their results to learn from each other.

Blinded Process

Our unique auditing process is based on blinded, independent reviews using a mix of approaches in parallel. The guiding principle is that every line of code should be covered by at least three pairs of eyes, which means there must be three or more auditors on each project, all working in parallel.

Our process is blinded, which means that the auditors work independently during the first phase of the project, initially not sharing their results. Depending on the technology stack, there are some mandatory steps, including static code analysis, test coverage verification, and running through checklists of common issues. However, this is kept lightweight on purpose to leave room for individual processes.

The auditors will reveal their findings in a Consensus meeting, collaborate on open leads and put together the final report which lists issues encountered together with recommendations.

Our blinded approach has several advantages:

• Auditors do not bias each other. If issues are shared as they are found, other auditors may get distracted by looking into them. That could lead to issues in the same location to be missed. It is also easy to introduce false positives through this type of bias.
• Unintentional freeloading is discouraged. Of course, our auditors are professionals, and we do not believe they would intentionally skip parts of the codebase. Nevertheless, it is very easy to subconsciously not dive deeply into a really hard part of a codebase, believing this to be covered by someone else. Friendly internal competition ensures that nobody wants to be that one person at the debriefing meeting who has missed an important issue. Therefore, everyone ensures that everything is covered deeply.
• We can monitor the contributions of our auditors and apply quality control measures. This can be used to ensure high and consistent quality across projects. For instance, good overlap in findings increases confidence in our audit, whereas low overlap might make us go back and dive a bit deeper. We can also identify weaknesses and train team members accordingly.

Once the compiled findings have been delivered to the client, all auditors remain available for another three weeks and we review the implementation of fixes and update the report accordingly.

Transparency

We believe that security requires transparency in process and result. Therefore, we have implemented the following measures:

• Our auditor team is visible to the client. Auditors participate in calls and message interchanges and can ask and answer questions directly. This makes communication easier and allows the client to see who they are dealing with.
• All our audit reports are made public under a Creative Commons Attribution No-derivatives License. This means that each report is owned by us as, an independent party. It can be freely shared and is visible to the community, but cannot be modified by the client.
• Our final reports contain all issues identified, unless these turn out to be false positives. Rather than removing fixed issues, we mark them as resolved.
• We allow the project team to add a team reply to each issue, in case they do not agree with our finding or choose not to address an issue.

The final version of our report is transparently published to our GitHub repositories (https://github.com/oak-security/audit-reports and https://github.com/solidified-platform/audits).

Final Thoughts

The process described above has been developed over the cause of seven years and has stood the test of time. We truly believe it helps to reduce the remaining security risk significantly. However, we are continuously evolving and adapting. We find that our approach (and audits in general) works best in combination with other approaches, such as bug bounties and open competitions.

Please reach out to us if you have any questions about blockchain security. We are always happy to chat.