Layer 3: Testing and Code Review

The Art of Blockchain Testing: Mastering Functional, Static, and Fuzzing Approaches

TriWei.io
6 min readJan 14, 2024

This chapter marks a significant shift towards proactive vulnerability detection in code, employing a range of techniques and tools tailored to suit the specific needs of each project. From the essential processes of functional testing and static analysis to the advanced strategies of fuzz testing and symbolic execution, the article offers a comprehensive guide for developers and auditors in the Web3 space.

1. Functional Testing

Functional testing, the process of verifying that your smart contracts and blockchain-based applications behave as expected, can be executed both manually and automatically. However, manual testing is often less efficient and more error-prone. Automated testing, conducted through meticulously crafted test scripts, is essential for maximizing test coverage.

Functional testing typically consists of three phases:

  • Unit Testing: This phase focuses on the smallest parts of the application, typically individual functions or contracts. In unit testing, each component is tested in isolation to ensure it performs as expected. This granular approach helps in identifying and fixing issues early in the development cycle.
  • Integration Testing: Here, the interaction between integrated units/modules is tested to expose faults in their interactions. This phase is crucial for ensuring that the combined units function together seamlessly.
  • System Testing: In this phase, the entire system undergoes comprehensive testing on a Testnet. This includes simulating full user action flows to validate the system’s performance under real-world conditions.

Re-running these tests after every code commit is vital, even for minor changes, to preempt the ‘butterfly effect’ where small modifications can have significant unforeseen impacts.

To assess the quality of these tests, mutation testing tools like SuMo, designed for Solidity-based smart contracts, are invaluable. By introducing minor code alterations and checking if the test suite detects them, tools like SuMo help enhance the robustness of your tests. Another recent mutation testing tool is Certora’s Gambit, open source and well-performant.

When it comes to platforms for functional testing, Hardhat and Foundry are the industry benchmarksRIP Truffle 2017–2023. These platforms offer similar functionality but differ in terms of speed and the programming languages used for writing tests.

Foundry’s tests are written in Solidity, making it a suitable choice for those less experienced with JavaScript and TypeScript. However, some tests may require functionalities that are challenging to implement in Solidity, while it would be easy to import a JS library on Hardhat.

Foundry, written in Rust, is renowned for its exceptional speed compared to Hardhat, and it also offers native capabilities such as fuzzing, making it handy to handle the whole testing process with the same tool.

2. Static Analysis

Static code analysis is a critical process for debugging code without executing it. It involves the use of specialized tools to detect issues in the codebase.

We will overview two of the most prominent static analysis tools: Slither and MythX.

Slither is widely recognized as the benchmark in static analysis within the Web3 development sphere. As an open-source tool, it not only offers robust functionality but also provides comprehensive documentation, facilitating the addition of custom detectors for specific vulnerability patterns. An exemplary implementation of this capability is Slitherin, developed by Pessimistic.io, which is a collection of new detectors tailored for audit purposes.

On the other hand, MythX adopts a more holistic approach by integrating static analysis with dynamic analysis and symbolic execution. Although it is a paid software, our extensive experience suggests that, in terms of vulnerability detection, its efficacy tends to fall short when compared to Slither, which relies solely on Abstract Syntax Tree analysis.

3. Fuzzing

Fuzz testing is a powerful automated method used in smart contract testing. It works by sending a large number of random inputs to smart contracts. These inputs, which might be incorrect or improperly formed, help uncover hidden vulnerabilities by causing unexpected outcomes. To make fuzz testing effective, it’s important to first establish a set of rules or ‘invariants’ that the testing will try to break.

There are two types of fuzz testing: stateless and stateful. Stateless fuzzing tests individual functions and is usually done before stateful fuzzing, which tests sequences of operations within the contract. This two-step approach is effective because fuzz testing is great at finding issues, but some problems might only show up under very specific conditions. Testing individual functions first increases the chances of finding these kinds of issues. For stateful fuzzing, you want to design a set of conditions or rules that remain constant and must be satisfied during the whole fuzzing process. Such properties, named Invariants, should always hold true, no matter what input the program receives.

It’s important to remember that fuzz testing requires a lot of computing power. However, the success of fuzz testing doesn’t just depend on how powerful your computer is. The way you set up your invariants is also crucial. Therefore, it’s important to spend enough time and effort to define these invariants well.

Fuzz testing can be tricky: it looks great at first, but when you start you might face challenges like setting up complex smart contracts, working with existing protocols, dealing with data feeds from oracles, and other specific limitations depending on the tools you use, like time checks in Echidna. The difficulty of these challenges can vary based on the tools you choose for your fuzz testing.

Overview of Fuzzing Tools

While coverage is a critical factor, the user experience and capabilities of these tools are equally important.

Echidna, created by TrailOfBits in 2018, has long been considered the benchmark in fuzzing tools. However, the landscape is dynamic, with tools like Foundry and Harvey also gaining prominence. Notably, a benchmark study conducted in April 2023 identified Harvey as having the highest coverage. It’s important to note, though, that this research was conducted by Consensys, the developer of Harvey. Consensys has also introduced the Diligence Fuzzer, a user-friendly, cloud-based tool that integrates Scribble for defining invariants and utilizes Harvey for the actual fuzzing process.

Another noteworthy tool is Medusa, also developed by TrailOfBits. Designed with Echidna’s limitations in mind, Medusa introduces advanced features like parallel fuzzing and a testing API. Although still in the experimental phase and not recommended for production environments yet, Medusa shows great promise and is definitely a tool to watch in the evolving space of fuzz testing.

Given that these tools are constantly evolving, a detailed comparative analysis might quickly become outdated. Therefore, it’s advisable to conduct your own research to determine which tool aligns best with the needs of your project.

4. Symbolic Execution

Symbolic execution is a technique that explores all possible execution paths of a program without actually executing it. It treats variables symbolically rather than as fixed, concrete values.

This approach can be combined with static analysis to perform concolic testing, achieving even more accurate results.

Although symbolic execution is often overlooked in the Web3 community, it can be a valuable addition to your testing process. To delve deeper into this topic, you’re encouraged to read this comprehensive article by SaferMaker on HackMD.

5. Formal Verification

Formal verification is a rigorous process that evaluates the correctness of a system based on formal specifications. It allows you to verify whether a smart contract behaves as intended by using formal modeling and specification languages.

While formal verification is a powerful technique, it’s often considered overkill for most Web3 projects. It involves translating protocol logic into a mathematical representation, which can be complex and resource-intensive.

If you wish to explore formal verification further, the Ethereum Foundation provides a detailed resource on the topic: Formal Verification for Smart Contracts.

As we conclude our exploration of the third layer in our seven-part series on Web3 security, remember that this is just the beginning. Each layer offers unique insights and strategies crucial for the security and success of any Web3 project. Continue your journey with us to uncover more essential aspects of Web3 security.

Dive into the remaining layers and enrich your knowledge at triwei.io/education. Stay informed, stay secure, and lead the way in the Web3 space.

--

--

TriWei.io
TriWei.io

Written by TriWei.io

Three experienced solo auditors joined to create TriWei, a Smart Contract auditing firm. Lean process, high quality, competitive pricing. www.triwei.io

No responses yet