An infinite game is one where each player’s move prompts continual adaptation and strategy refinement by their opponent. Security in the blockchain space is one such example of this type of game. Here, developers must execute flawlessly and their opponents triumph by exploiting a single mistake.
Hackers tirelessly search every layer of the stack in the ever-expanding world of cryptocurrency, where over a trillion dollars in assets presents a lucrative target. In 2022 alone, approximately 4 billion dollars of cryptocurrency was stolen from DeFi protocols and bridges.
As developers and hackers continue to push each other, growing more sophisticated in their strategies, they design systems that better resist attacks, and attacks that better resist the new defenses. The intricacy of this high-stakes game did not materialize overnight. It’s a story written over a decade, marked by lessons that required billions in losses.
To fully grasp the dynamism and risks of today’s landscape, we must explore the history of smart contract development and tooling. How have these tools and techniques shaped this never-ending duel between developers and hackers, and how did they evolve to become the weapons of choice in this battlefield? Walk with me as we go on a journey that traces back the steps, uncovering the genesis of this game.
A History of Security and Smart Contract Development
In the beginning of ethereum, testing and writing smart contract systems was hard. The tools were rough around the edges, and the ethereum stack exchange had very little knowledge on how to solve problems. There were very few codebases that had good examples of how to test and structure codebases. The solidity compiler was new, and there was even a high severity finding in the compiler that was patched in version 0.4.4. After developers started building smart contract systems, it became clear that standardized frameworks were needed to help build systems faster. The evolution of these frameworks roughly tracks the size and complexity of smart contract codebases. As developers better understood how the technology works, and how to build systems, codebase size and complexity increased over time. As codebase size expanded, new frameworks were needed to handle the increase in complexity.
Truffle
Tim Coulter made the first commit to the truffle framework on June 23rd, 2015. This framework gained wide adoption at the time among the small smart contract developer community. Truffle was better than having to roll a framework from scratch each time a project was set up, and standardized how tests and deployment scripts were written.
Tests written in Truffle projects were of limited value as they didn’t guarantee the system all worked together because components were tested mostly in isolation. There were no fuzz or invariant tests in this framework. It was possible to test random values using js libraries, but fuzzing was not supported by the underlying framework. Additionally, it was hard writing tests as there were numerous bignumber errors, intermittent failures, and context switching between solidity and javascript.
Truffle was recently sunset by Consensys, citing recent hard times. This move, contextualized with lagging adoption of Truffle compared to other tools makes lots of sense.
Dapptools
Daniel Brockman made the first commit for dapptools on May 17th, 2016. The usage of dapptools was very niche, however it was incredibly powerful once it reached maturity with fuzz and invariant tests. This tool was mainly used and developed by Maker engineers, and had a steep learning curve for newcomers. Switching away from truffle, a javascript based framework, to a Solidity based framework took some getting used to.
Hardhat
Patricio Palladino made the first commit for Hardhat on April 19th, 2018. Hardhat was widely adopted and used as it made testing and deployment an order of magnitude easier than truffle with its native support for mainnet fork testing and powerful scripting features. It featured its own Ethereum Virtual Machine (EVM) implementation in the framework which significantly sped up testing times.
Brownie
Ben Hauser made the first commit to brownie on November 2nd, 2018. Brownie never gained much traction as its usage was more niche, and its adoption was confined to developers who like python.
Foundry
Georgios Konstantopoulos made the first commit for foundry on September 9th, 2021. Foundry gained widespread adoption almost as soon as it was released even though documentation was sparse. Adoption ramped quickly because it offered developers all of the features which they were accustomed to with hardhat and more in Solidity. Additionally, Foundry supported fuzzing and invariant testing, which are powerful tools for smart contract developers.
Smart Contract Complexity Over Time
As the tooling for smart contract systems matured, so did the codebase size and complexity. However, doing a statistical survey over all the deployed systems and measuring complexity is infeasible. This would require on chain sleuthing to determine which contracts are currently a part of any given system and a systematic review of all protocols deployed and used. The EVM has a twenty four kilobyte size limit for smart contracts, this means that as systems grow in complexity, developers are forced to split contracts out into separate components that call each other. As contracts are split into separate pieces and composed on top of pre-existing systems, the amount of cross contract calls increases. To measure this increase in complexity, the number of traces per transaction was measured. Over time, there has been a marked increase in the number of contract calls in each transaction, pointing to increasing smart contract system complexity.
Transaction Complexity Over Time
As the Ethereum ecosystem increased in complexity, so did the attacks. In order to find attack vectors as systems got more complex, attackers would need to interface with multiple smart contract systems.
History of Attack Complexity
As developers struggled with tooling as their systems grew more complex, attackers faced similar challenges in building and developing exploits. They too had to contend with systems that were difficult to test, which led to early attacks targeting low-hanging fruit-like functions without proper modifiers or other bugs that were easy to find manually. Later as tooling improved, attackers built complex, multi-step attacks involving flash loans, multiple hops across protocols, and custom exploit contracts.
In analyzing the period from the genesis of Ethereum in 2015 until 2017, a pattern of simplistic attacks emerge, devoid of flash loan complexities, and characterized by issues that developers could easily resolve. Notably, these exploits did not traverse through multiple protocols to realize an exploit, underscoring the rudimentary nature of these attacks. This can be attributed to the immature stage of frameworks for building and testing these systems at the time. A hallmark of this era was the 2016 DAO attack, where an attacker was able to drain around 3.6 million ETH due to a reentrancy vulnerability, showcasing a straightforward exploit. These simple, direct attacks extended to the Parity Multisig Wallet incident in 2017, where a flaw in the smart contract allowed an attacker to take ownership of multisig wallets, resulting in a loss of approximately 150,000 ETH. These attacks from the embryonic phase of Ethereum underscore the simplicity of attack vectors before the ecosystem matured to witness more intricate, multi-protocol exploits in the subsequent years.
From 2018 to 2020, the landscape of smart contract attacks underwent significant evolution, mirroring the growing sophistication of the Ethereum ecosystem. This was further catalyzed by the adoption of development tools like Hardhat and the birth of DeFi. The period started with the SpankChain Attack in October 2018, where $38,000 was lost due to a reentrancy vulnerability echoing the earlier DAO hack. As this era, earmarked as the DeFi Era due to the rise of Decentralized Finance platforms, unfolded, the complexity of attacks markedly increased. In February 2020, the bZx platform was attacked in a series of exploits that siphoned off $1 million, leveraging highly complex techniques such as oracle manipulation. In June 2020, the Balancer Pool Hack caused a loss of $500,000, showcasing an advanced attack using deflationary token bugs. This series of attacks revealed a trend of escalating complexity and leveraging improved tooling. Attackers, now equipped with superior development tools and a deeper grasp of smart contract interactions, were discovering new ways to exploit DeFi protocols.
The current period of smart contracts attacks started in late 2020 and is continuing to this day. Standout issues from this era involve complex price oracle manipulation bugs where attackers manipulate share prices across tightly coupled protocols, donation attacks, and surprisingly, a replay attack from a misconfigured bridge. In November 2020, the Pickle Finance Attack resulted in a $20 million loss due to the exploitation of jars swapping logic, specifically targeting an implementation detail of Pickle’s jars. That same year, on October 26, Harvest Finance was hacked for $34 million. The method involved an attacker capitalizing on pricing discrepancies through flash loans. On February 4, 2021, Yearn Finance experienced an exploit, leading to a loss of $11 million. This attack stemmed from a strategy that allowed attackers to manipulate DeFi protocols and drain Yearn’s v1 yDAI vault. On April 28, 2021, Uranium Finance lost $50 million when an attacker took advantage of a flawed change they made to Uniswap v2, breaking the K invariant. A significant incident occurred on August 10, 2021, when the Poly Network was exploited across multiple chains, causing a loss of $611 million. The hacker managed to exploit contract calls between different chains in the system and escalate their privileges. The following year, in August 2022, the Nomad Bridge suffered a $200 million loss from a simple replay attack; any address could replay an old signature and redirect funds. In March 2023, Euler was hacked with the attacker flashloaning system collateral, borrowing, repeatedly donating funds to the protocol until the account had bad debt, and then liquidating said account from another attacker controlled account. This bug allowed the theft of approximately 195m. The cream finance hack happened on October 27th, 2021 and caused a loss of 130m. It took advantage of an oracle that relied on other DeFi protocols, then executed an attack that involved over 10 steps to set up the price to be manipulated correctly, which then enabled them to liquidate an account with bad debt they controlled.
As tooling, developers, and the smart contract systems themselves became more advanced over time, so did the attackers. Over time, hacks have gotten increasingly complex, requiring intimate knowledge of the workings of each composed DeFi protocol, and how they fit together. Using this knowledge, hackers then construct attacks that rely on the nuances of the individual protocols to extract funds. To find bugs in well known code bases, security researchers and hackers alike chain multiple actions together until a bug is found. As the attacks got more complex, so did the steps developers take to secure their codebases.
State of Security Today
Auditing, which used to be a small one off expense for DeFi companies, has grown into an entire industry with different categories of service providers. Big firms, audit contests and private audits, all exist, and serve important functions in the SDLC. Developers need as many eyes on a codebase as possible to ensure severe bugs are found before going live. Additionally, developer tooling and static analysis has significantly lowered the cost to find well known categories and types of bugs.
Formal verification, which used to have a much higher barrier to entry has now become significantly more accessible for developers as companies like certora offer pay per API call formal verification tooling and a developer friendly language to write formal specifications.
Developer processes have also improved, and the number of tools used to bring a smart contract system to market has increased by an order of magnitude. This brings us to the state of security today.
The Security Stack
Each item a developer adds to the security stack is one more layer of swiss cheese that increases the odds they find the bug before the attacker does. The lower the findings are in the security stack, the cheaper it is to find and fix bugs. Certain types of evasive bugs are harder to find with tools lower in the stack, meaning they have a higher cost to discover.
A helpful mental model is to think about development and a stack of tools that can be used to harden a smart contract system. Each tool a developer incorporates adds an additional layer of protection, increasing the likelihood of detecting and fixing critical bugs before an attacker can exploit them. The lower the layer at which a vulnerability is discovered, the less costly it is to remediate. In an ideal world, where a project is infinitely resourced, all of these layers of the stack can be implemented. In the real world, with resource constraints, developers must pick and choose which items have the highest impact for their project’s needs.
First consider the most foundational layer, threat modeling. This layer is often missed, and allows a developer to construct a mental map of system components most likely to contain vulnerabilities. This could be public or external functions that involve changing state across multiple contracts, or functions that handle user funds. Having a keen understanding and intuition around where things are likely to go wrong can help in designing and redesigning a system if needed. This layer does not directly prevent bugs, but it can help in the subsequent layers to find issues that are further beneath the surface.
Building on this foundation is the first layer of defense: unit testing. Now the developer has a good understanding of where things are likely to go wrong from threat modeling. The first layer of active defense against bugs is to write simple unit tests. Writing these tests are table stakes for building smart contract systems.
The subsequent layer, integration tests, are crucial for simulating code execution against a chainforked network. This layer reveals discrepancies between developer expectations and actual blockchain behavior. Notable examples of bugs that can be caught this way include using a solidity version above 0.8.19 which fails due to the absence of the PUSH0 opcode in layer-2 scaling solutions such as Arbitrum, Base, and Optimism. Additionally, there are issues specific to non-EVM equivalent platforms: for instance, zkSync did not support the Solidity language feature .transfer(), which led to raw Ethereum getting stuck in a contract. These nuanced issues underscore the importance of this testing layer in identifying challenges not evident during the unit testing phase.
At the third layer, we introduce fuzz and invariant tests, employing advanced tools and methods to probe for elusive vulnerabilities. Tools like Foundry natively support these tests but don’t utilize symbolic execution for identifying issues. On the other hand, stateful fuzzers like Echidna use symbolic execution to challenge both contract and system-wide invariants. Another valuable resource is Hevm, originally developed by dapptools engineers and now maintained by the Ethereum Foundation. This tool rapidly identifies property violations on contract bytecode. Symbolic tests are particularly effective for DeFi primitives with minimal external dependencies. However, their utility diminishes in larger, monolithic applications that integrate directly with other protocols. Combined with prior layers, these advanced methods can help find non obvious issues.
Proceeding to the fourth layer, mutation testing tools come into play. These tools, like Certora’s Gambit, alter the program code and check if the existing unit and integration test suite identifies the mutated code change by failing. Failing tests after code mutation confirms the test suite’s quality.
The fifth layer employs static analysis tools such as Slither, MythX, and Pyrometer to identify low-hanging issues in the codebase. While these tools have a high false positive rate, they offer valuable insights with a relatively small time investment. For example, just an hour spent reviewing Slither’s output could flag high-severity issues like reentrancy vulnerabilities or divide-before-multiply problems. A recent study from the Imperial College of London indicated that static analyzers could have prevented $149 million in losses related to reentrancy issues alone, had their output been properly reviewed and findings remediated. This layer is fast due to its automated nature, but requires human review to ensure the accuracy of its findings.
Adding more depth, the sixth layer involves internal code reviews conducted by another engineer within the organization. This second set of eyes provides valuable feedback on design, identifies potential issues in the code by walking through things line by line, and questions the assumptions made by the developer. It serves as a comprehensive review of everything done or not done from the prior layers, finding bugs, filling gaps in test coverage and highlighting design and architecture flaws. This is a recommended template for structuring synchronous code reviews.
The seventh layer introduces external reviewers or auditors in a time-boxed engagement, offering an additional, unbiased layer of scrutiny. These experts meticulously examine the codebase for vulnerabilities and design flaws. If a significant number of issues are identified at this stage, it often signals a lapse in earlier phases of the Software Development Life Cycle (SDLC). For example, if 10 critical or high issues arise despite thorough testing, and reviews, this may indicate shortcomings not only in the testing phase but potentially in the architecture and threat modeling stages as well. This external review serves to validate the security measures and development processes undertaken up to this point.
Taking a formal approach, the eighth layer involves formal verification through the use of specialized tools like the Certora Prover or the K Framework. In this phase, develop a formal specification of the system and leverage SMT solvers on the backend to rigorously examine whether there exists a state in which the program could violate the specified formulas. By applying mathematical logic in this way, one validates that the system conforms to its desired specifications, ensuring the code meets rigorously defined criteria.
Next, the ninth layer introduces auditing contests, a strategy where the codebase is made publicly accessible for a time-boxed audit. Here, security experts from various levels of expertise can submit findings, which are then evaluated by judges who surface only the valid issues in a report format for the sponsor’s consideration. These competitions not only engage a broader community of security experts but also offer a high leverage on security spending by pooling collective intelligence to compete for a fixed prize. This enhances the security measures of previous layers by adding a diversified layer of scrutiny.
The tenth layer incorporates bug bounties, serving as an ongoing audit contest without a time constraint and involving live software with high stakes. Platforms like Immunefi facilitate responsible disclosure, and some protocols offer incentives as high as 10% of the at-risk funds for revealing vulnerabilities. This layer provides a continuous incentive for review of live systems, catching vulnerabilities that might have been overlooked in earlier stages.
The eleventh and more exploratory layer offers a unique approach, allowing ethical hackers and security researchers to exploit smart contracts in the wild. In a single operation, researchers can siphon funds and then deposit them into a secured address, claim a share of the Total Value Locked (TVL) or the maximum fee specified by the project. To legitimize this approach, comprehensive legal contracts are necessary. These contracts serve to indemnify the hackers, absolving them of liability across jurisdictions, provided they return the stolen funds to a designated address within an agreed-upon timeframe. This is an area still under active research, and is worth watching if only to see how these white hat hackers are treated in their local jurisdictions when income is reported from this source.
State of Security in the Future
In the future, when DeFi matures and hits scale, it is feasible that bug bounties exceed one hundred million dollars for zero days. Smart Contract Security will become a multi-billion dollar industry as the arms race between developers and hackers becomes increasingly sophisticated. As the security industry matures, Smart Contract security tooling will become a billion dollar industry in its own right. Audit contest times lead times will compress, with new contests being run daily, and smaller and smaller pieces of code getting reviewed as developers integrate security researchers across the entire SDLC.
Attacks will likely involve over steps twenty to find and exploit a bug across multiple protocols. Composability will significantly increase complexity in new protocols. Table stakes for protocol engineers is having knowledge of multiple DeFi codebases, core algorithms and invariants that allow their primitives to function to avoid the numerous footguns that come with integrating external systems. Additionally, novel attack vectors and bug types are discovered outside of the standard reentrancy and ACL bugs.
The floor of acceptable security best practices is raised, and formal verification, auditing, fuzzing, invariants, mutation testing, internal security reviews all become table stakes for investors to put funds in protocols. Ratings agencies like DeFi Safety become industry trust brokers, aggregating security related information to help non-technical investors understand the relative safety of protocols.
When DeFi matures and hits scale, we could imagine a world where there are 100 security researchers employed by the top protocols that swarm each pull request, feature or upgrade to the protocol, thinking through all the edge cases, writing formal specifications for the system, writing reentrancy tests, leveraging mutation testing, and that’s just on the smart contract side. Frontends and other parts of the stack will also require large amounts of scrutiny as a single malicious package can compromise the entire decentralized application and put user funds at risk as has happened many times before.
Conclusion
As the field of blockchain and smart contract security matures, there will likely be even more tools that we currently haven’t thought of that will emerge to make it less and less likely that bugs make it into production. Established developer teams execute on the lower layers of the stack unconsciously, threat modeling, writing thorough unit, integration, fuzz and invariant tests, and writing formal proofs, while also bringing in security advisors, going through many iterations of internal reviews, getting multiple rounds of audit feedback, and leveraging bug bounties and audit contests to secure their protocol.
Security spend in the next cycle at top protocols will soar as TVL and token valuations for the largest DeFi primitives continue to rise. As the field evolves, new products and services will be developed, driving down the cost of finding bugs that once took security researchers weeks or months to find. Security is an infinite game, and it is a constantly evolving arms race between the developers and hackers to harden systems before attackers get through. It almost feels like overnight, the security industry has grown tenfold. In ten years time, it’s hard to imagine all of the new security techniques that will be invented to secure the world’s wealth stored on chain.
Sources
https://soliditylang.org/blog/2016/11/01/security-alert-solidity-variables-can-overwritten-storage/
https://github.com/trufflesuite/truffle/commits/develop?after=a9ca7ea1bbc125710e6c8b1d49a4b57eb2aad57c+15935&branch=develop&qualified_name=refs%2Fheads%2Fdevelop
https://twitter.com/trufflesuite/status/1704946902393860589
https://github.com/dapphub/dapptools/commits/master?after=ba9fbcf95c01d247a30b26d7e064e8abdfba79db+3375&branch=master&qualified_name=refs%2Fheads%2Fmaster
https://github.com/NomicFoundation/hardhat/commits/main?after=373547d3a2fe27f0caeae3ca9f37404b77fb07af+8717&branch=main&qualified_name=refs%2Fheads%2Fmain
https://github.com/foundry-rs/foundry/commits/master?after=4f661a978914e116937ea9628d475692d89a368f+2525&branch=master&qualified_name=refs%2Fheads%2Fmaster
https://twitter.com/bantg/status/1692467461952582087
