POST-MORTEM DISCLOSURE
CivTrade Hack: Analysis
Vulnerability and incident dynamic
CivTrade suffered an unfortunate hack. Here are all the details, plus some potential learnings
Known Facts as of Now
On July 8th 2023 at 5.42am UTC the security company De.Fi was first to report on Twitter the hack of our CivTrade v2 contract.
As it was the middle of the night in Europe, some CIVians were first alerted as they woke up, namely Hex among our first and most alert community members (maybe actually still awake and following the events) and thereafter both her and Iceman alerted me (DexMan) to the hack.
We immediately paused CivTrade v2 at 7:07am UTC. As we started to look into the incident, this had all the telltale signs of professional black hat people, much more skilled than our “first attackers” in 2021.
Namely, hackers had to:
- Reverse engineer a contract whose source code was not public
- Find a vulnerability without having access to such code
- Build a replica of Uniswap to interact with the contract
- Implement a batch analysis of all contract wallet histories
- Design a batch-execution strategy to drain funds.
Following the industry best practice, we notified the Civilization community, first at 9:09am UTC with a general disclosure, and then again at 9:50am UTC recommending that all approvals be revoked — by that point we had ascertained that “unlimited approvals” were the attack vehicle.
The damage at this point had been suffered: 96.7 ETH taken and transferred to “Tornado Cash”. Funds traded via CivTrade, or deposited on the CivTrade contract, have not been affected; only the tokens sitting in people’s wallet, with an unlimited approval to the CivTrade contract address, were drained.
The attackers piggybacked a contract vulnerability and executed malicious exploits of “approvals” that users had granted the contract, to deplete funds sitting idle in wallets.
It is very important that those unlimited approvals be revoked, e.g. using Etherscan’s tool.
At this point we still did not know the exact dynamic of the incident, but already had two potential hypoteheses as to the source of weakness:
- A v2-specific OpenZeppelin-Forwarder dependency, introduced to implement the “Low Gas” trading option
- A CivTrade-general (v1 and v2) dependency on Uniswap
At this stage, it was not yet possible to determine the ultimate source of vulnerability, and the attack on CivTrade v2 by that point appeared to have completed.
Later in the day — maybe alerted by our own announcement that this was a v2-specific issue (and therefore a v1 version was available) — a second attack was carried out against the v1 contract approvals.
It took the hackers a while to figure out the differences between v1 and v2. The second attack, to v1, was carried out by 3:42pm UTC, with 23ETH and 50k DAI tokens currently sitting in the attacker’s wallet.
While this was a terrible second set of news, in hindsight it eliminated the first hypothesis of attack vector, and therefore ultimately pointed us in the direction of the contract vulnerability.
While it’s of course impossible to get 100% certainty around the dynamic of a hack, we believe to now have a robust hypothesis as to what happened and how. All CivTrade contracts have been paused as a temporary emergency counter-measure.
The source of the hack
The vulnerable source code had been audited by Certik, not just once but twice as I will detail shortly.
The vulnerable function had been correctly identified by De.Fi as “UniswapV3Callback”: when using CivTrade to open a new position, the CivTrade contract calls UniswapV3, which actually calls the CivTrade contract back immediately. Uniswap does not execute token transfers from the user to Uniswap itself, but asks the caller contract (CivTrade) to do that through this “callback function”.
This rather convoluted mechanisms means that, while we are able to determine with 100% accuracy who is calling our contract, we must expose an additional function that only Uniswap pools should be able to call. This function is tasked with transferring tokens from the user wallet directly to Uniswap. Tokens never actually touch the CivTrade smart contract, except for trading commissions paid on closing profitable positions (out of liquidity fees earned).
Now the issue is that any Uniswap pool can technically call this callback function, and legitimately so. The mechanism used to prevent anyone else from calling it is where the vulnerability has emerged: the hackers have built a replica Uniswap pool and circumvented the one line of code which was meant to prevent this rather complex (and so far unheard of) type of attack.
This particular line of code was written in October 2021, back in the day when Civilization developers were a team of anonymous volunteers working for limited compensation based on donations, and without knowing each other. The exact time this line of code was written is less than two weeks after some of those volunteer anonymous people branched out to create a clone of CivTrade. The remaining dev team was rather small.
In order to complete the development of CivTrade v1, we looked into reference contracts published by audited third parties, and identified Visor Finance as an interesting project with $50+ million in TVL (using Bollinger Bands to rebalance v3 liquidity positions) and a Certik audit. At this stage we saw that audit as an authoritative stamp of approval.
Digging through our historical records on GitHub, which keeps a detailed timestamp of every code edit over time, I have been able to verify that the vulnerable line of code came straight from that third party audited source contract.
As you may remember, in December 2021 we received our own Certik audit. This weakness was, yet again, not surfaced during our own audit.
Shortly after our audit (on December 21st 2021, at 02:29pm UTC), Visor Finance was hacked into for a total damage of $8 million. Both our own experience, and that hack, made it rather clear that Certik were not a reliable nor trustworthy auditor, but this was after our own audit had been completed. Furthermore, we tracked the exploited vulnerability of Visor Finance, and confirmed that our contract was not exposed to it.
Since that day I have been routinely reading all the publicly available hack reports (including the excellent Rekt.News and many other sources). To the very best of my knowledge, this is the first instance of this type of hack I have ever heard of (cloning Uniswap v3 and doing a complex triangular trade to drain tokens with unlimited approvals). Our team of devs has over time continued to check new sources of vulnerability as they surfaced, and at all times confirmed that our contracts were “safe from known vectors of attack”.
Sadly, we are now contributing in our own little but meaningful way to the progress of the world’s awareness of DeFi’s sources of vulnerabilities. Hope that something good will come out of this incidents, not just for our own good but also for other people and projects.
Some initial learnings
Development resources
The initial idea of building world-class code based on volunteer anonymous developers, without a stable source of funding but just based on donations, was idealistic.
This limited our initial ability to attract and retain top talent, and narrowed our choice of auditors. Technological decentralisation is great, but writing and auditing good code is hard, therefore achieving technological decentralisation costs (a lot of) hard cash.
The limits of the initial idealistic approach were already clear in early 2022: not just because the bull market was coming to an end, but because we knew that our roadmap ahead was still long and our resource insufficient.
This why we introduced the “community treasury” and raised funds into it; while the code of CivTrade had already been written by that point, the newer code (of CivFarm as released, and now of CivFund being built) has been following a better path thanks to it.
Product focus
In hindsight, we wrote a roadmap with several ambitious products written from scratch (CivTrade, CivFarm, and now CivFund).
It’s fair to say that in the time since its launch, we haven’t seen the widespread adoption of CivTrade we’d anticipated for. In spite of its world-class (innovative and patent-protected) features, we envisaged a tool that would be accessible to everyone, but saw limited accomplishment of its potential.
Maybe there just is not much demand for DeFi limit orders, or our own execution was not sufficiently appealing, or our marketing aggressive enough. But overall the uptake of CivTrade was fairly narrow.
Therefore, arguably the amount of further developer love it received after the launch of its amazing v2 was limited. While we would likely not have realistically been able to spot this world’s-first weakness anyways, I imagine we would have sourced a second audit were we to have more money and users for CivTrade.
Going forward, I believe that a narrower focus might be a better idea.
Choice of auditors
It’s been pretty clear to us for a while that Certik are not a stamp of approval. This only confirms it.
In fact, in hindsight, I believe that choosing Certik as an auditor should be considered a warning sign per se. The sad trophies of many “audited contract hacks” over time should speak for themselves.
For clarity, Certik weren’t even considered for the initial audit of CivFund, as we had come to learn of their track record. Additionally, now we also know that a cheap audit is a very expensive audit indeed.
Metamask approvals
Previous versions of Metamask defaulted to “unlimited approval” for when a contract was asking (legitimately) to spend our tokens.
While this made sense to limit the subsequent gas costs, ultimately it was a poor default behaviour by design, as it exposed a lot of wallets to a trail of sometimes very old approvals becoming insecure over time.
The new default of “only approving the current spend amounts” is way safer and preferrable. I strongly encourage everyone to stick to this policy, and to withdraw any unlimited approvals given at any point.
Open source code
I believe that De.Fi is wrong in claiming: “The contract’s code is not publicly available, making it vulnerable to attacks”.
I can’t see how a hacker’s life would be made easier by not having access to the contract source code.
However, I can see how sharing that code (perhaps with a “no-copy” restrictive license) could have allowed “white-hat”/good hackers to help us strengthen its security over time. While we didn’t want to gift the innovative fruits of our own development to third parties keen on copying us, we clearly wanted to also make hackers’ lives harder.
Maybe open-sourcing the code would not have prevented this hack, or on the other hand it may have even made it happen earlier/worse (who knows) but truth is that our security-by-obscurity defense was not effective.
Probably worth considering a change of this policy in the future.
Next steps
First priority for me should be to define how to permanently close the vulnerability.
Currently the CivTrade contracts are paused, not a long-term viable solution. We could deploy a new patched version of CivTrade, but first must manage the transition away from the old, insecure version. Or we could discontinue all other products at this stage, and exclusively focus on CivFund. Decisions, decisions.
We then must also urgently help our community to revoke unnecessary legacy approvals, both to CivTrade and to other tools, based on the old default of “unlimited approval”. Even if more gas-expensive, approving the current “spend-amount” is way safer. A targeted campaign would be appropriate.
We then must try to recover funds. While this effort may not succeed, keep in mind that there is no “Tornado Cash” that makes money totally untraceable. It’s therefore in the hackers’ best interests to negotiate a settlement, thus avoiding police enforcement and legal action.
Then, we must think of our current and future dev plan and balance its requirements with the expectation of affected users to be compensated.
At this point, the dev team are new compared to 2021. Only myself and Iceman are still “idealistic volunteers” working for free, everyone else in the dev team is remunerated, albeit at a lower level than the market.
We do have some resource left in the community treasury, but we also urgently must raise additional funds, to continue to guarantee stability and long-term viability to the project.
Even if our latest CivFund audit has been world-class and nothing comparable to the joke that was Certik, we may also want to consider additional audit investment before releasing products to market. Costs time and money, but better be safe than sorry.
On the other hand, we must consider how to meet the needs of all our community members past and present who have been affected by the hack, trying to compensate them in the best way without threatening the viability of our product roadmap and CivFund release.
My duty is to the community and I will align to whatever the community will decide, however it’s important to make a careful assessment of costs and times versus current resources even while we continue to try and raise additional funds.
Finally, I invite anyone who feels like reflecting and pushing our thinking, to come forward and share their own thinking and learnings. Whatever doesn’t kill us can only make us stronger, and many successful DeFi and TradFi companies have suffered incidents in the past, in many cases much bigger than this regretful one.
The key is to draw the maximum amount of learning from such sad opportunities, and build even better and stronger from here onwards.
May Decentralisation win. #WeAreCIV
About Civilization
Civilization is the world’s first Decentralized Investment Fund: owned and managed directly by its own users. Token $CIV: the Community Investment Vehicle.
CivTrade offers more trade types, earns extra tokens with each trade, and saves gas versus Uniswap. The ProView upgrade achieved functional equivalence with Binance, yet fully decentralized. CivFarm provides compelling yield while mitigating the Impermanent Loss of providing liquidity.
Also branded the Burning Man of Finance, Civilization is now building the CivFund and CivFiat products, to make the benefits of Decentralized Finance accessible to everyone. Any questions? Please join the global Telegram group.
Civilization | Medium |Discord| Telegram | Twitter | DAO | Github | Reddit
