iExec Dev Letter #6: The Importance of Security — 01 Aug 2017

Hello and welcome to the IEXEC bi-weekly development letter. Today, we’ll report on the Parity multisig hack and give some perspective on research & development.

The day we lost 120K ETH…

On Jul-19–2017 06:35:06 PM, I received an automatic notification email from etherscan.io that was signaling some activity on the Ethereum multisig wallet, where the IEXEC funds are stored. I rapidly checked the balance and discovered that it was zero: all the funds (>120,000 ETH) were transferred to an unknown address.

Some background: to secure the funds collected during the crowdsale, we kept the ETH in a cold multisig wallet whose keys are securely stored in bank vaults in China, France, and another place in Europe. But the code of the multisig wallet provided by the Parity client had a bug that was easy to exploit by a malicious hacker. Simply explained, the functions that setup the wallet were left unprotected, allowing anyone to reset the ownership of the wallet (which they did) and drain the funds from the wallet.

About one hour after the hack, we started to hear rumors saying that the exploit may have been implemented by a group of white hat hackers. This information was first confirmed by etherscan.io when they displayed a statement on the web page of the address where the funds were moved.

… and got them back!

Something incredible happened! A team of super-heroes who called themselves the White Hat Group had rescued all the funds from all the affected Parity multisig wallets! Not only IEXEC, but also BAT, ICONOMI, cofound.it and many more. The total of the funds recovered was exceeding 200 millions USD. WHG issued a first statement on reddit (A Modified Version of a Common Multisig Had A Vulnerability — The WHG Took Action & Will Return the Funds) and later regularly communicated using the /r/ethereum thread. You can read the full story from their side here.

At IEXEC, we want to congratulate the WHG and express our gratitude for their outstanding action that saved many projects essential to the community. Everyone has to understand that it also takes a lot of courage to take responsibility for this rescue! Today, all the funds have been transferred back to IEXEC and are now safely stored in a newly deployed wallet.

Some experience and lessons learned

• Despite all our efforts to use and ensure the best quality software and services, all this story makes clearer - if possible - on how security is an all day effort. The multisig wallet was one element amongst many when we ran the ICO (see this post Unterstanding the iEx.ec crowdsale smart contract). Although we took a lot of security measures to provide the best protection for the crowdsale participants (including review of the EF multisig code), we think that very simple best practices related to smart contracts can further improve the security for everyone: provide separate github repo for all solidity code, version number and special naming (for instance sar-1.0.0, where sar would stands security audited release), publish all code reviews and security audits in the github repo (i.e avoid using comments) + answers from the developer team (i.e how reviews have been addressed), publish reproducible tests, etc.
• We took several actions before and during the crisis to decrease our exposition to ETH. In case of black hat success, the project would have been able to continue, even though we may probably had to seek additional funds earlier than expected.
• During those past two weeks, the IEXEC team has been working on improving its security infrastructure. We helped reviewing the new multisig contract (see François’s tests of the new multisig) and we coordinated with the WHG to secure the details of the ETH transfer. In the future, IEXEC will happily participate with the community to improve Ethereum security: we already supported the Underhanded Solidity Coding Contest and we’ll contribute to the WHG donation funds.
• Of course our developments have stepped back due to this unexpected event, but still we have made progress on several fronts: a new CI/CD infrastructure is being developed, the web site design is almost done and implementation has started. I won’t elaborate now, but we’re also starting R&D on Intel SGX, which is a very promising technology. More on that later in the next IEXEC development letter that will be published at the end of August. So stay tuned!