A postmortem on the BlockParty deployment account security issue.

It has been almost a month after I found out the incident. I still haven’t got to the bottom of it but would like to share what I know so that you can avoid making same mistake as I did.

How I used to manage my deployment environment.

When I first got into Ethereum back in 2016 March, the only way to interact with the blockchain was (apart from online service such as myetherwallet) to download Ethereum database (aka geth node) and sync fully.

I remember the early days of our Ethereum codeup where majority of time was spent on how to sync the node.

Metamask chrome extension based mobile wallet arrived around August. This lowered the barriers to entry massively but you still need fully synced node if you actually want to deploy smart contract.

This has improved massively when geth --light mode arrived around the end of 2016.

geth removedb
geth --light

With the light mode, the geth node downloads only required header in real time, hence took only a few minutes or so to be able to interact with blockchain. It was also quite revalating that removing the full node saved me massive 50GB of hard disk.

Then in 2017 August , I was preparing for my prime time to pilot BlockParty for London Ethereum meetup.

At this time however, the geth --light mode wasn't working well. I kept getting errors and wasn't able to deploy at all. I was so frustrated and decided to ditch deploying to local node.

Instead, I decided to deploy directly to infura.io , remote geth node many services such as Metamask is relying on. To deploy remotely, I used truffle-hdwallet-provider to generate private key and sign it locally

This was my killer commit.

Can you see my killer comment?

// todo: Think about more secure way

(Lessons learnt: Whatever you put on todo comment will come back and haunt you).

By providing the seed as a command line argument, the migration script will sign the transaction locally and send the signed transaction directly to infura without exposing the seed.

truffle deploy network mainnet --mnemonic $SECRET

When did it happen?

As mentioned in the original security alert, the BlockParty deployment account was compromised and ETH 1.2 was transferred into an unknown account.

The transfer was made at 07:16:36 PM UTC on 6th December. Incidentally I was at the Heathrow airport to take a flight for my long Christmas vacation, which naturally suspects that I was hacked while at the airport. I have VPN but cannot remember whether I had it on or off. Even if it was off, there was less likelihood that the mnemonic was intercepted on network as I never sent it over network or sent any transactions on that day.

The only possibilities I can think of are either I generated seed from malicious library (though I remember that I generated locally offline) or someone tapped into my file system at the airport.

I only noticed this on 13th January (almost a month after) when I was about to deploy a new contract for the upcoming London Ethereum meetup.

After the transaction was sent to a different account, the hacker sent to another account a day after, then split the transactions into smaller pieces.

What did I do after the incident

You may think there’s not much the hacker can do after all the Ether was gone, but that was not true.

Some of the participants are slow at withdrawing deposits and when they don’t withdraw within one week of the event, the event organiser (the person who deployed the contract = me) can call clear() function to take the deposits which I actually did a week before the hack.

I was going to either return or donate to charity if any I get any response but I didn’t so they were left on the account which unfortunately got stolen.

However, I had one more participants who haven’t withdrawn and I haven’t cleared either which I had to rescue before the hacker find out.

Because there’s no more Ether left on the owner account, I first had to send Ether back to the compromise account, called clear function, then send all the Ether to uncompromised account (I met Richard Brady at the January Ethereum meetup and he was okay for me to keep so it’s still mine :-) ).

What else can it go wrong? Well the famous selfdestruct

You cannot get anything by calling the function but that will wipe out the state hence I can no longer see who have participated which will be very annoying. To mitigate the risk, I list up all the meetups deployed by the compromised accounts and called transferOwnership so that the hacker can no longer call the function.

MyEtherWallet was quite handing doing such repetitive tasks because I did not have user interface for all the non usual admin tasks. It’s a lot easier than doing via command line.

After doing the all the work, I reinstalled my laptop OS to be the clean slate.

And also to be in the safe side, I announced that I am not using BlockParty for both Ethereum London and Codeup in January.

Ironically for CodeUp, we had exactly 50 % attendance rate :-(

What am I going to do going forward.

For the short time, I will probably revert from deploying remotely to locally using geth --light . This way the account file is at least password protected. And most importantly I should not leave significant amount on deployment account anyway.

For more permanent solutions, I am thinking about the following.

1. Move ownership to Gnosis multisig contract after each deployment so that admin account is more secure. Using multisig for every admin operation could be quite hard to use, so I should probably separate tasks into admin role which is only used for emergency purpose and operator role which is used for more normal operations.
2. Pass encrypted mnemonic and decrypt from the deployment script itself . Even better if I can use HD wallet but I don’t know if there are any solutions available. Aragon is offering $50k bounty for OS-level signing provider so this may be a good project to take on if you are expert on this area.

Unlike smart contract security, operation security side is not widely shared.

Consensys Smart Contract best practices github page has an issue page with bounty to encourage users share their best practices so I encourage other people to share their operational security tips.

(Special thanks to Maurelian for reviewing this article)