No Sleep till Multi-Sig

How Bitstamp and BitGo launched a multi-sig hot wallet in less than 48 hours.

On Monday, Jan 5, 2015, Bitstamp alerted customers that they were suspending operations due to a theft of funds from their hot wallet. I’ve known the Bitstamp founders since October 2013, and we at BitGo have been talking with them for some time about ways to work together. My heart sank on hearing the news, both because another hack is the last thing Bitcoin needs, and because I knew what Nejc, Damijan and the whole team must be going through right now.

Not knowing any further details at this point, we reached out to see how we could help. The BitGo engineering team had our first phone call with Bitstamp on Wednesday morning. Since the hack, the Bitstamp team, as well as their major investors at Pantera Capital, had already been working furiously around the clock in San Francisco and Slovenia to put in place their pre-existing business continuity plan. They were in the process of setting up a clean replica of their entire infrastructure on AWS, while the existing machines were being preserved for forensic investigation.

Our goal was to help them secure their operational (hot) wallet using BitGo’s multi-sig platform, but under the circumstances, we knew it was a long shot that the Bitstamp team, already slammed with work, would want to pursue a new API integration on such a tight timetable. And yet, by Friday morning, Bitstamp reopened for business, and began processing deposits and withdrawals out of a brand-new multi-sig wallet secured with BitGo. The integration required a great deal of work from both teams, but resulted in changing only a single line of code in their core software. So how did we do it?

During our phone call, the tech team at Bitstamp let us know their existing hot wallet used bitcoind. They asked if we had an API that was bitcoind-compatible. Unfortunately, we did not. Moreover, we did not yet even have an existing SDK in the language Bitstamp uses on their servers. But it was this question that essentially told us what we ought to do: build a translation layer that looks like bitcoind, but uses the BitGo API as its back-end. Our platform was ready and scalable — they just needed an easier way to talk to it.

A brief word of explanation of BitGo and its platform. BitGo has been working on multi-sig since the beginning of 2013. BitGo, at its core, is a security platform for Bitcoin. Our web-based enterprise multi-user wallet is built on our own platform, and we were nearing the wide-availability launch of our platform API when the Bitstamp hack occurred.

BitGo’s API uses 2-of-3 multi-sig HD wallets. This means that a wallet consists of 3 HD (BIP32) root keys: the operational key, the BitGo key, and the backup key. These keys are generated on 3 separate machines, and 2 are never present on the same device. Such a wallet can generate an unlimited number of new addresses in order to maintain privacy. To spend funds, the user must first construct a transaction and attach his signature using the operational key, all on the client. Then, he submits the transaction to the BitGo server API for co-signing.

Before deciding to co-sign, BitGo applies security policy checks on the wallet, such as enforcing velocity limits, address target whitelists, IP restrictions, and so on. If the transaction passes the security checks, BitGo issues the second signature on the transaction using its key, and submits it to the network. If not, then BitGo may either reject the transaction, or hold it for additional approval from another administrator on the wallet. The final (backup) key does not come into play during normal operation. It is a cold-storage key which is for disaster recovery, and also allows the customer to retain ultimate custody of the bitcoin.

Back to the story… We knew we couldn’t possibly fully replicate all of bitcoind’s functionality in such a short time. But the Bitstamp eng team was very quickly able to send to us the pieces of their code which interacted with bitcoind. With that code, we now had a map. We now knew we only needed to implement a limited set of API calls. And we realized that for the non-wallet-specific blockchain calls, we could just proxy to Bitstamp’s existing bitcoind instance. If we succeeded, we would be able to drop in a replacement, and Bitstamp’s existing systems wouldn’t know the difference.

Over the next 24 hours, while the Bitstamp team kept forging ahead with their infrastructure work and re-launch plans, the BitGo engineering team built our multi-sig bitcoind replacement, which we named BitGoD. We built BitGoD in NodeJS, using the json-rpc2 module, and the BitGoJS SDK. The proxied calls were very straightforward to get up and running, and a number of the wallet calls, such as getnewaddress, getbalance and listunspent had 1-for-1 equivalents already in BitGo’s SDK. The core functionality of sendtoaddress, despite being a multi-sig transaction, involving client-side signing as well as server-side signing, was able to be replicated in 37 lines of code using our existing SDK. A slightly simplified version of that code is below:

handleSendToAddress = function(address, btcAmount, comment) {
  this.ensureWallet();
  var self = this;
  var satoshis = Math.floor(Number(btcAmount) * 1e8);
  
  return this.getWallet()
  .then(function(wallet) {
    var recipients = {};
    recipients[address] = satoshis;
    return self.wallet.createTransaction({
      recipients: recipients,
      keychain: self.getKeychain()
    });
  }).then(function(tx) {
    return self.wallet.sendTransaction({
      tx: tx.tx,
      message: comment
    });
 }).then(function(result) {
   return result.hash;
 }).catch(function(err) {
  // handle errors...
 });
}

The most tricky call to replicate was listtransactions. BitGo has an API to return the transactions on a wallet, but it returns 1 record per transaction. On the other hand, bitcoind returns one record per transaction output, excluding change addresses. This wasn’t fundamentally difficult — it amounts to expanding each of our tx records into multiple records in BitGoD. But we had to be very careful about ensuring that we were producing the same results as bitcoind for all types of transactions. Additionally, we had to add some key bits of info to the records our API was returning in order to fully provide the needed information.

Finally, we added a few additional JSON-RPC commands to BitGoD to allow passing in the required BitGo authorization token and the operational key. We shipped the code for BitGoD to the Bitstamp team on Thursday afternoon and began testing the integration with their team. There were a few small issues with the new code that had to be ironed out. But within a few hours, all the site functionality was working. New deposit addresses could be generated, deposits were credited, and withdrawals could be processed.

The next morning, the Bitstamp team made the final call to go live with the new implementation, and they opened their doors again to new customers. Almost immediately, transactions began flowing through the new hot wallet, and every customer that deposited funds saw brand-new multi-sig addresses. Our hats are off to Nejc, Damijan and the entire Bitstamp team and their investors for successfully managing their way through this crisis. I believe this unfortunate event will ultimately only make them stronger as a company.

We are happy to have played a small part in helping Bitstamp re-launch their exchange. And we’re looking forward to continuing to work with them to determine how to use multi-sig most effectively for their operational security, now that the immediate crisis has passed.

Note: We will be open-sourcing BitGoD in the very near future after implementing the remaining API calls. We think it can be a great easy first step to multi-sig for businesses that are already using bitcoind in their stack.

Update (Feb 20 2015): We have released BitGoD — check it out!

Ben Davenport is co-founder and Chief Product Officer at BitGo.