How CryptoKitties securely signs and sends Ethereum transactions
And why you might not want to host your own Geth node
While engineers understand the significance of security when developing blockchain applications, subtle details that result in vastly different security implications can often be overlooked. Read on to learn about how CryptoKitties defends against various attack vectors during signing and sending Ethereum transactions.
CryptoKitties’ most special cats have always been the Gen 0s. One was minted every fifteen minutes for the first year of the game by an account we called the Kitty Clock. To this day, Gen 0s remain some of the most sought after on the marketplace — many regularly sell for an ETH or more.
In technical terms, this Kitty Clock was an account with special permissions coded into the CryptoKitties smart contract — we called it the contract COO. One special privilege was the ability to create brand new Gen 0 CryptoKitties with certain genes as well as a sale auction for this asset, achieved by calling the createGen0Auction smart contract function. As this is a function only exposed to the COO (see onlyCOO modifier), it was necessary to use its private key to sign transactions that called this function. This meant that CryptoKitties needed to satisfy 2 main requirements:
- The ability to sign transactions using the private key of the COO (Kitty Clock), and
- The ability to send a signed transaction to the Ethereum network.
In this article, we will go through the considerations CryptoKitties had when designing such a process in practice.
Don’t put all your eggs in one basket
Private keys should always be managed securely. Since possessing a private key means absolute signing power, the attack vectors could have serious implications for the user depending on the type of wallet that is associated. For Hierarchical-Deterministic (HD) wallets such as a Metamask account, a compromised private key would allow the malicious actor to simply transfer and drain all the assets owned by the associated wallets. For smart contract wallets such as Dapper, a compromised private key would equally allow the malicious actor to sign a transaction to transfer assets, but since the multi-signature scheme dictated by the smart contract requires Dapper to co-sign all transactions, fraud prevention mechanisms can be leveraged to flag suspicious user actions. While the CryptoKitties COO doesn't own any of the Gen 0s that are minted (they are immediately put on auction and the auction contract owns it), a compromised COO private key would still allow the malicious actor to significantly disrupt the game by calling createGen0Auction.
During local development, private keys are often loaded from a .env file for simplicity in debugging and testing. However, this can be a rather insecure method in exposing application logic to the private key as an environmental variable, depending on how the application is run. If Docker containers are used, attackers would only need access to the container itself in order to compromise the secret. And we all know how many times this attack vector has been abused (just give it a Google). As a consequence, any form of Dockerized, private-key-as-environment approach wouldn't be viable in production.
One alternative to this is not to expose the private key as an environment variable, but rather a variable embedded in the source code. While this approach may seem more secure, there are existing tools that are capable of decompiling machine code / bytecode back into source code, depending on the programming language. Though difficult, this extent of reverse engineering isn’t impossible and would eventually allow attackers to recover the private key in memory. Moreover, this approach would have hindered the collaboration between CryptoKitties engineers as secrets cannot be committed directly to a Version Control System.
So far, the approaches we’ve described involve managing a secret at the same place as application logic, which highlights the security vulnerability of a compromised secret. Indeed, another attack vector targets the application itself — given the application blindly trusts the private key provided in the runtime environment, a compromised application effectively means a compromised secret as well. Thus, a more secure architecture should separate out the sensitive entities and attempt to isolate as much as possible the environment within which the private key exists as plaintext. In short, don’t put all your eggs in one basket.
This was the rationale that motivated CryptoKitties to use a self-hosted Geth node.
Self-hosted Geth nodes
A convenient functionality of Geth nodes is their interface through which users maintain unlocked accounts on the node. After entering the correct passphrase to decrypt an account’s private key, the full node is able to sign transactions with this private key and propagate the updated chain state to the network — specifically, the full node places signed transactions into its own pool and communicates this state change to other node pools. This was a great opportunity for CryptoKitties to maintain an unlocked COO account on a Geth node — the secret wasn’t exposed via some environment variable, and the security model now had the private key in a completely separate entity than the application logic. For added security, the Geth node was run in an isolated environment and node access was controlled to prevent unknown requests from reaching it. While this model is still vulnerable to a compromised application issuing malicious requests to the Geth node, it is notably more secure than before due to this subtle difference:
- When the private key exists within the application, a compromised application results in a compromised key.
- When the private key exists separately from the application within the isolated environment as mentioned above, a compromised application allows the malicious actor to sign transactions, but the private key remains safe from exfiltration.
In other words, the former is a single exploit, but the latter requires the malicious actor to persist a detectable compromise.
Additionally, this approach presented a sensible tradeoff between engineering complexity and security, as CryptoKitties is now able to sign and send transactions to the blockchain, as well as query for the latest global network state, all through the same self-hosted Geth node.
Unfortunately (or fortunately!), this was rather short-lived due to a number of performance and storage reasons:
- Maintaining a full Geth node quickly consumed more than the 500GB that was allocated to the VM running the instance. To address this, CryptoKitties attempted to maintain 2 additional full nodes to failover whenever one of the VMs would run out of disk space. While this particular problem was solved, the manual overhead involved was far from ideal — failovers occurred often and the process of clearing chain data from Geth and then starting a clean sync from scratch would render the node unusable until after 1 day. This was necessary because Geth nodes lacked automatic database compaction at the time.
- Reading the blockchain via the Geth nodes was unreliable as they were often slow and out-of-sync with the latest global network state.
- Writing to the blockchain was also unreliable as the CryptoKitties applications lacked (few months post-launch) a robust mechanism to resolve transactions that were “stuck” at a particular nonce due to gas-related issues (eg. insufficient gas during network spikes), which blocked all subsequent transactions being sent through the unlocked COO account.
As unlocked accounts made sense to only be on Geth nodes maintained by CryptoKitties, this model wasn’t exactly portable to blockchain infrastructure providers such as Alchemy either. In order to capitalize on Alchemy’s API for reliably sending signed Ethereum transactions to the blockchain, it was necessary to decouple signing transactions from sending them.
Sign here, send there
Today, instead of minting Gen 0s CryptoKitties, the COO maintains an integral role in the game by calling giveBirth promptly to make pregnant kitties give birth. This is achieved by utilizing Google Cloud KMS (key management service) with Cloud HSM (hardware security module) to sign transactions, which prevents any private keys from ever being exposed to any client - data is always passed into the module for signing with the private key. Indeed, CryptoKitties then sends these signed transactions to Alchemy, and the decoupling of signing and sending mechanisms has presented a much more flexible, scalable and less complicated architecture without the headaches of having to manage self-hosted infrastructure.
How do you sign and send your transactions? We’re curious to learn about your process!
A lot of our challenges building CryptoKitties informed our approach to Flow, a developer-friendly blockchain Dapper Labs is helping to create. You can learn more about Flow here!
Acknowledgements
Many thanks to Andrew Burian, Jordan Schalm, Eric Lin, Fabio Kenji, Jordan Castro, Alan Carr and Caty Tedman for reviewing this article, suggesting improvements prior to publication, and Summer-Lee Schoenfeld for the beautiful graphics.
