Open Protocol Interfaces
Getting useful data out of Ethereum is really hard
At Balance, we want to make it easier for people to manage their tokens. Our first product, Balance Manager, will help you see and send the tokens stored in your MetaMask or Ledger wallets. It is really important to us that we have up-to-date token balance data. When we started the company, we relied on a closed source, third party Application Programming Interface (API) for banking data called Plaid. We spent a year building on their platform and a couple of weeks before we launched they lost their data feeds and our product kept getting balances wrong. This experience taught us a hard lesson about building on closed APIs. We never want to depend on a third party for the data that our customers need. We have used Etherscan to build our prototype and launch our private data. The team there are amazing but we really want to have our own way to read token balances from Ethereum directly. This sounds like a relatively simple task but it turns out be incredibly complicated.
Tokens are smart contracts that live on the Ethereum blockchain. When you send a token from one address to another, it is very different from when you send Ether. You are actually telling the smart contract to update its ledger and move some tokens from one address to another.
This means that if you want to get the token balances for a single wallet address, you need to know two things:
- The contract addresses for all tokens.
- All of the transactions within those contracts.
At this point, you need to crunch a serious amount of data on some expensive hardware to build up a picture of what is going on. Only then can you figure out what all of the wallets’ token balances are. Then you can query that new database to power the user interface.
Today, pretty much everyone uses Etherscan to solve this problem. The whole tecosystem is relying on a single closed source provider of data to get token balances and transaction history. That feels in incredibly fragile.
We feel like great open sources tools for this should exist. There are a few projects trying to create open source block explorers but nothing is live.
We also need to start thinking about how to solve this at a protocol level. APIs gave web developers amazing new ways to build interesting apps. Open Protocol Interfaces, or OPIs, should give dapp developers the data they need.
As these protocols mature, we have to figure out ways to make them easier to develop against. Truffle helps developers write smart contracts. Infura provides node infrastructure for people who do not want to manage their own. We need similar tooling for exploring the data within the Ethereum blockchain.
We invited a few people to get together and discuss this issue:
- Lane Rettig from WASM and CryptoNYC
- Jin Chung from Balance
- Qiao from Messari
- Chris Burniske from Placeholder
- Kurt Fenstermacher from Digital Assets Data
- Thomas Rush from Quickblocks
Thomas is one of the only people we have found who is working on this problem. He has been working on a system to index and understand the Ethereum blockchain on a laptop. The discussion was wide ranging and covered a lot of the problems with understanding the state of the Ethereum blockchain.
Notes from the meeting
We had a great discussion for around 45 minutes. Here is the raw audio:
Here is the transcript: https://paper.dropbox.com/doc/Transcript-of-Meetup-YeaNA1IywtCqAN8SYPzux
Opening up the Protocol
If we are going to ensure that Ethereum becomes a common computing system for everyone, we need to find ways to get access to the data that do no rely on centralised third parties. Opening Protocol Interfaces are hugely important to this process.
What now?
If you are interested in working on this or talking more about it, I am throwing everything I learn into this thread on our community page: https://spectrum.chat/balance?thread=90e90547-a45d-40a6-b7a6-a75a1ea0aaab
Moving forward
