What are you working on? Checking in with Nathan Hutton
Hi! I’m Nathan. I’m a senior software engineer and I have been working on the Fetch.AI ledger for the last 18 months.
As you may already know, the ledger confirms transactions between two or more parties in a decentralised manner. This involves some considerable design challenges. Two of the aspects I have been working to resolve are the networking capability and storage.
Networking is an interesting problem because the Fetch.AI framework allows clients or peers (users and miners) to make RPC invocations to your node at any time. As a result, it is necessary to write the code very carefully to avoid race conditions or memory errors. Other than just making and receiving RPC from node to node, there are also higher level networking requirements. Miners need to self-organise into a structure that avoids bottlenecks and minimises latency when transmitting information. Two examples of such a transfer are block and transaction synchronisation.
These issues are solved in different ways: when a miner creates a block, to minimise the time until it is seen by all miners, a gossip protocol is used to propagate it through the network. Transaction synchronisation is markedly different. Here, each miner maintains a small ‘transaction sync pool’ of recently seen transactions. Peers of this miner can then make requests of their own to ‘pull’ transactions they haven’t seen from this pool. This allows miners, on both sides of the call, control of what is potentially a fairly expensive operation.
With regards to the storage, this again presents its own challenges. Fetch.AI’s novel solution to achieve high transaction throughput at scale is to shard the ‘world state’ into a number of resource lanes, allowing miners to increase or decrease the number of shards at runtime to respond to changing system demands and economic incentives.
For those who are unfamiliar with this term, the world state in the cryptocurrency space refers to the conceptual ledger that all nodes agree on. Executing a valid transaction will necessarily alter this world state — in the case of a token transfer this would decrement the tokens at one address in the world state, and increment it at another address in the world state.
The consequence of this sharding scheme is that in addition to the usual challenges and stipulations one might associate with database design (mainly access times for items as the database grows, but also importantly on-disk size), you have to support splitting or merging the database. Moreover, the database must support reverting back to a specific block, which may have a different lane count. By reverting, I mean reverting the world state (all shards) to a specific commit. Therefore, if the miner switches from one fork to another, the database will revert to the state immediately before the fork. This process will enable the miner to execute the blocks from this branching point up to the head of the new fork.
If you are interested in learning more about Fetch.AI’s transaction lane concept, I recommend checking out our yellow paper and keeping an eye out for my next blog post.
