EOSIO Development Update (Multi Index API, Context Free Actions, and new Interpreter)
The past couple of weeks at block.one has quite productive with some very significant updates that impact developers, scalability, and stability.
Generic Multi Index Database API
Developing smart contracts involves defining a database schema to track, store, and find data. It is quite common for developers to need the same data sorted/indexed by multiple fields and to maintain consistency among all the indices.
Those familiar with Steem and BitShares development know the power of the boost::multi_index_container. These containers provide an API that is powerful, flexible, and safe. I have long wanted to have a similarly flexible and powerful system for our WebAssembly contracts. I am excited to report that we have made changes to how smart contracts access the database so that we could deliver a very similar API to developers.
A large part of making this happen is the creation of a new database API that is based upon iterators. Iterators give WebAssembly a handle by which it can quickly find and iterate over database objects. This new API gives a dramatic performance increase by changing the complexity of finding the next or previous item in a database from O(log(n)) to O(1).
Context Free Actions
A context free action involves computations that depend only on transaction data. A primary example of such a computation is signature validation. Given only the transaction data and a signature we can compute the public key that signed the transaction. This computation is one of the most expensive individual computations a blockchain must perform. Because this computation is context free (doesn’t depend upon blockchain state), it can be performed in parallel.
Context free actions are like other user actions, except they lack access to the blockchain state to perform validation. This enables EOSIO to process all context free actions in parallel just like signature verification. More importantly, this enables generalized signature verification.
To understand the importance of generalized signature verification, consider the use case of inter-blockchain communication. In this use case a user must provide a merkle proof that involves as many as 128 sha256 operations and/or 14 signature verifications. All of this data and computation is done simply to prove that a particular action occurred in a particular block on a foreign chain and/or to prove that a particular block was validated by +2/3 producers. The business logic only needs to validate that the block ID was valid and that the action was in that block. The calculations to prove that the action was in the block are not needed by the business logic any more than the calculations to derive a public key from a signature.
With support for Context Free Actions, most of the scalability techniques proposed by Ethereum (Sharding, Raiden, Plasma, State Channels, etc) become much more efficient, parallelizable, and practical. This will work with other aspects of EOSIO to enable efficient inter-blockchain communication and unlimited scalability.
The concept of SegWit, or Segregated Witness, is that transaction signatures are not really relevant after a transaction is immutably included in the blockchain. Once it is immutable the signature data can be pruned and everyone else can still derive the current state. Since signatures represent a large percentage of most transactions this represents a significant savings in disk usage and syncing time.
This same concept can apply to merkle proofs used for inter-blockchain communication. Once a proof is accepted and irreversibly logged into the blockchain, the 2KB of sha256 hashes used by the proof are no longer necessary to derive the proper blockchain state. In the case of inter-blockchain communication this savings is 32x greater than the savings on normal signatures.
Another example of SegWit would be for Steem blog posts. Under this model a post would contain only the sha256(blog content) and the blog content would be in the segregated witness data. The block producer would verify that the content exists and has the given hash, but the blog content would not need to be stored in order to recover the current state from the blockchain log. This enables proof that the content was once known without having to store said content forever.
There is come controversy over how SegWit is implemented in Bitcoin, but most of the arguments against SegWit that are relevant to Bitcoin are not relevant in the context of Delegate Proof of Stake. For example, with DPOS it is possible to:
- Require Block Producers to keep full records
- Inter-blockchain Merkle Proofs can be regenerated from public data
- Block Producers, being public, are liable for providing evidence that their blocks were valid and that proper signature authorizations exist for transactions.
- With BFT-DPOS it would require 100% collusion of the block producers to cause a signature to be lost and we have 21 independent validations.
EOSIO now supports TWO different execution engines for Web Assembly. The original JIT implementation provides fast execution after an initial “slow” compilation step, and the new Binaryen interpreter provides a standards conformant reference implementation without any slow start up steps.
Block producers will be able to use the interpreter while the compiler optimizes the x86. Furthermore, in the event of a dispute over which output is the “correct output” we will be able to use the standard-defining Binaryen interpreter leaving the JIT implementation as an optimization that must produce the same output.
In our performance tests the Binaryen interpreter is about 20% the speed of the JIT implementation; however, for most transactions the WASM execution time is a small percentage of the entire transaction processing time. Database access and other blockchain calculations still represent the majority of the processing time. In our estimate, the interpreter only impacts real-world performance by about 5% for simple contracts (like currencies).
We will continue to focus on correctness, stability, and security over raw performance for our initial release of EOSIO in June. That said, scalability and optimization are constantly on our mind and everything in EOSIO is being designed to scale.
New Team Member
We would like to welcome Arhag to the block.one team. Arhag has been a long-time contributor to BitShares and Steem and is currently working with us on the database API.
Development is progressing at a rapid rate and we are very excited about these latest developments.