Theta Mainnet Source Code and New Whitepaper Release
Today we are pleased to announce the release of a major update of the Theta Ledger source code, and the new whitepaper.
Theta Ledger source code: https://github.com/thetatoken/theta-protocol-ledger
Updated whitepaper link: https://s3.us-east-2.amazonaws.com/assets.thetatoken.org/Theta-white-paper-latest.pdf
Multi-level BFT technical report: https://github.com/thetatoken/theta-protocol-ledger/blob/master/docs/multi-level-bft-tech-report.pdf
We first open sourced the initial version of the Theta Ledger along with the launch of the testnet back in June 2018. It was built on top of the Tendermint consensus engine. Though being pretty reliable, our live testing with hundreds of thousands of users identified several potential issues with the Tendermint engine for video streaming applications. Firstly, token transactions for video streaming are dominated by frequent micropayments. To achieve the finest granularity of a token reward — one micropayment per video segment — even a live stream with a moderate ten thousand concurrent viewers could generate a couple thousand microtransactions per second. Such a high throughput is only achievable via off-chain transactions, which the Tendermint core engine is not capable of. Secondly, high transaction throughput also poses challenges on the storage system of the ledger. Tendermint is designed to run on a single computer, and has no mechanism for unused data removal. Finally, we want a high level of decentralization for transaction security considerations. Tendermint only allows a small number of validators, leading to potential security and centralization concerns.
This led us to redesign and re-implement the Theta Ledger from scratch. In this blog post, we describe our new implementation at a high level. For full technical details, please refer to our updated whitepaper, and the multi-level BFT technical report.
Scalable Consensus Mechanism
One of the core innovations with our new design is a multi-level BFT consensus mechanism which allows 1000+ of nodes to participate in the consensus process, and can still support very high on-chain transaction throughput (1000+ TPS). With this mechanism, a small set of nodes, which forms the validator committee, is elected to produce a chain of blocks as fast as possible using a PBFT algorithm similar to the Ethereum Casper FFG and the Hot-Stuff BFT consensus protocol. With a sufficient number of validators (e.g. 10 to 20), the validator committee can produce blocks at a fast rate, while maintaining a sufficiently high level of difficulty to deter adversaries from compromising security. Hence, with high probability, consensus participants called guardians can expect that the validator committee will produce a chain of blocks without forks. Then, the thousands of guardians, can finalize the chain generated by the validator committee.
Here “finalizing the chain” means to convince each honest guardian that indeed more than 2/3 of all guardians have the same chain of blocks. Since there are many more guardians than validators, it could take a longer time for them to reach consensus. In order for the guardians to finalize the chain of blocks at the same speed that the validator committee produces new blocks, the guardian nodes process blocks at a much coarser grain. To be more specific, they only need to vote for the checkpoint blocks, i.e. blocks whose heights are a multiple of some integer T (e.g. T = 100). It is worth pointing out that the guardians only need to agree on the hash of the checkpoint blocks, thanks to the immutability of the blockchain data structure. Once they confirm that more than 2/3 of all guardians have the same hash for a checkpoint block, with overwhelming probability, they can be ensured that all the honest guardians have an identical copy of the blockchain from the genesis block up to that checkpoint.
For the guardians to reach agreement on the checkpoint block hash, a naive all-to-all broadcasting of the hash could work. However, it yields quadratic communication overhead, and so cannot scale to large numbers of guardians. Instead, we propose an aggregated signature gossip algorithm which could reduce the number of network messages from O(n2) to O(nlogn) and works robustly with the presence of byzantine guardian nodes. The core idea is to have each guardian keeps combining signatures from all its neighbors, and then gossip this aggregated signature out, along with a compact bitmap which encodes the list of signers. This way the signature share of each node can reach other nodes at exponential speed thanks to the gossip protocol. Within O(logn) iterations, with high probability, all the honest observer nodes should have received an aggregated signature with shares from all honest nodes. The non-interactive BLS signature aggregation scheme is used in our implementation. We have proven the correctness of the aggregated signature gossip protocol, and analyzed its efficiency. In our analysis, with 1000+ guardians, finalizing one checkpoint just requires a guardian to send/receive around 100 messages to/from its neighbors, each about a couple kilobytes long. This renders the aggregated signature gossip protocol rather practical to implement and can easily scale to support thousands of guardian nodes.
The name “multi-level BFT consensus mechanism” reflects the fact that the validator/guardian division provides multiple levels of security guarantee. The validator committee provides the first level of protection — with 10 to 20 validators, the committee can come to consensus quickly. Yet it is resistant enough to attacks — in fact, it already provides similar level of security compared to the DPoS mechanism if each validator nodes is run by an independent entity. Thus, a transaction can already be considered safe when it has been included in a block settled by the validator committee, especially for low stake transactions such as micropayments. The guardian pool forms the second line of defense. With thousands of nodes, it is substantially more difficult for attackers to compromise, and thus provides a much higher level of security. In the unlikely event that the validator committee is fully controlled by attackers, the guardians can re-elect the validators, and the blockchain can restart advancing from the most recent block finalized by the guardians. A transaction is considered irreversible when it is included in a finalized block. We believe this mechanism achieves a good balance among transaction throughput, consistency, and level of decentralization, the three corners of the so-called “impossible triangle”.
Full-Fledged Support for Turing-Complete Smart Contracts
This Theta Ledger source release also comes with a smart contract runtime environment compatible with the Ethereum Virtual Machine. It provides full-fledged support for Turing-Complete smart contracts. Solidity-based Ethereum smart contracts can be ported to the Theta Ledger with little effort. Solidity has grown a large developer community and the prospect of allowing that proven talent pool to also contribute to Theta without reinventing the wheel was a prime consideration in enabling compatibility with the Ethereum Virtual Machine.
Smart contracts enable rich user experiences and new attribution models for video platform DApps built on the Theta Ledger. For example, video platforms can write smart contracts for loyalty programs to engage users. Based on users’ activity, or the volume of video segments / data they have relayed, platform DApps may promote users to a higher tier, which unlocks certain privileges or exclusive capabilities. As another example, video platforms can issue virtual items backed by the ledger blockchain (e.g. a virtual rose) for gifting to their favorite content creators. To expand on such a concept, built on the “non-fungible token” standard, the virtual items could be rare or entirely unique, such that they are essentially “crypto collectibles”, which can be kept as trophies or traded for other sought after collectibles, all without additional permissions from 3rd parties.
Moreover, video platforms are able to write smart contracts that enable more fluid payment-consumption models, such as pay-as-you-go or per-use models. Instead of traditional annual or monthly subscriptions, user consumption can be priced at a bite-sized granularity, such that users only need to pay for what they use. This is a feasible way to allow low-priced, short-form content to be transacted in an economically sensible way, that accrues benefits to both the video platform and user. Theta Ledger’s properties of tracking micropayments and video segments enables such smart contracts to be executed.
Smart contracts can also be designed to the benefit of content creators (e.g. user-generated content producers, larger production studios) as a way to fairly and transparently distribute royalties. The traditional royalty settlement processes, with all their complexities and obscurities, can be accommodated with clear smart contract terms that are mutually agreed upon by creators and distributors — and made available to users that consume the content.
Leveraging smart contracts on the Theta Ledger to enable fully digitized item ownership, innovative payment-consumption models, and transparent royalty distributions provide an additional layer of social and economic interactivity that supplements the core functionality of video/content delivery.
Please refer to this link for a simple example which demonstrates how to deploy and execute a Solidity smart contract on the Theta Ledger.
Native Support for Off-Chain Payments
Another innovation is the built-in support for off-chain micropayments, which has been introduced in our previous technical blogposts. The off-chain micropayment is supported through a “resource oriented micropayment pool” that is purpose-built for video streaming. It allows a user to create an off-chain micropayment pool that any other user can withdraw from using off-chain transactions, and is double-spend resistant. It is much more flexible compared to off-chain payment channels. In particular, for the video streaming use case, it allows a viewer to pay for video content pulled from multiple caching nodes without on-chain transactions. By replacing on-chain transactions with off-chain payments, the built-in “resource oriented micropayment pool” significantly improves the performance of the blockchain.
Microservice Storage Architecture With History Pruning
The off-chain payment pool not just helps improve the transaction throughput, but also dramatically reduces the volume of on-chain transactions. However, even with such a reduction, the block and state data could still grow quickly. Our third innovation is a storage system that can effectively address this problem, and can adapt to different types of machines, be it a powerful server cluster running in data centers, or a commodity desktop PC. Two techniques are employed to achieve these goals. The first technique is the adoption of the microservice storage architecture commonly seen for modern web service backends, where different modules of the ledger can be configured to run on different machines. In particular, the transaction processing engine and the block/state storage module can be separated. For the storage module, the Theta Ledger has implemented the interfaces for multiple databases, ranging from single machine LevelDB to cloud based NoSQL database such as MongoDB, which can store virtually unlimited amount of data. Thus the ledger can run on one single computer, and can also be configured to run on server clusters. The second technique is state history pruning with reference counting. We have implemented a Merkle-Patricia trie that records the state history for each block. By attaching an additional attribute “reference count” to each trie node stored in the database, the ledger can keep track of which nodes are referenced only by older blocks and can thus be deleted from the database. At a minimum, only the nodes referenced by the most recent finalized block need to be kept. Block history pruning can be done using a similar technique. With both state and block history pruning, the storage requirement can be reduced to a minimum. Combining these two techniques, a validator/guardian has the option to use a local or remote storage backend, and has the flexibility of keeping the full history or just the most recent states and blocks. The Theta Ledger is thus adaptable to different computing environments.
Summary
In summary, the new version of the Theta Ledger is a decentralized ledger system designed for the video streaming industry. It employs a novel multi-level BFT consensus engine, which supports high transaction throughput, fast block confirmation, and yet allows mass participation in the consensus process. Turing-Complete smart contracts are supported through an EVM compatible virtual machine, which enables rich user experiences for DApps built on top of the Theta Ledger. Off-chain payment mechanism is built directly into the ledger through the resource-oriented micropayment pool, which is designed specifically to achieve the “pay-per-byte” granularity for streaming use cases. Moreover, the ledger storage system leverages the microservice architecture and reference counting based history pruning techniques, and is thus able to adapt to different computing environments, ranging from high-end data center server clusters to commodity PCs and laptops. We encourage the readers to read our whitepaper, technical report, and source code for more details. As always, any feedback is appreciated!
