Tendermint’s New Application Blockchain Interface — ABCI++

The critical boundary layer between Tendermint and the application, known simply as the ABCI (Application BlockChain Interface), was designed in the very early days when the co-founders decided to split out the state machine from the consensus engine they were working on. The abstraction that really created Tendermint — a generalizable, language agnostic, BFT consensus engine — has been one that has largely stood the test of time and has enabled this rapid growth in sovereign interoperable blockchains.

However, this continual expansion of applications building on top of Tendermint means developers are increasingly pushing the limits of this interface. As the team has come to better understand the usage of the technology developing in this field, we have looked to rethink how such an interface could better serve its users. This accumulation of ideas, initially coined as “ABCI++”, is what is set to land in the upcoming Tendermint releases.

This article provides a wide overview on what these changes are, the motivation behind them, and what they unlock for applications. If you want to dive into the nuts and bolts of the interface itself, the expectations on how it should be used, and pitfalls that application developers may want to be wary of, you can read a more detailed description within the specification.

The original ABCI was designed in a conservative fashion to protect the application as much as possible from inadvertently introducing non-deterministic behavior that would prevent consensus. With the advent of the Cosmos SDK, which provides the resources for a more secure execution environment, application developers need to deal less with the complexity of the consensus engine itself. Rather, developers are now able to build applications on a decentralized network almost as if it were a single computer. This progression allows Tendermint to relax some of the safeguards it had previously put around the state machine and to give more control directly to the application.

Fine-grained transaction control

One challenging engineering problem in this space involves the mempool, a critical component that is interwoven into the fabric of both the application and Tendermint. Tendermint is responsible for block generation and transaction broadcasting, yet applications are concerned with transaction contents, their validity, ordering, and execution. The former ABCI protocol gave the application very little control over which candidate transactions to select out of the mempool for proposal, and this caused headaches in application design. Previous mechanisms (e.g., priority mempool) attempted to patch this but didn’t go far enough.

To address this, ABCI now adds a new PrepareProposal method. The proposer for each height and round will call PrepareProposal, taking the highest prioritized transactions and passing them to the application. The application can then reorder, drop, delay, modify, and even add transactions as they see necessary. Tendermint uses the returned transaction list to formulate the block that is then signed and gossiped around the network as the proposed block.

Block validity

Before signing a block, every node performs a few validity checks. Tendermint has built-in checks for the header and commit, but the array of transactions included in the block are, for the most part, ignored, and could even potentially be invalid. Historically, Tendermint didn’t care what got agreed upon so long as agreement was reached. This is now being changed to give the application more control over block validation. The new ProcessProposal method complements the first method (PrepareProposal) by allowing the application to also check all transactions in a proposed block. Only if the proposed block is valid to both Tendermint and the application will the validator send the prevote for the block. The combination of these two methods means that it is possible to guarantee that no invalid transactions are ever committed. Furthermore, such a setup can give rise to other innovative ideas like scheduled transactions.

Parallel Execution and Concurrency Management

The most critical guarantee that Tendermint requires from applications is a deterministic execution environment. To that end, Tendermint originally serialized calls to the application through a mutex lock, to avert nondeterminism from parallel execution. When a block was committed, each transaction was passed to the application via a separate and synchronous call to DeliverTx. The new interface has consolidated the BeginBlock, DeliverTx, and EndBlock calls into a single FinalizeBlock. This gives the opportunity for applications to execute certain independent transactions in parallel.

Along with this change, Tendermint has removed the mutual exclusion lock, so the application is now entirely responsible for managing concurrent execution. Now, applications will be able to choose whether queries can be made concurrently alongside block execution and transaction validation. Their choices will largely depend on the logic of the state machine itself rather than Tendermint’s assumptions on how it works.

Further to allowing for execution speedup’s, giving the application knowledge about a potential candidate block creates the possibility for optimistic execution. Adopting this approach, applications can concurrently start to execute parts or all of the block while the validators pass votes and reach consensus. If the candidate block is discarded so too must the application discard the candidate state but if the candidate block is finalized, the application can swiftly progress to the next height.

Vote Extensions

This update also includes a new and experimental feature called vote extensions. In a round of consensus, validators and nodes alike exchange votes until agreement on a block is reached. The main feature of these votes are the signatures, produced by a validator’s private key signing over the hash of the block’s header. For some applications it is convenient to have the validator sign over additional data, not required for consensus but useful to the application itself. Vote extensions enable this by adding two new ABCI methods: ExtendVote and VerifyVoteExtension.

When a validator is ready to sign and send its precommit vote, it calls ExtendVote. In response, the application returns an array of bytes (the “vote extension”) for the validator to sign over in addition to the ordinary vote contents. Similarly, for every precommit received by a validator, the node uses the public key to verify the signatures of both the vote and the extension. If this succeeds, it then calls VerifyVoteExtension so that the application can also validate this data. Only when all checks pass does the validator tally the voting power.

Whereas previously in a round only the proposer could include transactions in a block, vote extensions can be seen to allow at least 2/3 (in terms of voting power) of the validators the ability to also append “transactions” or any other arbitrary information as part of the execution of a block. Given the nature of the voting, these transactions aren’t actually processed until the following block. This new functionality unlocks a few very powerful use cases, one of the more obvious being Oracles. Oracle protocols enable off-chain data to be agreed upon and used on-chain. A naive algorithm may allow each validator to post the price of a commodity, say apples. The application then takes the median of the values submitted via the vote extensions as the accepted price of an apple and can use that data with the same byzantine fault tolerance guarantees as the consensus engine itself.

Another exciting use case is threshold decryption. This is similar to regular public private key cryptography but splits the private key into many parts, in our case distributing them to the validators. This public key can be used to encrypt user transactions. These transactions are now opaque to the validators proposing and agreeing on them. They can only be revealed after 2/3 have agreed on the block itself, thus preventing front running and other forms of miner-extracted value (MEV).

Given the applications’ power to reject votes, it is possible that incorrectly coding these methods could affect liveness as Tendermint is unable to receive 2/3 valid precommits to finalize a block. Developers must thus exercise greater caution when using these methods.

Rollout

In conjunction with the Cosmos SDK, Tendermint is moving to a position whereby it can distribute smaller yet more frequent releases. Coupled with adjustments to the QA process, the engineering team plans to deliver high quality, stable features into the hands of users faster. For the ABCI modifications, these will be rolled out across multiple versions. The first, in v0.37, will include PrepareProposal and ProcessProposal. The following release will include FinalizeBlock and Vote Extensions.

Application developers can already begin to build on top of the first part of the new interface by checking out the first v0.37 alpha release here. Cosmos SDK developers will see the new interface incorporated in v0.47.

As always, we are ever appreciative of your engagement and contribution to the project. If you have any issues or feature requests, you may file them on our github repo or talk with some of the engineers on Discord or Github.

About the Author

Callum Waters is an engineer working on Tendermint, at Interchain Gmbh.