Tendermint 0.34, Protocol Buffers, and You

Hello, application engineer! Tendermint Core team here. We wanted to write you a note explaining the upcoming serialization format change in Tendermint Core 0.34. This is a state-breaking change, and you’ll have to conduct a network upgrade if you want to use Tendermint Core 0.34 or later.

And you should! 0.34 is going to have some super-useful new features, like state sync, in addition to this big breaking change. It’s also the version of Tendermint which will be required for the Stargate Upgrade.

So here’s what you need to know: This is what we’ve decided to do, and more importantly, what you need to do in order to use new versions of Tendermint Core.

Why is this happening?

Amino is the in-house serialization library built and maintained by Tendermint Inc., also known as All In Bits. Amino was instrumental to early versions of Tendermint Core, and it allowed the team to rapidly build a complex piece of software. Alas, most technology decisions reach their end-of-life, and Amino is no different. As Tendermint Core has matured, Amino has turned into a pain point: it was only available in a limited number of programming languages (with Go the only widely-used choice), and it often required writing custom encoding and decoding logic.

Additionally, one of the biggest performance bottlenecks in Tendermint Core was Amino serialization. By switching to a more performant serialization library, we expect to see big performance improvements!

Finally, Amino is a complex library with a small bus factor. Many of the original contributors to Amino are no longer working full time on Tendermint Core. The cost of maintaining our own serialization library — as well as all the requisite tools for it — is just too much for a small team.

We have the opportunity to adopt a more widely-used serialization library: Protocol Buffers (or “protobuf,” or sometimes simply, “proto”). Protocol Buffers are rapidly becoming the industry standard for wire and/or disk serialization, and are used by other tools and protocols such as gRPC. We are confident that Protocol Buffers will be a solid building block that we will be able to depend on as Tendermint Core grows and matures.

Protocol Buffers also provide a handy code-generation tool for a wide variety of languages, including Go. (More on the nuances of that later.)

What exactly is happening?

Serialization

Tendermint Core uses Amino for both wire and disk serialization, and for these use cases, we’ll be replacing it completely with Protocol Buffers. Previously, we defined our own types in Go, and then Amino would use reflection to serialize them for us. But now we’ll be using types generated by Protocol Buffers throughout the codebase, including in the Go APIs we expose to the Cosmos SDK and other applications. Protocol Buffers will serialize these data structures for us “for free,” without us needing to write additional code.

JSON Encoding

Tendermint Core also uses Amino to generate JSON. This has caused some pain with the Rust implementation, so we’ve been keen to replace Amino here as well. However, the JSON that Amino produces includes distinctive choices about how to encode numbers and missing/nil values. Perhaps most notably, it also has a particular way of handling Go interfaces while encoding to JSON. All of this means that switching to a different JSON encoder right off the bat would deeply break applications. (There’s also the question of which JSON encoder we’d want to use: There are pros and cons associated with every option, including the native Go encoder, the json-pb encoder, and the JSON schema encoder.)

For 0.34, we’ve decided to write a small wrapper which will produce “Amino-flavored” JSON from our generated types. This will allow us to remove Amino as a JSON encoding dependency, without breaking the interface for applications. It also means that we will be maintaining our own serialization code; however, it is small (~500 lines), limited in scope, and relatively easy to understand.

What do you need to do to ensure compatibility with Tendermint 0.34+?

The Protocol Buffers migration itself doesn’t necessitate any code changes from client applications; however, there are a handful of code changes necessary for other features in 0.34. Fortunately, these changes are fairly minor and limited to the Application-Blockchain Interface (ABCI). The ABCI changes include a number of new methods, all of which run on a new ABCI connection and which enable the new state sync feature:

• ListSnapshots
• LoadSnapshotChunk
• OfferSnapshot
• ApplySnapshotChunk

At a minimum, you can simply implement these functions as no-ops in order to maintain the interface. However, if you want your application to implement state sync, you’ll have to implement these methods within your application. We’ll be releasing a “user guide” for state sync soon, with more information on how to best implement these methods.

There is also one change to an existing ABCI message:

• The “Proof” field of the “ResponseQuery” message is now called “ProofOps.”

These changes comprise ABCI version 0.17.0.

In addition to making changes to ABCI, this upgrade does change the way that state is encoded, and so this is a “state-breaking” change. Since encoded state is hashed, all the hashes for replayed blocks will change as well. For more details on which pieces of state are getting encoded differently, please see upgrading.md.

Networks upgrading to Tendermint 0.34 will need to conduct a network upgrade in order to update. This coordination will be done by the networks themselves; for example, if you are operating a node on the Cosmos Hub, this upgrade will likely be orchestrated through the next Hub upgrade.

One caveat: The IBC implementation inside the Cosmos SDK itself will need to make more expansive changes to the IBC Tendermint client in order to work with Tendermint 0.34. Because that implementation uses types that are generated by Protocol Buffers, and those types live inside Tendermint, some coordination between the Cosmos SDK and Tendermint was required.

Risks

Although there are lots of good reasons to make this change, it’s still a big change. Many lines of code have been added, deleted, generated and regenerated. And any time there’s this much churn, there’s an increased risk of introducing bugs or causing other accidents. To defend against this, we’re taking a number of precautions, and will be rigorous about testing and verifying this release in order to mitigate the risks.

Mitigation: Integration and Unit Testing

As a baseline risk mitigation strategy, we’re emphasizing unit and integration testing across the Tendermint Core codebase. The team has worked hard to improve unit tests, increase coverage, and reduce non-determinism in tests as part of this release. Additionally, we’ve doubled down on increasing our “unhappy path” coverage.

Mitigation: Testnets

We’ll be running a number of testnets, internally and publicly, in order to test this release, both with and without the Cosmos SDK; and with and without previous Hub data. The exact details are still forthcoming, but the public testnets will be a cross-entity collaboration including teams working on the Cosmos SDK, Tendermint KMS, and more.

If you are a validator and are interested in participating in these testnets, please let us know! The more participants we have, the more confidence we will have in this exciting new release. The Tendermint Core team is available in the Cosmos Discord and we’d love to chat.

Mitigation: Public Release Candidates

Ahead of the official 0.34 release, we’ll be working with a number of partner projects, who will run public release candidates with their own software and projects. Successfully integrating the release candidates with ecosystem projects will help us build confidence in the official release.

If you’re interested in being one of our test partners, please get in touch! Again, getting a broad set of partners involved will help us release software that is stable and secure.

Future work

Protobuf Tooling Change

Although Protocol Buffers themselves are highly standardized, there exists a wide range of tooling to generate source code from .proto files. At the moment, both the SDK and Tendermint Core use a combination of compiler “plugins” and “extensions” which modify the way that code gets generated. However, due to divergent requirements, Tendermint Core and the SDK would like to use slightly different Protocol Buffer tooling sets. See the Proto Compilers, Extensions Future Work document for more details on the background and requirements.

In theory, it should be OK for different applications to use different proto tooling — as long as the generated types are only used internally. Unfortunately, Tendermint currently exposes generated types in its public Go APIs, which means that we have to use exactly the same generated types as our applications, which means that we must use the same proto compilation tools.

So in order to maintain compatibility with the SDK, we are using the same tools — for now. We intend to ultimately switch to a different set of compilation tools, although this will require us to rewrite our public Go API to use “domain” types rather than generated types.

JSON Encoding, Part II

As described above, we’re currently continuing to encode JSON in an “Amino-flavored” way. We intend to change this at some point in the future, to something more standard. This decision will be influenced by other open questions, such as whether we want to adopt gRPC. As it will not directly impact Tendermint 0.34, we haven’t made this decision yet.

TL;DR

That was a lot of words about a serialization library swap, so let’s briefly recap:

• We’re switching from Amino to Protocol Buffers in order to improve stability, maintainability, performance, and interoperability;
• Tendermint 0.34 also comes with lots of other goodies and important new features;
• In order to use Tendermint 0.34, you’ll have to make a few ABCI updates and conduct a network upgrade;
• This is a big release and it comes with some risks, but we’re working aggressively to mitigate those risks;
• If you want to help us test this release, either by joining a testnet or by integrating a release candidate, please get in touch with us by joining our Discord.

Thank you to Marko Baricevic and Erik Grinaker for their contributions to this post.