Koinos: Now with Google’s Protobuf!
Koinos is being developed to help everyone take advantage of blockchain technology which is still far too inaccessible. That’s why we’re building Koinos with a focus on maximizing accessibility. For end-users that means enabling them to use decentralized applications without having to spend any money (free-to-use). For developers that means allowing them to use the programming languages they already know instead of forcing them to learn something new (universal language support).
Of course, delivering a blockchain with these capabilities requires innovative solutions, some of which we’ve built ourselves, but whenever possible we lean on the state-of-the-art technologies developed by other amazing open source teams and maintained by their independent communities. In fact, the unparalleled upgradeability of Koinos makes it ideally suited for integrating existing open source solutions, unlike other blockchains that are designed to be monolithic blocks of highly specialized solutions that are difficult, sometimes nearly impossible, to improve.
Universal Language Support
This is how we plan to achieve our unparalleled language support; not by reinventing the wheel, but by leveraging the best open source tools that already have incredible language support like WebAssembly and Protocol Buffers. WebAssembly (WASM) is what allows us to move so much of the system logic of Koinos into smart contracts because it allows those smart contracts to be executed as if they were implemented natively (in the blockchain itself). But WASM also has fantastic, and ever-increasing language support, which will be a big help for allowing developers to write smart contracts in any programming language.
The second technology is what we’re covering in today’s post and that’s Google’s Protocol Buffers. There are two core innovations that set Koinos apart from other blockchains in terms of upgradeability and scalability. Koinos is the first blockchain framework built on a microservice architecture, and within that architecture lies the first proto-blockchain architected to rely almost entirely on upgradeable WASM smart contracts to describe system behaviors (consensus, staking, governance, etc.). Serialization plays an important role in both innovations.
The Importance of Serialization
Within a Koinos node tiny programs (microservices) are sending countless messages to one another about the blockchain, and the blockchain (one of those microservices) is sending messages to the VM (virtual machine). Serialization is how Koinos puts all of these messages into a common format and how this serialization is done can impact how large the messages are and how fast they can move.
“Free” Language Support
Koinos-Types was the serialization solution that we developed specifically for Koinos, so it accomplished everything we needed exactly the way we wanted it. Koinos-Types is a great solution to the problem, but because it is a novel solution built specifically for Koinos, targeting new languages is very labor intensive. In other words, Koinos-Types doesn’t give us any language support “for free.”
Google’s Protocol Buffers (Protobuf) was always a potential solution that we were evaluating because of its high performance and fantastic language support. Protobuf is an open source method for serializing data that was developed by Google specifically to be faster and more efficient than existing solutions.
Protobuf gives us a ton of language support “for free,” already supporting generated code in Java, Python, Objective-C, C++, Dart, Go, Ruby, and C#, with more languages to come. There are also third-party implementations available for Erlang, Haskell, JavaScript, Perl, PHP, R, Rust, Scala, and even Swift!
Canonical Serialization
The problem with Protobuf, and the reason why we didn’t go with it originally, is that it intentionally does not always serialize data in the same exact way (“canonically”) to maximize flexibility. Technically, Protobuf doesn’t specify a deterministic serialization for each type. But when cryptographic integrity is required, this is a liability. In other words, we love the language support, but don’t love the flexibility of Protobuf for a blockchain.
Enforcing Canonicity
But even once Koinos-Types was complete, we continued to investigate Protobuf as a solution because we’re always eager to find ways to make Koinos better, even if it means redoing previous work. What we found was that Protobuf serialization was ultimately easy to understand which made enforcing canonicity relatively straightforward. Essentially it came down to every field of a Protobuf message must have an integer index which is a natural sort order and maps are not guaranteed to be serialized in any particular order.
The reason for this seems to be because not all targeted languages can guarantee a particular order. For example, Golang purposefully randomizes the iteration order of a map (a higher-order function in a programming language) to prevent developers from relying on a particular ordering.
Fortunately, the Koinos Blockchain Framework has no need for maps, so this is not an issue for us. From these restraints the canonical serialization is as follows:
- Serialize fields in field number order
- Do not allow maps
This will primarily impact developers in transaction signing because all messages within Koinos that are cryptographically referenced or verified will be serialized using this serialization.
ANY-Language Support
When it comes to delivering on our promise of ANY-language support, people typically assume that the goal should be to enable developers to write smart contracts in any programming language, however, we don’t think this will actually be the case. The smart contract component within a given application is almost by definition fairly small and the majority of the smart contract development “heavy lifting” will be done by the contract development toolkit (CDT).
TypeScript Smart Contracts
While the first CDT we release will be for C++, that doesn’t mean only C++ developers will be able to work with it. In fact, it should be accessible to most developers! That being said, a TypeScript CDT would take that accessibility to the ultimate level. A TypeScript CDT won’t just be accessible to all developers, it will be easy for them to use. With these CDTs in place, writing fantastic Koinos smart contracts will be accessible to all developers and adding additional CDTs may be nice to have (and something any engineer can develop themselves), but will certainly not be necessary because the smart contract is again only a very, very small part of an application.
The vast majority of an application’s code will be higher level than the blockchain, in middleware and front end code. This is where client libraries and microservices will be the most useful to developers and where more diverse language support will add insane value to developers. This is also precisely where Protobuf does the most heavy lifting.
Example Application
Imagine you like building applications using Ruby-on-Rails. Does that mean that you need to be able to write smart contracts in Ruby to integrate Koinos? Absolutely not! Interacting with Koinos will primarily involve using client libraries to make the right API calls and maybe building a microservice that “listens” for the relevant transactions needed for the application.
When it comes to client libraries and microservices, Protobuf has already paid incredible dividends! Because we use RabbitMQ (another open source solution with fantastic language support) for the Koinos message queue, we can already write microservices in any language that has Protobuf and RabbitMQ libraries which are practically all of the most popular languages! These include (but are not limited to) Ruby, Java, JavaScript, .NET, Python, PHP, Swift, and Go.
That’s right, thanks to Protobuf, Koinos application developers can already write microservices in any of these languages! TODAY!
Client Libraries
When it comes to the problem of ensuring the availability of high quality client libraries, our experience developing Steem has been invaluable. The biggest challenge to writing Steem client libraries was transaction signing.
Signing a transaction has 3 steps:
- Serialize the transaction to bytes
- sha256 those bytes to get a 256 bit digest
- Sign the digest using secp256k1 ECDSA
Obviously, Protobuf will be handling the serialization and has amazing language support. Sha256 comes from OpenSSL and has a wrapper in practically every language imaginable or a suitable drop-in replacement. Secp256k1 is the Elliptic Curve Digital Signature Algorithm (or ECDSA) used by Bitcoin and, for all intents and purposes, already exists in any language we would want to target.
Step ONE was the major hold up for Steem client libraries and Protobuf handles that entirely!
As you can see, figuring out how to leverage Protobuf in Koinos will provide incredible long term value to the platform. It did take a bit of additional work to enable Koinos to function with Protobuf instead of Koinos-Types, but we are excited to announce that internally we have stopped using Koinos-Types entirely and are relying on Protobuf exclusively for serialization!
This is a major step forward for meeting our objectives of allowing developers to work in the programming languages they already know and love, helping Koinos evolve at an even more rapid rate, and making it into the most accessible blockchain ever!
