remote-trait-object: One of CodeChain Foundry’s core inter-module communication systems

remote-trait-object is a Rust Remote Method Invocation (RMI) library based on trait objects.

We’re going to discuss the library in the context of why we need it and how it will be used for CodeChain Foundry. If you’re interested in using it as a general purpose communication mechanism in your applications, please check the following links instead.

• Rust docs
• Github Repository
• crates.io

Backgrounds

remote-trait-object is a general purpose library, but it plays a special role in Foundry.

Foundry

CodeChain Foundry is a blockchain engine based on a composable module system called Mold. Users can write their own modules in addition to bringing in those written by others in order to construct an arbitrary blockchain application. The reason why we provide such a composable and user-configurable module system is because we want to make as much of the application configurable as possible and foster an ecosystem of reusable modules, while reusing the underlying consensus engine across all the different kinds of applications.

Upon execution of a transaction that is essentially a state transition, the coordinator will be told to execute the transaction from the underlying consensus engine. Then the coordinator literally coordinates multiple modules constituting an application by invoking services exported by the modules to the coordinator in an appropriate order passing appropriate arguments. Transactions will be delivered to the responsible modules, and those modules will handle the executions of the transactions, which might also involve communications with other modules.

Modules

One of the most important goals in Foundry’s module system is to give module authors enough freedom in writing their modules. For this, it is necessary to support multiple kinds of modules differing in their implementation languages and the runtimes required to run them. That is, a module can be a plain Unix executable, a JavaScript module, a Python module, or a WASM module. In addition, they should also be sandboxed well by one of the sandboxing mechanisms available to them.

The most important one at the moment is an Unix executable compiled from Rust, because Foundry is also written in Rust.

Module System

As explained, the coordinator (or just ‘host’ from the module’s point of view) and modules will work together to run a blockchain node. In this process, there are of course numerous communications between the coordinator and the modules, and also among the modules.

However, since we will support various kinds of modules differing in nature, it is not straightforward to define a single mechanism in which they communicate with one another. Roughly, we have three major aspects to consider:

1. The common wire protocol used in the communication between modules — remote-trait-object is basically about this aspect!
2. The mechanisms for enabling message exchanges between modules — this aspect is dependent on the sandbox mechanisms involved.
3. How they’re linked together, meaning exchanging the initial set of services to bootstrap — this is the role of foundry-module-rt. See the section To be a module.

Communication

For the first aspect discussed above, there must be a common communication protocol that can be provided to all such different modules.

In short, we have such a common protocol based on service objects that can be exported from a party involved and imported as proxy objects into another. Such a proxy object can be seen as a stub, reference, or handle to the service if seen from the client side.

Objects have methods to call. If one module implements a service to handle the method calls, and exports it to the other module, then the importing module can use the object just like its own local object, by calling whatever method the module needs. Also, the method might carry another object as an argument or a return value of a method. With this functionality, modules (and the coordinator) can easily exchange objects as they want.

This kind of structure is the only way of inter-module or module-host communication in Foundry’s module system. It has some cool properties:

1. Simple: you can communicate with other modules as if you have an object from it.
2. Module exposes its interface exactly as much as it wants: this is because the exporter explicitly implements the service and explicitly exports the service object to the other module (or coordinator). This makes the system more decoupled, safe, and clear in one’s responsibilities.
3. Object-oriented: each service is not just a global interface; it is assumed to have its own state, and serves as a real object that the user can access via its methods. This makes the communication more object-oriented and consequently easy to maintain multiple modules.

If you want to know the details about this system, please read the design document.

remote-trait-object

So far we had a look at the overall background of Foundry’s module system. Then, what is remote-trait-object and how exactly is remote-trait-object related to Foundry?

Introduction

Again, remote-trait-object is a remote method invocation library based on Rust trait objects. It is...

1. Based on a point-to-point connection
2. Based on services
3. Uses trait objects as service & proxy objects
4. Designed to easily export and import services
5. Independent from the transport model
6. Concurrent — user can both request and handle method calls concurrently

Services

The key abstraction provided by this library is Service.

A service is the basic unit of communication that has its own methods and state. Once there is a service object on the server side, it is exported to the client side as a proxy object. You can invoke a method on it, and you can also return or pass other service objects as well. remote-trait-object takes a transport medium and constructs a context of connection upon which services' call, response, export, and import will be conducted.

The service in remote-trait-object roughly corresponds to that in the module system, though the service in remote-trait-objectdesignates a more general concept.

Service & Proxy Objects

The key idea of remote-trait-object is its use of Rust trait objects. Both the service object and the proxy object are trait objects.

Take a look at the following diagram.

This shows how a call from the client side is actually delivered to the service object. Here, the proxy object, which is a trait object for some service trait, converts the call to the byte array and passes it to the connection end that it belongs to. It goes over to the other end, and gets forwarded to the target skeleton. Skeleton has a service object in it, and it is also a trait object for the same (or compatible) trait.

Here, you can put any trait object in the skeleton and export it and make the client side (or importer side) use it as a trait object for the same (or compatible) trait.

A service will be exported when you return the object as a result of method invocation of another service. Or you can export one by passing it as an argument of some method invocation of another service as well. In such ways, modules can export and import services easily.

Macro

All the magic happens in the proc-macro that remote-trait-object provides. The ‘service’ attribute is attached to a service trait, then skeleton and stub for the trait are generated at compile time. You can use the trait as both a proxy object to import and invoke, or a service object to implement and export. Here’s a short example:

#[service]
pub trait InitChain: Service {
    fn init_chain(&self) -> (CompactValidatorSet, ConsensusParams);
}

It might take some additional arguments for advanced customization of service traits.

Context

If you establish a remote-trait-object connection, there must be two ends and each will be provided as a context to each user on both sides. A context holds multiple things to function as a remote-trait-object connection end. Since the connection is symmetric, it manages both server and client toward the other end. It also manages a registry that contains all exported services. The server will look up the registry to find a target object for handling an incoming method invocation. Since one context represents a single connection to another one, a Foundry module would have N+1 contexts of its own where N is the number of other modules that have anything to do with the module, and 1 is the Coordinator.

Transport

To use this library, you must provide an implementation of transport first. Since this library is only about the high-level communication protocol using trait objects, the underlying transport for actual data is abstracted and expected to be given by the user.

remote-trait-object defines some required abstracted traits for transports, for example trait TransportSend and trait TransportRecv. The user must implement those traits and provide instances of them while creating a connection of remote-trait-object.

It can be anything as far as it meets really basic interface such as fn send(&self, data: &[u8], timeout: Option<std::time::Duration>) -> Result<(), TransportError>, fn recv(&self, timeout: Option<std::time::Duration>) -> Result<Vec<u8>, RecvError>, and so on. It can be a plain in-process communication like crossbeam::channel, an inter-process communication using a Unix domain socket, or networking over a physically remote machine.

Foundry modules are to be sandboxed. Therefore the IPC mechanism to use is entirely up to the sandbox schemes involved. foundry-sandbox repository provides sandbox implementations as well as an implementation of remote-trait-object transport abstractions.

Role in Foundry

remote-trait-object is one implementation of Foundry's inter-module communication protocol that you can use where the underlying transport is an IPC (It is an one example of aspect 2 above). Using IPC as an underlying transport means that the module passes raw bytes for communication in some way. One example of non-IPC transport is a direct reference exchange among modules running in separate V8 runtimecontexts. Any combination of two modules where either or both are just Unix binary may have no other choice but to use an IPC. And don't forget the coordinator! It is not a module but participates as a linkable entity in the inter-module communication anyway,and it is of course a Rust program. Most of the other modules are likely to be Rust programs too, for now, as we consider that to be a most important case currently. As such, we can assume that in most cases remote-trait-object will be used as the main communication protocol.

Note that the concept module is completely irrelevant to remote-trait-object itself. remote-trait-object is responsible only for a single connection between two parties. Foundry module is just one usage of remote-trait-object that happens to have multiple instances (or contexts) of remote-trait-object as it will be connected to multiple other modules and the coordinator.

Example

#[service]
pub trait Stateful: Service {
    fn set_storage(&mut self, storage: ServiceRef<dyn SubStorageAccess>);
}

This is one example of a service that module should export to the coordinator. Each module will implement this module with whatever logic it wants, but mostly it will be just a trivial variable setting for the module’s global context that holds a current substorage.

You can notice ServiceRef<dyn SubStorageAccess>, which is a service too. When the coordinator calls this method, it will pass an instance of a service object of such trait. remote-trait-object will keep it in the registry and export it by passing a handle which is a small index that indicates the registry entry of the target service object. On the other side, the module will receive the same ServiceRef<dyn SubStorageAccess> in the implementations, and it will be imported as a plain Box<dyn SubStorageAccess> that the module can freely use.

Future

To be a module

remote-trait-object is used as a communication protocol for modules. However, there is more to implementing a module. To complete a Foundry module with Rust, you also need another library called foundry-module-rt which is a module runtime for Rust. It is based on remote-trait-object, but additionally it defines the required interface that the user has to implement, initialization logic, other utilities and so on.

As noted here, “How they’re linked together” is the role of module-rt

We will explain that too, in detail, in the next article.

Other Usages

remote-trait-object was designed as a general purpose mechanism for high-level communication among parties written in Rust. You can use it for any Rust-to-Rust process communication over whatever transport you like as long as you provide a well implemented transport in the form of impl TransportSend and impl TransportRecv. If you're parsing a byte packet and dispatching it to somewhere manually every time, it's time to try this awesome library.

We’re planning to use this library outside the module system too, because it happens to be a useful general library anyway!

Features

remote-trait-object is now at 0.4.0. There are many useful features not mentioned in this blog, and you can find them in the documentation! Of course there are missing features also. We're planning to support the following features soon.

• Customizable error handling
• Interface Description Language (IDL) support for further interoperability with other programming languages
• Advanced concurrency model (maybe use of async if Rust supports async trait later)

Summary

We have explained what remote-trait-object is, and how it is used in Foundry. remote-trait-object is both a useful general library and core communication system for Foundry application modules. We hope this powerful library helps you easily implement your own blockchain application using Foundry.