DEIP tech: Event-Proxy

Event-proxy workflow schema

To inform the external world about changes to DEIP’s blockchain state, we have developed an Event-Proxy module. The module was developed using the Actor Model in Rust programming language, taking into account event-driven design.

Let’s take a look at the development in more detail.

Why we implemented Event-Proxy

Event-Proxy opens up a host of opportunities for people to build applications. Subscribers of events can act as:

• Indexers to optimize search
• ML/AI modules for creating different types of analytical models
• Web servers for providing third-party services
• Other apps which can read events flow from the Apache Kafka channel

Moreover, Event-Proxy carries out the important function of separating read and write models. This will allow us to boost network throughput by:

• Separate optimization of “read” and “write” layers
• Lowering the volume of transaction logs
• Shifting the focus of the blockchain protocol to validating business processes only

In addition, the event-oriented interface will allow us to build off-chain infrastructure to process, store, and rebuild read-models which are designed to be displayed to end users and apps in a specific format.

What Event-Proxy does

The module is basically a Rust application which uses an RPC-API framework to receive blocks and then extract data from these blocks. There are 2 sources of data for events (i.e. 2 event emitters): extrinsic operations from the external world, and operations from internal smart contracts that sit within our blockchain. In other words, extrinsics and smart contracts are triggered which make changes to the state of the blockchain and the module is able to reflect these changes.

We use the streaming platform Apache Kafka to transport the events because it’s a perfect solution for event streaming since it:

• Is open-source
• Offers high performance
• Allows to set topics and partitions to spec

The system we created has 2 main types of events: domain and infrastructural. Let’s take a close look at them.

Domain events

These types of events reflect changes in the domain model and are important for subscribers of what takes place on DEIP blockchain because these events form a chain of changes for business processes. All external changes in Substrate are classed as extrinsic, meaning that if extrinsic operations are executed successfully, they emit a domain event.

Examples of these types of events include:

• AssetCreatedEvent
• ContractAgreementSignedEvent
• DaoMemberAddedEvent

Furthermore, the protocol is able to inform subscribers about internal changes which take place as a result of some kind of condition being fulfilled (i.e. a kind of smart contract is being executed). For example, a time limit being reached, such as CrowdfundingFinishedEvent.

Infrastructure events

We created infrastructure events as a technical need for our solution. They are introduced to support the inner workings of the technology and our ecosystem as a whole. An example of an infrastructure event could be BlockCreatedEvent. Every subscriber should listen to this event as well as it gives him traction of the last processed block ID/number, that subscriber should let the Event-Proxy know on every reconnection.

Every event (domain and infrastructural) has a link to a block where the event is emitted. This allows subscribers to cancel changes produced by an event if there is a fork in the network. Moreover, events can be grouped according to channels, which allows us to distribute the load when processing events (sharding per portal/app). As a data format for events, we use JSON.

Replay

The Event-Proxy module is able to resend events from previous blocks that facilitate building infrastructure for the so-called “Replay” process in the Event Sourcing architecture. This feature allows subscribers to receive events flow, starting from any point in the network history.

Also, this process takes place automatically when the Event-Proxy is started to address a case when events might be missed if an events-subscriber isn’t working for whatever reason. If a subscriber (or subscriber’s server) was “idle” (lost connection, under maintenance, etc.), the subscriber should let Event-Proxy know the ID/number of the last block they processed to retrieve events that take place during this “idle period”.

Actors

The module’s fault tolerance is secured using an architecture solution based on the event-driven design. In this solution, errors (connection failures, invariant issues, etc.) are considered a normal part of the system’s behavior (as a system events).

Concurrent processing of the task queues is built into the foundations of the system. Each action — whether it be a request for the last block, a reaction to an error, or otherwise — is a task. Every type of task has a separate processing queue.

The module has several main system participants which are abstractions of the actor model:

• BlockchainActor
• OffchainActor
• MessageBrokerActor

Interactions between the actors take place using a special type of task (actor_task) and communication works based on event-driven principles of the architecture, whereby a request and the final outcome of a processed request creates a new task. As such, reactions to errors and reactions to expected behavior are built into the system’s design. This allows us to initiate processes from scratch if they aren’t executed successfully.

How Event-Proxy works

So let’s take a look at a short algorithm for how Event-Proxy works:

1. Launch, initialization, reading of configuration file
2. Request for the last processed block from the subscriber (last_known_block)
3. Subscribe to new blocks (new-blocks-subscription)
4. Receive current block (head_block) from the new-blocks-subscription
5. Start missed-blocks-replay process, i.e. iteration from last_known_block to head_block and publication of missed events
6. Processing of all of the following blocks received from the new-blocks-subscription and publication of new events

Dependencies

To create the Event-Proxy module, we used the following dependencies:

• substrate-subxt — the official client for Substrate nodes from Parity https://crates.io/crates/substrate-subxt
• rdkafka — a connector for Apache Kafka https://crates.io/crates/rdkafka
• tokio — asynchronous runtime and competitive programming environment https://crates.io/crates/tokio