Introducing Factom Live Feed
Overview
Factom will support a new Live Feed event mechanism in the next version, which can send real-time information about what is happening on the Factom network as well as provide status information on the FactomD node itself. The functionality is implemented in two layers of which the first one is inside FactomD itself and on top of that we’ve built a second layer implementation called the “Live Feed API”.
Why a Live Feed API?
One of the “drawbacks” of blockchain is that the state where data is fully immutable on disk, so-called finality, is typically reached after many minutes. Unlike some other blockchains whenever an entry in Factom has been confirmed in the network after a few seconds, the chances of not reaching finality are really slim. Finality is typically reached within 10 minutes. We are unaware of not reaching finality tbh.
A lot of applications on other blockchains solve this problem by continuously checking in the background for confirmations. This means unnecessary polling of something that will happen at an indeterminate time in the future. The Live Feed API basically changes this around. Instead of having to poll, FactomD can now send out a lot of internal events to interested parties. This allows for reactive solutions, where your application basically subscribes to the data you are interested in. Whenever data becomes available in FactomD or on the network, your application will immediately know about it. Whether this is actual end-user data in the entries, transactions, process messages or for instance node messages, doesn’t matter. eg you can decide.
Use cases
There are several use cases for this functionality:
Reactive applications
Take for instance a wallet application. Right now, a Factom wallet creates a transaction. There is not much feedback after submission. Basically you will see your balance update and it synchronizes with the blockchain in the background. With the Live Feed API, you could easily track (your) transactions and make the user aware of the state of the transaction as it happens. So after a few seconds, you would see your transaction progressing from submission to being part of the network. After a few minutes, you would see the transaction reach finality. Some User Experience features like notifications or color changes in the wallet make the whole experience much nicer for the end-user. Typically, a wallet queries all the transactions. With the Live Feed API, it could just make one call to register itself and say, feed me all the transactions. So transactions sent to your address(es) by somebody else would show up in your wallet immediately. This doesn’t even have to be a transaction that is finalized yet.
This change in communication and reactiveness is most visible in applications with a user interface like a wallet. However an identity application typically also wants to cache information about identities locally. It uses similar synchronization techniques as the wallet example. With the Live Feed API, an identity app could become aware of updates in the background. Meaning you will always have the latest data available.
Debugging and testing
Distributed applications like Factom are much harder than your typical application. This also has an impact on the development and automated testing of such application. The Live Feed API could for instance record node and process messages. These could then be replayed at a later stage to make sure the network and the communication between nodes stay consistent in-between versions. As soon as a new version emits different events for instance, you would know. Debugging and investigating also becomes easier. You simply deploy a node with the Live Feed API enabled and when something unforeseen happens in the network, you can replay the messages to see what was going on.
Event sourcing / Event bus
An interesting solution is to use the Live Feed API as an event bus. Multiple parties can subscribe for instance to one or more chains on Factom. Every subscriber defines what data they are interested in. Let’s say a consortium of companies wants to work together, using their own IT systems. By using the Live Feed API, every company immediately gets notified when new data is being anchored in a common blockchain. The interesting part is that their application landscape could vastly differ. In the future could you even could use the Live Feed API to do event sourcing, since replaying entries (events) is a milestone of the project. This means you can always derive the latest state of data by replaying all entries in a chain.
FactomD side architecture
Right from the start, the choice was made to keep the footprint and overhead of the Live Feed API on FactomD at a minimum. We have built a thin event emitter mechanism which just queues up event data from within the FactomD processes. A new event service takes the events out of the queue and sends them over to a single consuming server. (Preferably on the local machine or within a cluster) For data serialization, we chose Protocol Buffers because of its performance and wide support for different development platforms. We’ve mapped out most of the fields that are available inside FactomD in the so-called. proto files. From these files, the model code can be automatically generated for the desired target language. This means that alternative to using the Live Feed API, developers can use their language of choice to integrate.
Live Feed API architecture
For use cases where multiple consumers are required, we have built a second layer implementation, the Live Feed API. It is basically a subscriber-based event router and can be configured to filter and distribute certain parts of the event stream to each subscriber independently. The event router also needs to do some housekeeping for when a receiver is temporarily or permanently unreachable and for some use cases it needs to make sure no events/blocks are skipped and missing blocks can be replayed from history. To enable this, each subscriber session has its own queue and parallel sender instance. The diagram of what we’ve currently implemented is as follows:
In many cases, a subscriber is only after a specific subset of events, like transactions, node message or entry data, we need a filtering mechanism. At this point, we’ve implemented GraphQL to support filtering because of its flexibility. Since we already have the entire event model mapped out in protobuf files, it made sense to reuse these to generate the GraphQL models. We found a protoc plugin called opsee which was made for this very specific use case. Unfortunately, this was an archived repository that lacked protobuf v3 support and some features like enums. We choose to fork the project to protobuf-graphql-extension to freshen up the dependency versions and add the missing functionality. A detailed guide on how to use it can be found here, and examples of how to use them in conjunction with the Live Feed API events can be found here.
Getting started
The documentation on how to enable the Live Feed API functionality in FactomD can be found here. The documentation on getting started with the Live Feed API second layer can be found here. An example project on how to implement a consumer for the first layer directly is available here.
Road map
The full diagram of what we envision the final product to be:
Milestone 1 (99% done)
· A stable & low impact event emitter mechanism in FactomD
· A lightweight transport protocol based on protobuf or json over TCP/IP
· Working live feed API event consumer & event router
· Working subscriber API with in-memory & MySQL database support
· GraphQL
· A working CI pipeline & DOCKERFILE
· Sample consumer projects
· Provide documentation / instructions
Milestone 2
· History / replay functionality (also in FactomD)
· Authentication support for REST API & web hooks requests
· Other subscription database types
Milestone 3
· Other event delivery methods as indicated in the design drawing
· UDP protocol support
· Auto-replay missed events
