Computer processing speeds have been increasing in accordance with Moore’s law: roughly doubling every two years or so for a period of decades. This means that we now have the fastest speeds ever, but also that we are generating masses of amounts of data, with some estimates indicating that, for every person on earth, the world will generate 1.7MB of new data every second by 2020. Illustrative of Moore’s law is the fact that over the last two years alone, 90% of the world’s data was generated with over 2.5 quintillion bytes of data now created every single day.
If we are creating so much data, how do we process it fast enough to store it without long delays? This is a question that has puzzled blockchain platform developers who have struggled with improving network latency for data storage on distributed ledgers, until now.
Blockchain works by storing data across a distributed environment in which each agent keeps a copy of the ledger. This means that when new data is added this has to be processed throughout the ledger as a whole. This makes it difficult to process large amounts of data since the update takes place in many places and across many different nodes (computers) within the network. On traditional blockchain platforms, this hinders the network from reaching speeds which are required in industries or applications where many data entries can be made at once. For example, the first blockchain — the Bitcoin network — can handle just 4.6 data transactions per second: far from adequate for enterprise implementation. The problem that slows down transaction speeds in such networks is that validation of calculations takes place by (almost) all network participants.
There needs to be a different approach to transaction processing, for example, through partitioning and categorizing the work nodes complete, assigning differing roles to improve throughput. Insolar has introduced such features to overcome the DCS Theorem. However, let’s delve a little deeper into how the platform deals with large and long transactions.
Large transactions: calculations of large data sets or storage of large documents.
Long transactions: complex calculations which require lots of inter-contract calls.
Data processing cycles
Insolar introduces time-limited cycles in which data objects should be processed. This is so that the network can synchronize throughout, with the beginning of a new cycle signaling the start of a new object calculation or validation phase. On Insolar, when the cycle ends, a new node must be assigned as an Executor for a calculation on a certain object, so that no node can illicitly manipulate the processing outcome. But what happens when the Executor node can’t manage to process the data within the allotted time frame? This may occur due to the data being too large or because there are several other parameters needed to calculate the data: in that case, they have to be retrieved from elsewhere (long).
For such situations, Insolar has implemented a mechanism of continued processing: at the start of a new cycle, the original Executor which has not managed to complete the transaction calculation requests permission from the newly designated Executor. Continued processing of data may extend into the next object processing period due to:
- lots of documents relevant to the transaction being stored on chain (so they are large, and it takes some time to read them — lengthening the period it takes to process);
- the transaction being complex, involving lots of inter-contract calls and requests for data.
How it works
A request is sent for a transaction to be processed to Virtual Executor (VE), but the transaction is heavy, meaning it is too large and requires data collection from other sources (contracts) to execute. As such, on the next object processing period beginning, the first Virtual Executor (VE1) communicates to the next nominated VE(2) that it hasn’t finished computing the transaction. VE2 then gives the authorization to VE1 so that it can continue executing in the second Pulse. This authorization is provided via a delegation token, allowing VE1 to continue the transaction execution. This process can repeat across several cycles until the computation is complete.
The method implemented by Insolar Blockchain Platform differs in regards to other blockchain platforms: there is a synchronization tool (Pulse) which carries entropy that allows to assign roles at random — determining which nodes execute and validate transactions. On other platforms, either execution runtime can last so long that the calculation can’t even be performed — they don’t allow execution of large data sets to take place; either the cost of performing such a calculation would be uneconomical. This is mainly due to other platforms not partitioning the work nodes have to complete; instead all nodes simultaneously complete the same roles. On still other platforms, there is a centralized notary which confirms/allows the continuation of calculations.
The ability to cope with long and large data transactions is just one more industry-first feature setting the standard in the sphere of distributed ledger technology.
- Check our Github and leave feedback on the code.
Follow Insolar on social media: