ICON Network — Paradigm’s on-chain observation of ways to reach scalability

Dear ICONists! We, ICON’s P-Rep node, believe that one of the appropriate ways to contribute the network is to develop applications on the scalable ecosystem, but scalability is also one of the main problems that we need to be addressed. User confidence means a stable network and unique product. In the process of implementing some options, we found something to improve in the source code. In this article, we would like to describe the shortcomings found and discuss with the community how to solve them.

WARNING: The following text contains technical details

The Paradigm development team had the task of writing a script that would upload all the ICON node transactions in the PostgreSQL database in order to more conveniently analyze the network status and enable the end-user to monitor the transaction status as quickly as possible. Hence, we faced the following issues:

• Data structure. We faced the problem of different data displays, and we needed to further study the data structure in the ICON blockchain.
• Inappropriate DB. Leveldb, a key-value database, is not very suitable for storing the data because of the position of the data is not optimized, thereby duplicating the data appears, which leads to incrementation of the database size. Mostly, developers use popular and straightforward solutions. The opt for LevelDB is based on the fact that the data is stored on the node, so the ease of implementation prevailed over the convenience of its use. The choice of a database should be based on the nature of the data itself. The nature of the data in the blockchain is very similar to the branching used in Git. We have encountered attempts to develop a database specifically for the blockchain, such as BigchainDB and CovenantSQL.
• Low throughput capacity. Currently, a throughput of the ICON node on a server with one RabbitMQ instance is 250 tx/sec. By default, a node is delivered via Docker with one RabbitMQ instance. We consider that speed can increase by making several instances of RabbitMQ.

• RabbitMQ. Using RabbitMQ as a transport for data is a significant disadvantage of reading data from the ICON blockchain. We can describe the problem raised by the example wherein we had to choose data for each transaction hash:

We needed to get a list of transactions at a certain height.

Started with a request 11633728 ( the height of the ICON blockchain):

> tbears blockbyheight 11633728

We got the result:

{
 “jsonrpc”: “2.0”,
 “result”: {
 “version”: “0.3”,
 “height”: 11633728,
 “signature”: “Nv9S80hplTjVJzD74f6AL1Pnbt / l + VU2Bc0w6OyoxEJDeZ0RBRAvlma2ESE0M + vR1A9QQSwCsvdCCNfuC1DE9AA =”,
 “prev_block_hash”: “ddfc7bdcc228ec058a009240af70770a9526826cf9ca702855880e496af1dc04”,
 “merkle_tree_root_hash”: “885456d072d47335b1837680ae9294b70a67844fbbb4fd1b0d1ce3ee42c82c98”,
 “time_stamp”: 1575035142021706,
 “confirmed_transaction_list”: [
 {
 “version”: “0x3”,
 “timestamp”: “0x5987c7226224a”,
 “dataType”: “base”,
 “data”: {
 “prep”: {
 “irep”: “0x9d6e72f9fdfe97b1550”,
 “rrep”: “0x282”,
 “totalDelegation”: “0x9c41225c5a8c39165c4e66”,
 “value”: “0x3ed115e3d3bd3eb9”
 },
 “result”: {
 “coveredByFee”: “0x88354c9cf9000”,
 “coveredByOverIssuedICX”: “0x0”,
 “issue”: “0x3ec8928f09edaeb9”
 }
 },
 “txHash”: “0x63e4069bb7d4c52774020da6818868ec032eb6f5ae3011151697080878df405b”
 },
 {
 “version”: “0x3”,
 “from”: “hx60af9d44afa1fdfe03ddef1b5ea72ae6b9b69b4d”,
 “to”: “cx1b97c1abfd001d5cd0b5a3f93f22cccfea77e34e”,
 “timestamp”: “0x5987c7201bb20”,
 “nid”: “0x1”,
 “stepLimit”: “0x2625a00”,
 “dataType”: “call”,
 “value”: “0x4563918244f40000”,
 “data”: {
 “method”: “bet_on_numbers”,
 “params”: {
 “numbers”: “1,2,3,5,6,7,8,9,11,12,13,14,15,16,17,18,20”,
 “user_seed”: “yomuma”
 }
 },
 “signature”: “il991wwabHkAMBIejFY8kODnaAfG5ApOYvhDtdAhTR54 + xde / LMsb2DGVaxYJGM1OV0cN4t0CSHlzQy5WDkzcQA =”,
 “txHash”: “0x762d6093be8f2adb2d66e8e3e386c72156fe15b473fb3dfb8de92f3b65520e4d”
 }
 ],
 “block_hash”: “133dea300094b485331cb3e4d94f631f5f80ad304a9909867b65cb99fea0190a”,
 “peer_id”: “hx262afdeda4eba10fe41fa5ef21796ac2bdcc6629”,
 “next_leader”: “hx262afdeda4eba10fe41fa5ef21796ac2bdcc6629”
 },
 “id”: 1
}

Although there is a “confirmedTransactionList” field, in some cases it contains only basic transaction data.
To get all the data about the transaction, we need to send a request indicating the hash of the transaction:

> tbears txbyhash 0x762d6093be8f2adb2d66e8e3e386c72156fe15b473fb3dfb8de92f3b65520e4d

Result:

{
 “jsonrpc”: “2.0”,
 “result”: {
 “version”: “0x3”,
 “from”: “hx60af9d44afa1fdfe03ddef1b5ea72ae6b9b69b4d”,
 “to”: “cx1b97c1abfd001d5cd0b5a3f93f22cccfea77e34e”,
 “timestamp”: “0x5987c7201bb20”,
 “nid”: “0x1”,
 “stepLimit”: “0x2625a00”,
 “dataType”: “call”,
 “value”: “0x4563918244f40000”,
 “data”: {
 “method”: “bet_on_numbers”,
 “params”: {
 “numbers”: “1,2,3,5,6,7,8,9,11,12,13,14,15,16,17,18,20”,
 “user_seed”: “yomuma”
 }
 },
 “signature”: “il991wwabHkAMBIejFY8kODnaAfG5ApOYvhDtdAhTR54+xde/LMsb2DGVaxYJGM1OV0cN4t0CSHlzQy5WDkzcQA=”,
 “txHash”: “0x762d6093be8f2adb2d66e8e3e386c72156fe15b473fb3dfb8de92f3b65520e4d”,
 “txIndex”: “0x1”,
 “blockHeight”: “0xb18440”,
 “blockHash”: “0x133dea300094b485331cb3e4d94f631f5f80ad304a9909867b65cb99fea0190a”
 },
 “id”: 1
}

Note: Although the first request with the “blockByHeight” command contains the same data as the request with the “txByHash” command, this does not mean that all transactions have the same data. We need to make a request with the “txByHash” command for a specific hash to be sure in completeness of transaction data. Obviously, such a problem imposes restrictions on us to efficiently select data from the blockchain.

Our proposal

Develop a proper data structure description

It would be substantial to receive information from ICON Foundation considering data formats for different versions of the blockchain.

Change the database

We propose using a database that meets the requirements of convenience for developers and which takes into account the nature of the data in the blockchain. Limitations on the speed of data sampling from the ICON blockchain did not completely satisfy the development team. And we decided to read the blockchain without using ICON RPC. The data reading was done using the Plyvel library which reads the ICON blockchain with the LevelDB database.

Using several instances

We consider that speed can increase by making several instances of RabbitMQ or use different software for reading data from the ICON blockchain.

Paradigm team will gladly listen to comments and discussion on the topics raised above. For all questions and suggestions, contact CTO Gregory Shabalov @grshabl (in telegram) and stay tuned for updates on our social network pages!

Twitter
Telegram
Reddit