Crust DSM 3.0 Upgrading

Motivation

Since DSM(Decentralized Storage Market) 2.0 module went live with Crust Mainnet, it helped Uniswap, AAVE, RMRK and other projects to smoothly store to the Crust Network. The implementation of DSM’s core concept works pretty well with DSM 2.0 module. However, the performance issue showed-up on spower-claiming function when the whole market reaches almost 1 million storage orders. To solve this problem, we bring out DSM 3.0, re-designing the tieding part algorithm of spower calculation. In order to work better with the algorithm, we changed the data structure of FileInfo.

DSM 3.0 will:

1. Logarithmic the entire spower-claiming time complexity
2. Simplify order structure
3. Simplify market rewarding system
4. Cancel the extra performance cost from benefit module

More feature details can be checked in PR#803 and the relevant ones.

The Process

Due to the changing of the on-chain storage structure(FileInfo), we divide the whole upgrading process into 3 phases to make it as seamless and frictionless as possible.

Phase I — Logic upgrading

In this phase, both new and old storage order’s data structure will be supported after the 1st forkless upgrading, the detail of supported operations show below:

Phase II — Order migration

In this phase, old orders will be gradually migrated to new orders. The whole phase will took a time period, we tested on Rocky testnet and measured all the performance cost. We decide to do the migration in two steps. We will migrate 100 orders per block in the first two hours. If everything is ok and the network is stable, we would increase the migration speed to 300 orders per block. It would take around 5 hours more to migrate the remaining orders. Totally the migration process would take 7 hours. Also, the detail of supported operations shows below:

Ideally placing new orders should work. However if the overall load of the network is high, new orders would be temporarily banned until the migration process finished.

Phase III — New Version

DSM 3.0 will be fully supported after the 2nd forkless upgrading, the detail of supported operations shows below:

Impacts

2 kinds of impacts will happen during the upgrading process.

1. 2-Layer dApps

After the Phase I, new orders can only be queried with the new interface, dApps on Crust needs to upgrade their SDK, change the code and be aware of the data structure changes.

• Bump @crustio/type-definitions to the latest version(1.3.0)
• Code change from

const fileInfo = chainApi.query.market.files(cid);

to

const fileinfo = chainApi.query.market.filesV2(cid);

• FileInfo structure change from

"FileInfo": {
    "file_size": "u64",
    "spower": "u64",
    "expired_at": "BlockNumber",
    "calculated_at": "BlockNumber",
    "amount": "Compact<Balance>",
    "prepaid": "Compact<Balance>",
    "reported_replica_count": "u32",
    "replicas": "Vec<Replica<AccountId>>"
}

to

"FileInfoV2": { 
   "file_size": "u64",
   "spower": "u64",
   "expired_at": "BlockNumber",
   "calculated_at": "BlockNumber",
   "amount": "Compact<Balance>",
   "prepaid": "Compact<Balance>",
   "reported_replica_count": "u32",
   "remaining_paid_count": "u32",
   "replicas": "BTreeMap<AccountId, Replica<AccountId>>" 
}

2. Nodes

During the Phase II, the overall load of the network would increase. We did the migration test in Rocky testnet. The memory usage increased around 500 MB and the disk io rate increased by 4Mb/s. Both validator and storage nodes should monitor their own nodes hardware indications during the upgrading process to avoid any assets loss.

Schedules

Phase 1: ~4.14

Phase 2: ~4.21

Phase 3: ~4.21