Crust Network System Optimization: Enhancing On-Chain Computation Efficiency and File Spower Processing
Crust Network has recently made significant strides in optimizing its network’s underlying cost structure, setting the stage for a more efficient and robust decentralized storage infrastructure. These optimizations not only enhance on-chain computational efficiency but also streamline the processing of file storage power (spower), positioning Crust Network as a leader in the decentralized storage space. With a bold vision to create a highly adaptable framework, Crust Network is paving the way for seamless integration across various blockchain ecosystems. By offloading complex computations to off-chain processes and leveraging cutting-edge technologies like zero-knowledge proofs, Crust Network is not just keeping pace with the evolving needs of the decentralized web — it is actively shaping the future of multi-chain ecosystem development. This commitment to innovation ensures that Crust Network remains the go-to storage solution for a diverse range of blockchain projects, offering unparalleled adaptability and efficiency.
Next, we will delve into the technical optimizations implemented within the Crust Network’s underlying infrastructure. These enhancements have been carefully designed to boost computational efficiency while effectively mitigating the challenges of state explosion. By refining the processing of work reports and streamlining the spower calculation, Crust Network has significantly reduced the computational burden on the blockchain, ensuring a more scalable and resilient network.
This deep dive will provide a comprehensive overview of how these technical improvements contribute to the overall performance and stability of the Crust ecosystem, enabling it to support an ever-growing number of decentralized applications across various blockchain ecosystems.
Crust Network completed the design optimization of its on-chain system in Q2, focusing on two main improvements:
1. Batch aggregation processing of sworker work reports.
2. Port the file spower calculation to off-chain for batch processing, followed by batch updates on-chain.
Here are the detailed descriptions of these optimizations:
1. Batch Aggregation Processing of Work Reports
Current Issue
In the previous version of the on-chain logic, each work report submitted by sworker nodes would loop through every added_files and deleted_files in the report, calling the upsert_replica and delete_replica methods of the market pallet to update the FilesV2 on-chain storage data.
Currently, a single work report can contain up to 300 added_files and 300 deleted_files, with each file processing involving an average of 4 database read/write operations. This means that a single work report could result in up to ~2400 database read/write operations. Given the current network of over 1600 sworker nodes, each reporting once per report slot (600 blocks) within 400 blocks, with an average of 4 reports per block, the peak could involve nearly 10,000 database read/write operations per block.
Due to the previous on-chain computation logic, the work report was limited to 300 added_files and 300 deleted_files per report to prevent excessive consumption of node computing resources and avoid block congestion. However, if the number of sworker nodes continues to increase, this could easily lead to bottlenecks.
Optimization Solution
Based on the current reporting logic and data analysis, it was found that work reports submitted within close time frames often contain a significant number of overlapping files. This presents an opportunity for batch aggregation optimization.
In terms of implementation, during work report processing, data verification is performed to ensure the legality of the work report. The specific handling of `added_files` and `deleted_files` is queued and processed in batches after reaching a certain quantity or time interval.
After batch aggregation, the number of database read/write operations for processing a single work report is reduced from up to ~2400 to fewer than 100, a reduction of up to ~97%.
This also significantly decreases the read/write operations for the FilesV2 data structure. If a file appears N times in batch reports, it originally required processing N times, but now it only needs to be processed once. Further optimization reduces the database read/write operations for a single file from 4 to 2.
This optimization significantly reduces the computational resources for on-chain work reports, laying the foundation for gradually increasing the limit on `added_files` and `deleted_files` per report in the future. It also greatly enhances the theoretical maximum number of sworker nodes in the network.
2. Off-Chain Batch Calculation of File Power Values
Current Issue
In the previous version of the on-chain logic, all calculations for file spower were conducted on-chain, triggered by the `market.calculate_rewards` transaction. This transaction’s logic is quite complex, involving not only the calculation and updating of file spower values but also handling file liquidation, close, and renewal.
The `calculate_rewards` transaction processes one file at a time. With over 2.6 million files currently on the chain, updating all files presents several issues:
1. It would take an extremely long time to update all files.
2. A large number of transactions would need to be initiated, consuming significant on-chain computational resources and gas fees.
3. Updating a file’s spower value requires updating all replicas’ spower values associated with that file, averaging 40–50 replicas per file. Updating one file’s spower value involves ~100 database read/write operations, consuming substantial resources.
Considering these issues, the previous on-chain logic set the file spower effective time to 3 months, which reduces incentives because miners need to wait 3 months after pulling and storing a file to see the actual spower increase. This delay in incentive feedback reduces miners’ enthusiasm for storing files.
Optimization Solution
After analyzing on-chain transaction data and storage data, the following optimizations were implemented:
1. Off-Chain Calculation: The resource-intensive spower calculation logic is moved off-chain.
2. Incremental Updates: Off-chain monitoring of file modifications, calculating and updating spower values only for files with replica changes. Since a significant portion of files have low frequency of replica changes after stabilization, it is unnecessary to update all files every time.
3. Batch Aggregation Processing: There is some overlap in the miner nodes associated with different files’ replicas. After batch aggregation, updating the spower of a batch of files on-chain will significantly reduce database read/write operations.
With these optimizations, the efficiency of updating file spower will greatly improve, reducing the original 3-month effective time to 1 day (the actual effective time will be set based on economic and anti-cheat considerations). This timely incentive feedback will significantly enhance miners’ enthusiasm for storing files.
3. Ensuring Trustworthy Updates of Spower Values
File spower values directly affect miners’ stake limits and, consequently, their stake rewards. Therefore, accurate and trustworthy updates of spower values are crucial. In the new optimization, file spower calculation is moved off-chain, requiring corresponding measures to ensure trustworthiness:
1. Optimistic Challenge:
Similar to Ethereum Layer2 Rollup’s optimistic challenge mechanism, file spower calculation can adopt a similar approach. If a challenger (usually a miner) disputes a node’s spower value, they can submit detailed verification and calculation data, initiating a dispute resolution process. If the challenge is proven correct, the relevant transaction or state can be revoked or updated, and the challenger receives a reward. If the challenge is incorrect, the challenger is penalized. Currently, we will use the optimistic challenge scheme for subsequent updates.
2. Zero-Knowledge Proof:
The zero-knowledge proof field is rapidly evolving, with a batch of mature products and frameworks emerging, such as risc0. These zkEVM/zkVM products can significantly reduce the development cost of off-chain trusted computation and on-chain trusted verification.
In the future, Crust will continue to monitor and verify such zkVM products and plan to rewrite the spower off-chain calculation module using zk framework, and perform zk verification on the Crust mainnet, ensuring trustworthy updates of spower values.
Additionally, as zk products become more mature and stable, Crust will consider transitioning the sworker program running on miner nodes from TEE technology to zk technology. This change can reduce server configuration costs and complexity for miner nodes and increase server selection diversity. Crust will release relevant information separately as more progress and plans are made.
Summary
Overall, this optimization upgrade brings several key benefits to Crust Network:
- Significant Performance Improvement: By batch processing work reports, the number of database read/write operations has been reduced from a maximum of 2,400 to fewer than 100, lowering the computational load by approximately 97%. This means the system can operate more efficiently when processing large volumes of files, reducing performance bottlenecks caused by excessive database operations.
- Enhanced Scalability: Reducing the consumption of computational resources paves the way for network expansion. As the number of sworker nodes increases, this optimization ensures that the network can support more nodes without encountering bottlenecks due to limited resources. Consequently, the theoretical maximum number of nodes in the Crust Network has been significantly increased.
- More Efficient Operations: For users, especially miners and node operators, this means they can handle more files and reports without worrying about excessive resource consumption or network congestion. This improvement enhances system responsiveness and overall efficiency, boosting miner enthusiasm and improving user experience.
- Great Future Potential: These optimizations not only improve the current system’s efficiency but also lay the groundwork for gradually increasing the number of added and deleted files per work report in the future. This means that users and developers will be able to handle larger datasets without worrying about performance degradation.
- Increased Miner Incentives: By reducing the time for file storage power to take effect from 3 months to 1 day, miners can receive rewards more quickly. This timely incentive feedback significantly enhances miners’ motivation to store files, encouraging more miners to participate in the network.
- Reduced Operational Costs for Miners: The reduction in computational resource consumption and lower gas fees directly decrease miners’ operational costs, improving their economic efficiency. This allows miners to achieve greater returns with less investment in the network.
In conclusion, Crust Network is not only optimizing its underlying system to enhance performance and scalability, but it is also strategically aligning these improvements with Crust’s long-term development roadmap. These foundational optimizations bring Crust even closer to its goal of becoming a premier decentralized storage infrastructure.
Over the years, Crust has not limited itself to the Polkadot ecosystem. We’ve expanded our reach to support a wide array of Ethereum Layer 2 solutions, including Arbitrum, Optimism, Polygon, Base, Zksync and more. Through the introduction of the EthDA protocol, Crust leverages innovations like EIP-4844 to facilitate efficient data availability, addressing the high gas costs and potential congestion issues on Ethereum. Looking ahead, we are on the cusp of launching a dedicated storage protocol and tools specifically tailored for the TON ecosystem. Each of these initiatives is a crucial part of Crust’s mission to provide robust decentralized storage solutions across various blockchain ecosystems.
These efforts exemplify Crust’s commitment to innovating and experimenting with multi-track storage solutions, each designed to meet the unique demands of different blockchain ecosystems. As we continue to evolve, Crust Network remains steadfast in its pursuit to become the backbone of decentralized storage, providing scalable, efficient, and adaptable solutions for the ever-growing Web3 landscape.
