Detailed Explanation of CESS Mechanisms (3) Storage & Content Delivery Layer, and Proof of Data Reduplication & Recovery

In the “Detailed Explanation of CESS Mechanism (2): Blockchain Layer and the Random Rotational Selection (R²S) Consensus Mechanism” article, we provided a detailed explanation of CESS (Cumulus Encrypted Storage System)’s design concepts regarding consensus mechanisms and the blockchain layer. By utilizing the Random Rotational Selection consensus mechanism (R²S), CESS ensures fair and efficient consensus while preventing the domination of a few storage miners. Furthermore, CESS is one of the few storage public chains that records metadata on-chain, allowing for decentralized data processing on the chain thanks to its efficient on-chain transaction processing efficiency.

This article will delve into CESS’s storage and content distribution layer design, along with its proprietary multi-copy recovery storage proof mechanism.

Storage Network

Although CESS combines the storage and content distribution layers into a single network, it’s essential to look at each layer’s functions separately. The storage network serves as the most critical function of the decentralized storage network, making it the “heart” of the CESS network.

Comprising storage miners, the CESS storage network addresses the problem of current decentralized storage’s inability to provide elastic and scalable cloud storage. By virtualizing storage resources into a “pool,” the CESS storage network provides unified on-demand storage to top-level or external applications, shielding them from the instability caused by underlying hardware differences.

To provide effective and available storage space to the entire CESS network, miners must pass a series of on-chain proofs, such as effective storage space, storage data volume, traffic contribution, and operational performance. Unlike peer-to-peer transactions, CESS integrates storage resources provided by miners and distributes the demand to corresponding storage service providers based on their required capacity and bandwidth through algorithms. This is achieved through virtualization technology that abstracts specific services provided by miners into “virtual” storage services, providing storage services to applications when users have data storage needs.

When a user uploads data to CESS, the data undergoes several pre-processing steps, including encryption using a trusted execution environment(TEE), sharding, and redundancy using a decentralized proxy re-encryption mechanism. The processed data is then stored by selecting miners who meet the user’s storage requirements, such as storage duration. However, unlike Filecoin’s peer-to-peer storage mode, CESS does not select a single or a few miners to complete the storage task. Instead, it randomly distributes the sliced data segments to miners who meet the requirements. This approach ensures that there is no monopoly in the storage network, and it promotes fair competition among miners.

On the other hand, this approach also maximizes utilizing storage resources. In existing storage networks, when a large-scale data storage task is received (such as data exceeding 5TB), some consumer-grade miners may be unable to provide large storage space and thus lose competitiveness. This has not been well addressed in most existing storage networks, and as the network develops, Filecoin and Arweave, among others, will inevitably move towards centralization. For example, in a network with miner A (with 3TB storage capacity) and miner B (with 1TB storage capacity), the current storage network can only store 2TB of data with miner A, while CESS can achieve maximum utilization by allowing A and B to each store 1TB.

In addition to improving utilization, this approach also reduces the hardware threshold of storage facilities. On the one hand, miners only need to perform storage tasks without complex and professional tasks such as “receiving orders” and running nodes. On the other hand, miners will randomly receive data segments independent of their size.

In this way, CESS’s storage layer truly realizes the vision of “decentralized storage” and maximizes efficiency in resource utilization rather than simply increasing storage resources for additional profit.

The decentralized content delivery network (DeCDN) in CESS serves the same purpose as traditional cloud-based CDNs. One of the biggest challenges for decentralized storage networks is not just storage itself but “data uploading.” For miners, downloading data that users need to store is relatively easy, but the network cost to upload data when users need to use it is relatively high. This leads to many miners storing data and guaranteeing its continued existence through storage proofs but being unwilling to upload it to users, resulting in network unavailability.

This problem is not unique to decentralized storage networks. Traditional cloud-based systems cannot handle the high instant concurrency and traffic generated when users directly retrieve data from cloud data centers. This is also one of the reasons why CDNs exist.

In the content delivery network, CESS has assigned two additional miner types– cache and retrieval miners to help the network run more efficiently. Cache miners cache popular/hot data to achieve faster calling speeds, while retrieval miners help applications quickly locate the location of the required data.

Proof of Data Reduplication and Recovery (PoDR²)

The data pre-processing and distribution to miners mentioned above are part of CESS’s innovative Proof of Data Reduplication and Recovery (PoDR²) mechanism. The PoDR² mechanism ensures that the network effectively stores the replicas of user-uploaded data. After any data is uploaded to the CESS system, multiple data replicas (three copies by default, more by demand) are automatically generated, and metadata required for recovery proof verification is generated for each data replica, which is then saved to the blockchain system.

At this point, CESS can distribute the processed data to various storage miners. Within a specified period, miners must report the data they store to allow the CESS system to confirm whether it is missing or damaged.

It is worth mentioning that the PoDR² mechanism statistically monitors all data segments that comprise a single file (including all replicas). Once a data segment is identified as damaged or missing, the system automatically generates new data segments and sends them to new storage miners to ensure data recoverability and improve the robustness of the system’s data storage.

The main difference in CESS’s storage mechanism is that it uses encryption, redundancy, and other protection strategies at the system’s underlying layer instead of relying on miners to perform redundant protection operations on data. Miners only need to store the system’s processed data segments and ensure their validity. Even if some miners lose data, the system can restore the original data through other data segments, greatly reducing the possibility of a single point of failure and improving data security in decentralized storage networks.

Conclusion

CESS’s storage mechanism design achieves data privacy and security protection through PoDR² and implements disaster recovery proof measures. It also maximizes storage resource utilization through resource pooling, enables the true utilization of ‘scattered resources,’ and makes remarkable contributions to innovation and progress in decentralized storage mechanisms and design concepts.

Join CESS

CESS website Twitter Telegram Discord Github Medium LinkedIn Instagram: cess_storage