A Guide to Chia Data Layer Development and Creating Offer Files for P2P Data Exchange
Edit: 1/30/2023 Tweaked the description for what fees do for data layer offers to make it a more accurate description
Prelude to Chia Offers
In the realm of decentralized finance (DEFI), coin swapping is facilitated through the use of Automated Market Maker (AMM). To ensure liquidity, Smart Contract based AMM’s incentivize liquidity providers (LPs) to deposit tokens into the AMM, allowing for seamless swaps between tokens. However, this also exposes LPs to the risk of impermanent loss. To mitigate this risk, AMM’s offer yields to LPs as an incentive to provide liquidity. These yields are often provided in the form of inflation tokens, which hold no intrinsic value and are frequently used as a means of exploiting unsuspecting investors. The oversupply of these tokens eventually leads to their depreciation and the loss of incentive for LPs, resulting in the failure of the AMM’s. This practice raises ethical concerns, as it often misleads investors who are unaware of the token’s eventual depreciation and exploits the “HODL” mentality as an exit strategy for early adopters or venture capitalists.
Furthermore, the entire DEFI ecosystem is subject to the vulnerability of fund loss through exploits discovered by hackers in the underlying smart contracts. These exploits can take various forms, including smart contract defects, manipulation of oracle prices, flash loan attacks, and others. The process of patching these vulnerabilities is a constant battle, as new exploits are continuously developed. Unfortunately, this has resulted in the loss of billions of dollars through these platforms.
The Chia Network offers a solution to the aforementioned vulnerabilities in DEFI by eliminating the requirement for massive funds to be pooled in a single point of failure. Its system operates on a peer-to-peer (P2P) basis, with no intermediaries involved. A user who wishes to buy a particular asset will create an “Offer file” specifying the amount of the asset they wish to purchase and the item they are offering in exchange. This file can be sent directly to another user for a private exchange or submitted to an “Offer Marketplace” such as https://dexie.space . The accumulation of multiple offers in the marketplace eventually forms a traditional order book, but remains P2P. Funds never leave the user’s wallet during an open offer and there is no central pooling of funds, minimizing the risk of exploitation and loss of funds. Additionally, this system operates more ethically, as there is no need to incentivize LPs with yield tokens that would eventually be dumped onto innocent investors.
Overview of Data Layer Development
All Data Layer Remote Procedure Calls (RPCs) can be accessed through port 8562 by default, unless specified otherwise in the config.yaml file located in the CHIA_HOME directory. You will want to make sure your application includes the correct SSL certificates located at CHIA_HOME\mainnet\config\ssl\data_layer
Data Organization
The Chia datalayer can be thought of as a key-value store, making development on it straightforward with a few key considerations. One important aspect is organizing your data, as each instance of the data store is referred to as a Singleton, representing a single key-value store. However, there is no limit to the number of Singletons that can be created. Additionally, a key-value pair within a Singleton can store the ID of another Singleton, allowing for the creation of a complex hierarchy of Singletons.
Singleton Reference Patterns
Versioned Singleton Pattern
By having a version control singleton, you can separate different versions of your data structure within different sub-singletons. When upgrading your application, you can simply add a new version of your data structure while preserving the previous one. This approach ensures a seamless transition for users who may not have upgraded their application yet. It also enables you to easily switch between different versions of the data structure for testing or debugging purposes. The key/value pairs within the version control singleton can store the ID of each sub-singleton, enabling you to access and manipulate the data stored within each version.
Table Reference Pattern
Another useful pattern I call table reference pattern. In this pattern, I have a main singleton that acts as an index, it references other singleton stores which act as tables. Each table singleton stores a collection of related data. This allows me to organize my data in a more meaningful and structured way, making it easier to interact with. Additionally, it allows me to keep the data size of each singleton small and manageable, which helps to optimize performance.
Additionally, it is important to note that these patterns can be nested within each other. For example, one common approach is to have databases versioned as V1, V2, … V(n), each referencing a singleton that in turn holds a collection of tables. This allows for a highly organized and flexible data storage system within the Chia datalayer.
Creating A Singleton
Creating a new Singleton in the Chia datalayer can be done through a simple RPC call. The following is an example of a curl command for creating a new Singleton store:
curl --location --request POST 'https://<node_url>:8562/create_data_store' \
--header 'Content-Type: application/json' \
--data-raw '{
"fee": 300000000
}'This will create a new Singleton store in the Chia datalayer, ready to be populated with key-value pairs. The <node_url> should be replaced with the URL of the Chia node to which the RPC call will be sent.
In the response of the /create_data_store RPC call, you will receive a Singleton ID. It is crucial to save this ID as it will be needed for referencing the Singleton store in future interactions with the Chia datalayer. For example, you can store the Singleton ID in a variable or database for later use. This ID acts as a unique identifier for the Singleton store and is necessary for any future operations on the data stored within it.
Broadcast your singleton to the network
By calling the /add_mirror RPC, other users will be able to discover and access your datalayer from different nodes in the network, ensuring that it remains highly available and easily accessible.
Here is an example cUrl:
curl --location --request POST 'https://<node_url>:8562/add_mirror' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "e1ae7950f2a89736fa1268273ea7d4e3fc771fb99cccce719ceed3237d93fa49",
"urls": "http://<local_public_ip>:<http_datalayer_server_port>",
"amount": 1,
"fee": 300000000
}'The “id” field should be your singleton ID, the “urls” field should be the URL where your datalayer is hosted, and the “amount” is the default coin amount, and “fee” fields determine the amount you’re willing to pay for the mirror to be added. The data should be formatted in JSON and sent as a POST request to the specified endpoint. In most cases this will be your local IP address and Datalayer http server port.
Populating your Store
Once you have obtained your Singleton Store ID, you can start populating it with data. The Chia Datalayer offers two methods for manipulating the data store: “insert” and “delete”. To update existing data, you must perform a “delete” operation followed by an “insert” operation. This approach to updating data can be simplified by implementing a helper utility within your application, providing a more intuitive interface for managing data within the Singleton store. By limiting the available methods for data manipulation to “insert” and “delete”, the Chia Datalayer ensures the data store remains simple and straightforward to use, while providing a secure and reliable foundation for your application’s data.
ChangeList
Inserting Data
To insert data into the datalayer, you need to create a changelist and pass it to the /batch_update RPC. The changelist should contain a list of key/value pairs in hexadecimal format that you want to insert into the store. Here’s an example of a curl request that inserts data into a datalayer store:
curl --location --request POST 'https://<node_url>:8562/batch_update' \
--header 'Content-Type: application/json' \
--data-raw '{
"changelist": [
{
"action": "insert",
"key": "666f6f626172",
"value": "53657269616c697a656444617461"
}
],
"id": "e3110c4c1af920f84dab8cfc92f02a652c5e02dd6e9194bb935f89dbf4652bec",
"fee": 0
}'Here, “id” is the Singleton Store ID, “changelist” is the list of key/value pairs you want to insert, and “fee” is the transaction fee.
In this example, The hex value 666f6f626172 is the string “foobar” and the value 53657269616c697a656444617461 is the encoded string “SerializedData”
It’s important to note that the data layer requires the keys and values to be encoded in hexadecimal format, this allows for any data to be stored as long as it’s encoded in hexadecimal. This can include binary data, serialized data, JSON objects, base64, CLVM, or any other type of data as long as it’s encoded in hexadecimal. The data layer does not impose any specific structure for the data being stored.
Deleting Data
Here’s an example changelist for deleting data:
curl --location --request POST 'https://<node_url>:8562/batch_update' \
--header 'Content-Type: application/json' \
--data-raw '{
"changelist": [
{
"action": "delete",
"key": "666f6f626172",
}
],
"id": "e3110c4c1af920f84dab8cfc92f02a652c5e02dd6e9194bb935f89dbf4652bec",
"fee": 0
}'In the example, the “key” value “666f6f626172” represents the string “foobar”. you simply change the action to “delete” and you only need to supply the key that is being deleted.
Updating Data
Chia Datalayer does not have a native “update” concept, and it only supports “insert” and “delete” operations. An update is abstracted as a “delete” then an `“insert”`.
curl --location --request POST 'https://<node_url>:8562/batch_update' \
--header 'Content-Type: application/json' \
--data-raw '{
"changelist": [
{
"action": "delete",
"key": "666f6f626172",
},
{
"action": "insert",
"key": "666f6f626172",
"value": "53657269616c697a656444617461"
}
],
"id": "e3110c4c1af920f84dab8cfc92f02a652c5e02dd6e9194bb935f89dbf4652bec",
"fee": 0
}'In Chia Datalayer, there is no built-in support for updates, so to make updates, one must abstract it as a combination of deletion and insertion. It is important to note that the order of these operations in the changelist is critical as the Datalayer processes each command in the order in which it was received. However, if a deletion is attempted on a key that doesn’t exist, an error will occur and the operation will not be completed. As a result, it is important for the application to keep track of the data it has inserted if it intends to perform an update in the future.
Retrieving your Store Data
Root Hash
A “Root Hash” is a crucial component in the DataLayer architecture, representing the state of the store at a given moment in time. It is a cryptographic hash that serves as an identifier for a specific state of the store, providing a secure and efficient way to retrieve the data stored in the store.
Retrieving data from a DataLayer store requires knowledge of the store’s current Root Hash. The Root Hash represents the state of the store at a particular point in time and serves as a starting point for accessing the full history of the store from its creation to its current state. The Root Hash is also useful for determining when updates have been successfully committed to the store. Because the root hash serves as a unique identifier for the state of the store at any given time, allowing you to easily compare the current state to previous states. If the root hash remains the same, it means that the state of the store has not changed, and if it has changed, you can retrieve the updated data by accessing the store with the new root hash. Additionally, a store that has never had data inserted will always have a root hash of 0x0000000000000000000000000000000000000000000000000000000000000000.
By storing the previous Root Hash in your application, you can monitor for changes to the Root Hash and retrieve any updates that have been made to the store.
Here is an example cURL for getting the latest root hash, This RPC gets the latest Root Hash from the latest update.
curl --location --request POST 'https://<node_url>:8562/get_root' \
--header 'Content-Type: application/json' \
--data-raw '{"id":"3c35da0f22454874318419d710f2521db678d544e10599bc4ed99c094e663083"}'Here is an example for getting the last root hash for many stores, This RPC is useful for getting the latest updates from many stores at once.
curl --location --request POST 'https://<node_url>:8562/get_roots' \
--header 'Content-Type: application/json' \
--data-raw '{"ids":[
"37af613ae4547d3061f210aaad8cf92be7ddd199a8f344a0173bd95d150210a9",
"c7b28cbf8cf5593b3c13ac190531f7d5b20946338b913b0af538a5c55ca26423"
]}'Finally here is an example to get the entire Root History:
curl --location --request POST 'https://<node_url>:8562/get_root_history' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "5708a6c2a20a00414c51b70b994cce09bb36618aa4e1eab3876bb94ac641515a"
}'This RPC returns a list of Root Hashes, each representing a unique state of the store. Each Root Hash is accompanied by a block height, representing the point in time the state was recorded. This information allows you to retrieve the state of the store at any point in time, making it easy to review changes, auditing purposes, and building a timeline of changes.
Retrieving your data
Once you have your Root Hash you are now ready to retrieve your data from the store. You need two pieces of information. The singleton ID of the store you want to retrieve data from and the root hash you want to look at. The RPC to do this is called /get_keys_values
curl --location --request POST 'https://<node_url>:8562/get_keys_values' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "7a213fda75d97f762ad10e94c92f23d2441d48fd1fbfd2d18cb54fed3738e86e",
"root_hash": “b1d1c8cb0f0ed5a5f23891f45d2211f7fcddfeced99dc7c9f9a06a7c7a08b2df”
}'In this example, the singleton ID is the unique identifier for the datalayer store. The root is the specific state you want to retrieve data. The response from the RPC will be an array of the key/values from your store in hexadecimal format. It’s important to keep in mind that the DataLayer is a non-opinionated data store, meaning it doesn’t dictate how the data should be structured. As such, it’s up to the application to correctly interpret the data and decode it into a format that can be used within the application.
Overview of Data Layer Offers
Whereas the traditional Chia Offer file specifies an intent to trade an asset if the offer is accepted, A Data Layer offer allows for an exchange of data in the form of a database adjustment. If both parties agree on the proposed adjustments, the Data Layer will execute the changes to both databases simultaneously. This is known as a “Atomic Update”. This mechanism enables a secure and efficient exchange of data without the need for intermediaries. The Data Layer offer gives a flexible way for applications to exchange information in a decentralized manner, ensuring the data remains secure and tamper-proof.
Makers and Takers
In a Data Layer offer, the maker and taker both agree to alter their respective datastores in a specific way, as outlined in the inclusions list. The maker provides the store ID they wish to modify and the list of upserts they plan to make. The taker, in turn, receives the store ID the maker wants to modify and the list of upserts the maker requests to make in return. This allows both parties to exchange data in an atomic manner, ensuring that the changes are made simultaneously and without interference. To create a Data Layer offer, the maker and taker must both agree on the proposed changes and the store IDs to be altered. It’s important to note that the store ids specified must be owned by each party. In a traditional offer, any party can accept the offer, in a data layer offer, only the party who owns the store id listed can accept the offer.
Generating the offer
Here is an example of using the /make_offer RPC to generate an offer.
curl --location --request POST 'https://<node_url>:8562/make_offer' \
--header 'Content-Type: application/json' \
--data-raw '{
"maker": [
{
"store_id": "a92f96e3d6e893f23ac6783e289495d5e9821df3ee5962a9fea5fdfdcc5048f1",
"inclusions": [
{
"key": "70726f6a6563742d34",
"value": "7b2070726f6a6563744e616d653a2022666f6f22207d"
}
]
}
],
"taker": [
{
"store_id": "a92f96e3d6e893f23ac6783e289495d5e9821df3ee5962a9fea5fdfdcc5048f1",
"inclusions": [
{
"key": "70726f6a6563742d34",
"value": "7b2070726f6a6563744e616d653a202262617222207d"
}
]
}
],
"fee": 0
}'It is important to note that, unlike the changelist, everything in the inclusions list is treated as an upsert, meaning that the data on each side can be updated or replaced, but currently, there is no way to delete a key/value pair in the offer.
Furthermore, the optional "fee” for prepaying the blockchain transaction fee can be included by the maker. The taker also has the flexibility to use this fee or add to it later. Although both the maker and taker can opt to omit the fee, this is not recommended as it may result in uncertain processing times for the transaction to be committed to the blockchain.
The response from the /make_offer RPC is an “Offer File”, which is a representation of the proposed database adjustments agreed upon by the maker and taker. This file can be saved as a text file and sent to the other party for them to review and potentially accept. The acceptance of the offer would trigger the agreed upon simultaneous adjustments to both databases.
Canceling the Offer
When an offer file is generated a pending transaction will be opened on the wallet. This will prevent the user from making additional transactions until the offer is either accepted or canceled. If the maker would like to cancel the offer, they can call the following RPC. This will close the pending transaction and allow the user to make additional transactions. It’s important to have this feature in place as it helps maintain the integrity of the datalayer by preventing changes while a pending offer is in place.
curl --location --request POST 'https://<node_url>:8562/make_offer' \
--header 'Content-Type: application/json' \
--data-raw '{
"trade_id": "4e9cb14186b13b4fc043015efe213f5a4c507d8f837914647a19825e6d05dfa9",
"secure": true,
"fee": 300000
}'The “trade_id” can be found in the offer file.
Verifying the offer
To check the validity of the offer, the taker can use the RPC /verify_offer to retrieve the current status of the offer. If the response indicates that the offer is still open and not canceled, the taker can then proceed with accepting the offer.
curl --location --request POST 'https://<node_url>:8562/verify_offer' \
--header 'Content-Type: application/json' \
--data-raw '{
"fee": 0,
"offer": <the contents of the offer file>
}'The proposed changes will be in the offer file in hexidecimal format, it is good practice for your application to decode this information and display it to the taker so they can review the contents of the offer and understand the changes they are about to accept before making a decision.
Accepting the Offer
Once the taker has confirmed that the offer is not stale and they are happy with the changes proposed, they can proceed with accepting the offer. This will cause the two parties’ Data Layer databases to be altered simultaneously as specified in the offer file and the corresponding adjustment in the response. This ensures that both parties agree to the same changes, and that the exchange is atomic, meaning either both changes occur or none occur, so that neither party can be left in an inconsistent state.
The offer can be accepted using the /take_offer RPC:
curl --location --request POST 'https://<node_url>:8562/take_offer' \
--header 'Content-Type: application/json' \
--data-raw '{
"fee": 0,
"offer": <the contents of the offer file>
}'And this is it, the offer is completed and the corresponding adjustments are complete.
Conclusion
The Datalayer offers a new way for developers to create powerful peer-to-peer applications on the Chia Network. In future posts, I will explore the various types of applications that can be built on the Datalayer and how they can be designed and implemented.
