Storage deposit on the Shimmer Network — an ELI12
If you tried sending some tokens around on the shimmer network, you might have noticed this text saying you need to pay a storage deposit. In this post, I will explain why we need it and how it enables feeless microtransactions
Note: This is an updated version of an older post (link)
To better explain why it is needed, we’ll have to take a quick dive into some key inner workings of the future Shimmer network:
To stay up and synced, every node must constantly provide the CPU power, network bandwidth, RAM, and disk storage required by the network. For example, if a node has insufficient CPU power or lacks network bandwidth during a spam attack, it will start to fall behind. If the activity rises, nodes will become more expensive, which reduces decentralization. Being a feeless network, Shimmer needs to establish measures to prevent excessive resource usage. In this post, we will look at disk usage in detail.
Data Storage on the Shimmer network
Every Shimmer node has two main databases:
The tangle database
The tangle database stores all messages as they arrive. If another node requests a missing message to sync up with the network, it is read from this database and sends it. It also tracks some additional data like the confirmation status of a message and is used to validate token transactions.
Since you don’t have to keep old messages after the coordinator decided if they are confirmed or not, you can choose a maximum size in the node configuration. It is possible to run a node with just a few minutes of tangle history, in which case you would have an extremely small tangle database. The other extreme would be a permanode that stores the entire history of the tangle but might require several Terrabytes of disk space in the future (currently, the requirement for the IOTA Mainnet is ~1TB). As a result, this database isn’t too big of a problem.
The ledger state database
Just like bitcoin, Shimmer is a UTXO-based ledger, that works with outputs. Think of an output like a check, written to your address with a specific amount of tokens. Of course, you can have multiple checks to the same address, your net worth is the sum of all checks written to all of your addresses. If you want to make a value transaction, you cash in one or multiple checks and create new ones. As you have to create checks with exactly the balance of the checks you just cashed in, usually this results in one check to the user you want to send tokens to and one back to yourself with the remainder. For example, in the graphic below, transaction D spends a 15Mi output created in transaction B, then creates two new outputs, one with 10Mi to blue and one with 5Mi back to itself. Of course, just like you can only cash a check once, you can only spend an output once. Transactions that try to use an output that was already spent are rejected (see Transaction F2)
To maintain all balances, a node must therefore keep track of all unspent transaction outputs (UTXOs). This list of outputs is called the ledger state, as it states all token balances at a specific time. This state is also downloaded when you first set up your node and backed up to your disk when your node performs a snapshot. Whenever a value transaction is confirmed, all outputs it spent are deleted from the database and the newly created ones are added. If more outputs are created than are spent, the ledger state grows in size. However, we obviously cannot prune the ledger state database as that would remove tokens out of wallets and could also lead to different nodes having different balances for the same address.
As a result, if this database grows too large, we have a huge problem. Not only because we can’t delete data, but also because the only way to reduce the size is by token holders actively spending multiple outputs into a single one. This issue exists on all current DLTs, with Bitcoin having a state of about 4,8 GB with 84 million unspent outputs. On more utilized DLTs like Ethereum or Solana, the state goes well into the hundreds of gigabytes. The current ledger state of the IOTA mainnet is just at ~20MB. However, with more utilization and additional data like NFTs or native tokens available on the Shimmer network, it is just a matter of time until this increases. As Shimmer does not have any fees, it is pretty cheap to add new outputs. Depending on the resources you have available for POW, this might take you a few months, but nobody could ever clean it up. Therefore, it must not be cheap to fill up the ledger state.
Short recap
- We need to ensure that nodes do not get overloaded by too high resource requirements, in this case, a too high requirement for the ledger state. This is important as every new node has to download the state first before it can synchronize
- The ledger state can neither be pruned nor easily cleaned up. A feeless protocol makes it easy to inflate it
- Note that only value transactions use the ledger state database. Plain data messages don’t move funds and are therefore only stored in the tangle database
Mitigation on the IOTA mainnet
On the IOTA mainnet, the minimum amount to be sent is 1Mi. If you want to receive microtransactions, you need to create a dust allowance output, which functions the same as standard outputs but has a flag set that allows dust to be sent to this address. For each Mi, this dust output holds, 10 outputs <1Mi are accepted on this address, up to a maximum of 100. You can restock by making a transaction that spends all dust outputs into a single output, creating space for further micro-outputs.
However, this system has a few problems. First, a spammer could fill up your allowance with 1i outputs constantly. While it can be cleaned up by you, it can be hard to receive the microtransaction you want because any allowance is instantly filled. Furthermore, the network has to decide for each microtransaction if you are allowed to receive it. This requires total ordering and therefore does not work without the coordinator.
And on Shimmer?
On Shimmer, whoever creates an output has to pay a storage deposit in Shimmer tokens. As outputs can become more complex in the new system, the deposit is based on how much storage the output uses. Output sizes can differ a lot, it takes much more disk space to store an NFT than a simple SMR transaction, so NFTs require a higher deposit. When you spend the output, your deposit is fully refunded. Therefore, Shimmer stays feeless, every token you spend on storage is refunded when you free the storage again.
The deposit is made by simply adding some Shimmer tokens into the output. So if you want to send me an NFT, you also have to send me some of your SMR tokens to pay for the storage the output consumes in the ledger state database. The same goes for native token transfers. If you only send around SMR, those outputs usually have enough balance to back themselves.
To express it differently: You stake your Shimmer whenever you send them and the reward is disk space in the ledger state database until you spend the output. This also adds the use case to permanently store data in the ledger state, for example, DIDs. But to do so, you have to add some tokens to pay for the storage. Sounds like a fair deal, doesn’t it?
How the storage deposit is calculated
The storage requirement of an output is measured in virtual bytes (v_bytes). This is the case as we need to be able to find any output quickly without looking through the entire database. Like in a scientific book, this is done using an index that shows us where the output is. However, maintaining this index is much more costly, since it constantly has to be updated and adjusted to maintain high read and write speeds. Additionally, a larger index slows down the database, just like you would need longer to find what you are looking for if the index is 50 pages instead of 10. Therefore, writing an index (aka database key) is ten times more expensive than just writing data.
Because keys have a much higher weight, they play a major role in the v_byte size of an output. The outputID is needed to quickly find an output when somebody tries to spend it, as this is how you specify which output you want to spend. In the following case, this already makes up 80% of our storage deposit. Since the deposit cost is at 100 glow/v_byte, this results in the lowest amount we can send without requiring an additional deposit being 42,600 glow.
Further calculations
Let’s go back to the example from the beginning where I wanted to send a native token. So I choose a basic output (426 v_byte) and add the native token (+70). The size of the output would be 426+70=496 v_byte. At the cost of 100 glow/v_byte, I would have to put 49,600 glow into the output to fulfill the dust protection requirement. Which is exactly the amount firefly showed us in the beginning. Mystery solved!
With the new system, native token transfers and NFTs are microtransactions at well, they all need to carry some SMR around to pay their cost in the ledger state. If you want to send somebody a large NFT, you might have to add quite a few tokens. So how can you get these back?
The answer is simple, add a return condition. This condition makes it so that the recipient has to pay you your deposit (or a part of it) back. As he creates a new output, he now has to pay the deposit for the NFT by himself, possibly repeating this process if he wants to send the NFT again. To prevent this from being stalled indefinitely, also add an expiration condition. This will cause you to receive your money back in a reasonable time if your recipient does not respond. In my example, I have chosen to gift the storage deposit and not set an expiration time. By default, these two options are selected, increasing the storage deposit by 8000 to a total of 57,600 glow.
Let’s take the example below. Alice wants to send Bob a single glow. Instead of the token, Alice could also send a native token or an NFT. Let’s assume the output requires 1 SMR of deposit. Alice adds a return block to receive her 1 SMR back and an expiration block so Bob has to claim the glow within 3 days. If Bob spends the output, he has to make a transaction that refunds Alice the 1 SMR. Bob also has to ensure that he is fulfilling the dust requirements now, so he must use some SMR tokens out of his pocket.
If Bob does not respond, let’s say he was on vacation, it expires, causing Alice to regain ownership of the 1 SMR+1 glow output. After Bob returns, she could resend the tokens.
If the two are in regular exchange, they could also move to layer 2. Besides the obvious way of using a smart contract, Alice could also set up a payment channel with Bob using timelocks. I expect lots of microtransaction use cases to utilize layer 2, especially if they have to happen fast.
The new dust protection offers a well-regulated cost for writing data into the ledger state. As deposits are refundable, this creates an incentive for users to clean up the ledger database to have their tokens available. It also creates a great use case for the SMR token, making it mandatory to own if you want to keep permanent data inside the ledger.
But what if the value of the token rises
This is a major concern in the community, and it isn’t for nothing. A higher token price would cause deposits to become more expensive as the cost of writing data into the ledger state increases. In the future, we can always decide to decrease the cost per byte if, for example, SSDs get even cheaper and faster. While increasing it is theoretically possible, it creates some issues, such as already created outputs would not be affected by this, thus creating an incentive to not move these tokens to keep the lower price.
Ultimately, sharding is the only permanent solution to this problem. While the current system with currently used hardware could easily deal with 10 million users, it gets tricky when reaching the 3-digit millions. Also, not even 10.000 MPS will be enough in this system unless you make Solana-style requirements for validators (and even that might not be enough). It is just not possible to maintain a database of billions of users/devices with millions of messages in a second on a single machine in a decentralized way. While Shimmer’s technology does not have a fixed MPS limit, your hardware surely does.