Falling apart under pressure?

When I decided to [ANN] Veldspar, I certainly wasn’t expecting the mixed bunch of techies, miners & rogues to turn up so quickly. The number of miners has grown at quite an explosive rate recently.

And this has brought with it some problems, some quite significant problems.

Structural issues with the service

Initially the key index within the system was the token id “0009280E-000E621D-0001619C-000DDD7E-000CACBA-000096D3–0002946D-0000E4DF”. Which was fine for the first two million tokens, but after that the seed node started to time out handling requests, something that it had managed to cope with very well up until last weekend.

At only 1.2GB the blockchain file was hardly massive (although still larger than desired), but response times had reached almost 40 seconds in some instances, which is just crazy. So I assumed it would be an issue with the string index on the token identifier. But queries to lookup the existence of a token were taking around 35ms (for every single submission from a miner the node has to check in two databases that the token does not exist already in the blockchain, or the pending transactions table), which was pretty rapid, so the issue was clearly not that.

So regardless, I set about splitting the string up into numerical parts and added the 8 address columns into the database, after an hour of downtime and a painful population script it made little or no difference. So I started to look elsewhere for the issue, and found one.

It turns out, that the queries to sync the wallets are desperately expensive on the server (I knew they would be, but not to this level) as they have to lookup the status of every single token which has ever been referenced by the address (either IN or OUT), and as the number of records grew, the longer those queries got. In addition to that issue it became apparent that the token identifier was not suitable to be a key index within the system, because it starts with “<ore>-<algo>-<value>” before the address, there existed the premise that an algorithm change would essentially mean that tokens could be registered again at the same ore locations, which economically would be like a huge release of available tokens again. So I added an address column which is only the end part of the token, which added around 1GB to the DB and also made little difference to the performance issue.

There is also in inefficiency within SQLite string indexer, which is really shown up when a few million records all start with the same values. I swapped it out for a custom job, but the speed just was not there still.

How are we fixing these issues?

Well there isn’t much else to do except to store the ledger data in it’s component values, which would allow us to use numerical indexes which are more efficient and will shrink the query times and size of the database considerably. In the last release we already disconnected the submission from the DB activity, so all miners submissions are instantly accepted and a process then churns through, validating the finds a second or so later but with far fewer exclusive locks on the tables, this has cleared the backlogs and brought about some much needed stability (so I could go away for a couple of days).

This will become critical for transactions, as they will need to be replicated to all nodes as fast as possible. So if there are delays of 40 seconds writing a single token, and you actually wish to re-allocate say 8k tokens to another address, that is going to cause a pretty serious issue, very quickly, transactions really need to be validated within 0.5 seconds on any node.

So what about the future?

As we start to think about the move from Alpha to Beta or even Release, we have to be confident that all the right decisions and compromises have been made. I am broadly happy with how the ore mining system works in principal, I think we probably all need to have a discussion and poll on the economics side of things. My personal feelings are that 1MB ore is fine (maybe just a bit less), but we should use only 4x addresses which would reduce dramatically the supply and make all the system queries infinitely quicker.

So what are the actual issues?

Problem: Blockchain size is still too large to be sustainable if this is to become a major coin system.
Solution: Slow the emission further to stop new token generation from swamping the service. Increase the block time to say 10 mins. Reduce the address space of the service to be only 4x UInt32 (or 3x) addresses. This will dramatically reduce supply, and increase depletion. Allow only one find per specific ore location.

Problem: Wallet syncing is slow and really hurts the network as it grows.
Solution: Wallets will now have to “playback” the blockchain as every other coin does. If we offload the work to the wallets then the network doesn’t have to shoulder the burden of people leaving their wallets open all the time and all the stress that places onto the nodes.

Problem: PoW algo needs to be shored up and made faster to validate. Until recently the nodes would spend anything up to 700ms validating a token. Now it’s around 50ms which would be almost fine in production unless the network got far busier.
Solution: The miner needs to send what it found (the bean value) with the registration to stop the network having to calculate the find, so it can be quickly validated as opposed to almost the same PoW carried out.

Problem: Wallets are slow to load and save
Solution: Defo swapping a wallet file out to a database for speed, then just store the seed encrypted instead of the whole file.

The Future (and the Past)

Well we have quite the community already, much of that is thanks to TurtleCoin, which should act as a shining beacon of how to use a vibrant and productive community to really make something special happen. We are in many ways lucky to have some shared members and more than a cheeky hand in the ideas cookie jar. If you are not familiar with TurtleCoin, you should check it out. They started as a Fork, but are so much more than that now. And with the collection of keen devs they have over there you would be foolish to dismiss them as another “Shit” coin, as they are really committed to driving away from the original codebase and the changes made already have seen some very old and crusty code modernised (something that even the big coins struggle from).

Quite a bit of this project stems from ideas myself and RockSteady (from TurtleCoin) had whilst we were thinking about our own CryptoNote fork, having stopped work on an Aeon rebase due to, well, reasons. We would communicate ideas on how we could make crypto slightly fairer and open to all, and those principals are the defining factor of this network.

So credit does have to be passed over to the boys over the fence because lots of these ideas, and many to come, were a community effort of shooting concepts around on the discord channels and I suspect that will pay a huge part when it comes to sorting out the networking side of Veldspar.

But now for the future, and that stems around picking a few people to help out on the Discord, asking again for devs and people with other skills to help out with GitHub improvements and most importantly guides (an area we are sorely lacking in).

So what next?

Next is some polls, I want feedback on direction, because after the next phase we will be pretty locked in. Those will include, but not be limited to:

1. Denominations, 0.xx, 0.xxx, 0.xxxxxxxxxxxxxx (satoshi madness style!). What would people prefer, what compelling reasons are there to change?
2. Block speed, does anyone care if it’s not even related to mining?
3. Economy, what should the total supply be? What would be a nice balance between liquidity and scarcity. This question would also encompass addressing size, which I feel should be smaller for a more compact blockchain.

What happens when we transition to Beta or Release?

Well, primarily, all the software for mining, wallets, nodes etc is all ready well in advance so people can be ready to mine/transact right away.

Then we pick a day to turn the service off, reset the blockchain (again) based on the economics and structure we have decided as a community, and then I will import/regrade everyones current tokens into the new chain (having done this already, I know where I went wrong last time).

Then everyone is on the new chain, with all their tokens (new addresses) and we start again, but this time with people running their own nodes, and able to transfer between other Veldspar users.

Finally

Thanks for reading, dev has been a bit slow this week because I was away for a few days for a rest. But things will be busy this week, and progress will be made. I will need to catch up with CodIsAFish with regards to his C# windows wallet, as we will need to make sure we operate from the same API calls in time for any major changes.

Thanks

Napoleon