Beep!, Beep!, Beep! 6:45am on November 5th. I’m tired — exhausted really. I’d driven back from Vegas’ World Crypto Con and got home at about 12:45 in the morning, and didn’t sleep because of the adrenaline rush. Before going to bed, I decided to modify my issuebulk python script to trigger on block 435456 — the activation block for asset creation. I tested it a few times on different blocks to make sure it worked. Every second it checked for block 435456 and then allowed the remainder of the script to run.
Stopping the alarm, I climbed out of bed, with only four hours of sleep. I connected to Discord, and connected with the team. Everyone was ready to run their script to get desirable asset names. As each block ticked by, the anticipation grew. A scrolling counter on my screen showing that the script was alive and checking every second for the first block that would accept assets.
In the days before, the software had counted blocks to make sure that 1814 of 2016 blocks were mined with the new software. We can write code, but we can’t force anyone to run the latest and greatest. We know that it’s really the mining community that chooses the software that constructs blocks to add to the Ravencoin blockchain. There was really no good reason miners wouldn’t run the new software as it was an improvement and added long anticipated features. What I didn’t expect was a rogue mining pool with around twenty percent of the hash power to run the older non-asset-aware software. I found out from conference attendees that this group was holding up asset activation and we knew almost nothing about them. The community named them RSa5 which was the first four letters of the address to which they were mining.
This rogue miner held up the activation because they were mining 20% of the blocks with older software, only 80% were being mined by the latest software but in order to activate the new features, 90% was needed. The vote failed and the software rolled over to the next cycle. Each cycle is 2016 blocks — inherited from Bitcoin — but only 1.4 days, instead of bitcoin’s 7 days because of Ravencoin’s faster block times. The next cycle started, and it didn’t look good, RSa5 was still blocking activation. But then the mining community rallied. I didn’t see all of it, but hash power nearly doubled and dropped RSa5 down to nearly 10%. The battle was on. Community members rallied and rented NiceHash to force RSa5 out. It was a battle royale, with the activation winning by a hair.
The net result, and a fun side-effect of the battle, is that it forced the activation date to November 5th, which has its own mythology around freedom from oppression.
So there I am, bleary-eyed, and watching seconds scroll by when block 435456 hits. Crap! Failure after failure scroll by in a blur. “Error: Please enter the wallet passphrase with walletpassphrase first” over and over. Oh yeah, I’m on mainnet, I need to enter a my passphrase. I quickly enter my passphrase and tell it to be active for six minutes, and run the script again. Dust!?! What is this? Why are my asset issuances failing? Everything worked perfectly on testnet.
I check with the team. They’re seeing the same thing. I check with the Discord chatter. Same thing. Oh no! We tripped on our shoelaces leaving the starting block. It seems nobody can create assets, and everyone is seeing the same error.
With a sick feeling in my stomach, I start searching the code for the error. Other team members did the same. We figure out that there’s an additional check for a “Standard” transaction that isn’t on testnet, and is only on mainnet so it wasn’t caught in all of our testing, nor in our multiple dry-run attempts of asset activation on testnet4, testnet5, and testnet6.
After determining that this isn’t an excessive load problem, but a code issue, we started discussing the options. One leading option is to fix the code, distribute the binaries, and set a new activation date for a fair launch. But once we know the problem, we realize two things. One, this is a problem with transaction submission only and not a problem with the consensus rules. And two, since the consensus rules are active, the network would accept a mined block with an asset in it. We were discussing the ramifications of this and had come to the conclusion that if a solo miner or mining pool constructed their own block, then the network would accept it and that miner could sell exclusive access to creating assets.
A token named VOTE suddenly showed up on the network. This was no longer an academic discussion. The miner or mining pool that created this token could create any asset they desired without any competition. They could even sell their exclusive access to others. This was the worst case scenario. Everything had been designed for fairness, including the way asset names propagate on the network. Even the way asset name collisions resolved was designed to be fair. Not the highest fee, or fastest network, but first to each node, with the mining pools being connected to random nodes and therefore random selection into the mined block. This resolution really only applies when the same asset is registered in the same minute, but we had worked really hard to make this a fair launch, and this was anything but fair.
Thankfully, the brilliant miner that figured this out, is also a gem of an individual and didn’t offer a profitable pay-to-play solution.
But, he had fired the starting pistol, and our strategy instantly had to adapt. New strategy. Post the code, explain the situation, build and distribute the binaries, and don’t register any of our assets until others have the means to do so. Why wait to register our assets? Because it was the right thing to do.
The new problem we were dealing with is that only miners with the new code will accept assets into their mempool. The mempool is a bucket of transactions waiting to be added to a block. Some of the miners were online, and some were probably working their day jobs.
The only good news was that the RVN network was still functioning. Blocks were being created on schedule. But, it was as if there was a bouncer at every door at every node saying “Ya got an asset — NO ENTRY FOR YOU!” We needed lots more nodes with friendlier bouncers.
We watched as some mining pools updated the software to 2.1.2, and then downgraded to 2.1.1. What is happening?!? Why would they downgrade? It turns out a rogue transaction with bad inputs had snuck past the bouncer, or had gotten in and then its predecessor transaction had been orphaned. This prevented miners from creating blocks, and the miners had rightly downgraded and kept creating profitable non-asset blocks. It took some time to figure this out, but we let the miners know how to resolve the issue.
Then the network started processing asset transactions through a limited set of nodes. An asset creation transaction had to propagate only among the updated 2.1.2 nodes, and wind its way to a mining pool node running the updated software. Finally, a batch of transactions made it into a block. The network slowly increased in capacity, and soon the network was healthy enough to handle all the requests. Not every user has upgraded to 2.1.2, but the chain isn’t fragmented, and the network is operating smoothly.
I didn’t get all the assets I wanted, but I was gifted the ownership token for TRON and TRON_BLACK later that day, and even sent a GRATITUDE token. Thank you, you know who you are!
Medici didn’t get all the assets they wanted, not because they couldn’t, but because they didn’t. It was the right thing to do.
Remember, remember, the 5th of November. I know I will. It was one intense day!!!