Challenges in Upgrading a Blockchain
Our team at Consensus just released the latest version for the Sentient Hub, with a new daemon, a new mining pool, a new miner, and a new logo. For those unfamiliar with Sentient (SEN), or the application our blockchain is to be used for in general, there are some great articles to dive into here, here, and here. To the contrary, this article is meant to be more of an educational examination of some of the specific technical challenges we encountered.
Each section gives a brief overview of the problem for more casual readers, then dives deeper into the nitty gritty details that can be of interest for more technically inclined blockchain enthusiasts or developers.
To understand the significance of the current SEN update from a technical perspective, one must first understand the origin of SEN, and what particular set of problems it had to face. SEN was developed from a fork of Sia, a popular blockchain designed for a decentralized storage platform written in Go.
Using an existing blockchain has the benefit of not having to reinvent the wheel, however this doesn’t come without its own set of problems.
These range from high level design, to lower level implementation complexities. In particular, we will explore adding versioning support to blocks, fixing a broken mining pool, and adapting a heavily optimized GPU miner to support the changes.
Adding Versioning Support
In most blockchain networks, there is the idea that if members of the community agree to make changes, and there is a particular threshold of support for a given change, then the network can upgrade its functionality in a democratic fashion.
This type of change requires careful protocol design, in particular, the nodes all need to know which code they are running, ie version. Bitcoin, for instance, specifies the block version in its block header. Unfortunately, the codebase inherited from Sia, and thus also the initial Sentient release did not have a version field, and provided no clear way for the community to add changes to the network.
The Chicken or the Egg Problem (Catch-22):
We need a version field to add changes to the network, but we need to add changes to the network to add a version field.
This problem was non-trivial to solve, and required meticulous planning and testing to make sure we do not corrupt the entire chain in doing so. Needless to say, this is what is known as a “hard fork”, where once the change happens, there is no compatibility with nodes running old code. Along with adding a version field, we have also added support for version proposals on the blockchain, that miners can accept or reject, to support democratic changes to the network.
We can see that the original blocks and block headers had the following structure:
Whereas once the version upgrade occurs it will have the following structure:
These additions will of course make the block headers slightly larger (up from 80 bits to 100 bits in size) which necessitated an upgrade to the mining code as well.
One of the more complicated aspects of upgrading the blockchain is how to ensure that all of the nodes upgrade at the same time (on the same block), in sync. This is done with rules related to the Version Proposals. When a new version is ready for consideration, we will post the code for it in our github repository and create a Version Proposal transaction on the blockchain.
Miners who support upgrading to this new code version can express their support using the Supports field when they submit mined blocks to the network, which they can do most easily by just installing the new code. A Version Proposal has several fields that control for how long it is active (a deadline and timeout) and what percentage of blocks over most recent blocks (the activation threshold and window respectively) must have expressed support.
When a transaction with a version proposal is mined into a block, all nodes on the network keep track of these parameters and, if conditions are met, will upgrade together in sync. This will allow us to use more soft forks, as opposed to hard forks, when we want to add new features to SEN going forward.
Adding this functionality was not without its complexities. As an example of one of the many challenges we overcame, consider the point highlighted above about how the block header structure and size had to change. In order for nodes running our upgraded code to run on the blockchain right now, when the majority of the existing nodes are running earlier versions, and all of the blocks being produced are the original structure/size, our new code must be able to parse and validate blocks of either sort, and distinguish between them. This way, users can upgrade to the new code without dropping out of the existing blockchain.
However, since there is currently no field in the original block structure by which existing nodes can indicate that they have upgraded to the new code, we devised a solution where this can be encoded in the Arbitrary Data field of transactions without interfering with any of the other information in the block. With this trick, we are able to use a Version Proposal even now to help upgrade to the new versioned block structure, allowing us to know how many miners have installed the new code and when there is enough hash power to support the new block structure.
Fixing the Mining Pool
The current sentient mining pool was forked from another repository, which supported mining on a stratum-like protocol. When we first introduced the pool, we decided not to stray too far from the original implementation at first while we observed how it was adopted in our network. This meant that not much effort was made to adapt it to SEN, nor was design put into scalability. With the other upgrades to SEN that are concurrently happening, now is the time to do so. Given the history of the mining pool codebase and its lack of scalability, it had been clear for a while that it would need a complete rewrite.
For example, many users noted to us that the mining pool would frequently crash, or go on and offline several times a day at certain points. There were many minor issues, but fundamentally there was a flaw with the design of the entire database, which led to unnecessary data being stored, and needlessly complex queries carrying out seemingly basic functions.
On a high level, the mining pool allows groups of users to combine their hashpower to mine blocks more efficiently, and share the profits based on the proportion of hashpower. This way, even users with slower machines can get consistent rewards, without relying on luck to fortuitously mine an entire block, which becomes less and less likely as the network grows. Instead, once they reach a certain target, they submit it to the mining pool, and even though it might not be good enough to mine an entire block, the share still establishes that they have been mining for the pool and so get a small reward.
When someone in the pool does meet the target to find a block, the block reward is distributed to all members based on how many shares they have earned. In the initial design, the pool would record every share ever submitted by any user in the database forever. An average miner, without any special equipment can usually earn a share every few seconds, and over a short period of time, this table quickly grew to tens of millions of rows leading to slow query backlogs and even deadlocks crashing the entire pool on certain repetitive queries. Initially, we did the first thing we learned in our elementary school database classes to fix this: add an index.
We then began creating backups, and deleting outdated table entries, but this was also only a band-aid solution. Even though error logs and crash reports were cryptic, it could always be traced back to the shares table. But where exactly?
One of the first things most people notice in the Sentient Hub is the pretty graphs showing their hashrates and shares earned over time. There’s something quite amusing about clicking through the different time intervals and watching the Javascript dynamically draw the picture. Under the hood, it was anything but pretty. Note, hashrate depends on share difficulty and frequency, and we have several million shares that need to constantly be queried (while simultaneously being written to from active miners).
Case study: To select historics to draw the graph for the hashrate of a particular worker the original query was:
This monstrosity was called every time a user wanted to see a graph, and there were almost a dozen different queries of a similar form for various API calls. Let’s think about that — every time a user clicks through a different graph, the database does a summation over a double join nested in a select query from a table with tens of millions of rows.
An individual user, cycling through the different graphs can almost single-handedly DoS the mining pool.
Using this bit of insight, we realized that the shares table had to go. Any info related to the graphs would go into its own separate table, calculated once, and discarded once it became outdated, effectively capping the number of entries to 800 per worker (4 possible time intervals, up to 200 most recent graph points) assuming their miner is on nonstop. The final query for the same API call becomes:
Considering the shares are only relevant for payouts and generating the graphs, we now increment an account’s balance directly when a share is submitted. Regarding details relevant to the graph, we only hold on to the share stats until they can be consolidated into a single graph entry for a given time interval, and we effectively free our database from indexing tens of millions of rows, and speed up queries over 180 000 times!
Adapting the GPU Miner
As mentioned in the first section, the addition of a Version and Supports field to the block headers is not a change to be taken lightly. For the miner, on the surface it seems arbitrary, considering the miner is supposed to grind on an incomplete block header over and over until it reaches a certain target, it should not really care about the details of the data inside. Unfortunately, a change in the block header structure means that miners need to grind on different data, necessitating a change to the optimized GPU kernel, and the protocol for sending and receiving these headers. Marshalling and unmarshalling was also tricky here, in part because of the way the stratum protocol was implemented in the Sia solution that was originally adapted to SEN. This relates to the order of struct fields and the little/big endianness of integers and byte arrays being transmitted.
The miner is written mostly in Go, communicating with the mining pool or creating blocks for work itself for standalone mining with code mirroring what can be found in the sentient-network or sen-node repositories, however the actual mining is done using OpenCL bindings and a heavily optimized and slightly modified blake-2b hashing kernel written in C. We struggled trying to figure out why the exact same code gave the correct hash for the original block headers, and consistently returned invalid solutions for the new block headers.
Recall that the old headers were 80 bits and the new headers are 100 bits. Consider the fact that our byte arrays in Go are big-endian, and the GPU using OpenCL is little-endian. When writing an 80-bit array, we write 10 full bytes, keeping in mind each byte has bits in reverse endianness. When writing 100 bits, we have 12 full bytes and 4 extra bits at the end that are not so easily copied as done before:
Instead, because of limitations in the OpenCL implementation of C related to byte alignment, we had to use some real C magic to get them in the right spot:
In theory, this should have fixed everything. The hash would finally be correct since every bit was in its correct spot. We wouldn’t get off so easily however, and there was one more hidden landmine — pun intended.
Meticulously, line by line, we went through the blake-2b pseudocode, trying to trace how on earth the exact same code worked for the old un-versioned blocks but not for the new versioned ones.
We went through every AND, OR, and XOR for each matrix rotation, baffled by how an un-versioned header works, and a versioned header doesn’t work.
As it turned out, we overlooked the step that determines the length of an input. During the initialization phase of the hash function, we mix key size and the desired hash length. However since we are dealing with heavily optimized code that will always calculate a hash with the same length key, the length is baked into the constants of the blake-2b matrix. This meant this:
Had to be changed to:
Spotting the difference is an exercise left for the reader.
Conclusions:
The work done this year was tedious, but necessary for the foundation of a strong blockchain. We now have a functioning mining pool that can support an arbitrary number of miners to grow and strengthen the sentient network. Furthermore we have the capabilities to allow users to vote on proposals to changes to the network in a democratic way based on hashpower.
This paves the way forward for deployment of more interesting capabilities, short term being the release of digital identity and anonymous/verifiable voting using data aggregation based on discrete logarithms, developed but still in testing. Long term, we are planning the economy structure of a decentralized machine learning marketplace.
This article is by no means an exhaustive analysis of our research and development, but perhaps readers of all technical skill levels could learn something about blockchain technology and some of the peculiar challenges involved.
If you are looking to buy SEN, you can find it listed on Coinbene exchange, or start mining today with our newly upgraded Sentient Hub
