SDBS #29: StealthExplore Performance Improvements
Over the last couple of weeks, I did a significant rewrite of the StealthExplore API to improve its performance, changing much of the implementation, but little of the API itself. In today’s blog post, we present a video where I explain and demonstrate these performance improvements.
This video is divided into two parts. In the first part, I discuss the aforementioned improvements. In the second part of the video, I discuss fundamental problems with delegated proof-of-stake (dPoS), exemplified by dramatic events with the Steem blockchain. I touch on how the potential for these problems inspired design decisions for Stealth’s high performance Junaeth consensus protocol, which allows 5 second block times. Below, I hope to clarify some points I tried to make in the video.
Video: StealthExplore Performance Improvements
Before watching today’s video, viewers are encouraged to read SDBS #28 and the watch accompanying video, where I give a first look at StealthExplore. I also encourage viewers to check out SDBS #25 where I introduce StealthExplore, and SDBS #27 where I discuss how a rich list exemplifies central challenges of infrastructure scalability.
Reducing Disk Access Allows a 3X Performance Improvement
In our last development blog post (SDBS #28) I gave a first look at the StealthExplore API, an application programming interface we are creating for infrastructure development. At one point in that video, I grabbed all 1511 transaction outputs for a specific address (S8SEXpbtdDtteypN7sBxK72VDACqA2ffjp). If you watch the video, you will see that I was a little surprised at how long it took, estimating “about two seconds”, which I felt was a long time.
It wasn’t two seconds, but 4.6 seconds, according to my timing the video with a stopwatch this evening. I mentioned in the video and in the blog post that I thought I could speed up this operation.
Over the last couple of weeks, I implemented many changes that ameliorated the biggest bottleneck, which is disk access. As I mention in today’s video, the previous process for retrieving this data was as follows:
- Reading from the key-value database (LevelDB) to get the transaction identifier
- Using the transaction identifier to read from LevelDB to get block indexing information
- Using that indexing information to read from the blockchain flat file to retrieve the entirety of the transaction data
My revisions to the implementation resulted in the much simpler and faster process of simply reading from LevelDB to obtain just the essential transactional information.
Although I seemed to be unimpressed when making today’s video, the measured performance improvements are actually quite remarkable. Access for 1600 outputs (for address S8knUucS6iFwAt1S8BmRpiTykqQfRLcape) was 1.6 seconds. This 1.6 second timing is the average of un-cached retrieval of three different sets of outputs with times of 1.0 sec, 2.2 sec, and 1.6 sec (outputs 1–1600, 1601–3200, and 3201–4800, respectively).
The improvement represented greater than a 3X performance acceleration, which perfectly mirrors the reduction in disk access (three reads for the old implementation versus one read for the new implementation). The math is 3.044X = 4.6 * 1600 / (1.6 * 1511), where the numbers of records retrieved (1511 and 1600) are included in the calculation.
As mentioned in the video, I believe it will be difficult to squeeze more performance improvements without resorting to some database trickery, like using batch reads from an optimized database. This type of approach would require large-scale database reorganizations on the physical drive, and may be necessary as adoption becomes global in scale. One way to achieve these reorganizations is to have a scheduled process reorganize the database to a new part of the physical drive, creating a new copy of the database, and once this reorganization is complete, the client would “hot swap” database files during execution. We are looking into such approaches to maximize performance in the future.
— — — — — — —
Steem and Emerging Problems with dPoS
In the second part of today’s video, I talk about ongoing undesirable developments with Steem, and how these developments epitomize problems with delegated proof-of-stake (dPoS). The relevance to Stealth is that Stealth uses block scheduling, a technique required for high performance block chains, such as those represented by the major dPoS coins (like EOS, Steem, and BitShares). Unlike dPoS coins, Stealth does not have block signing delegates, avoiding problems like those experienced by Steem (ongoing as I type this blog post).
In dPoS, block validators, called “witnesses”, are chosen by stakeholders, who vote with their stake. Ostensibly, witnesses in this system earn their reputation with the community. At any point a witness (who we will call Witness B) could gain favor with the community and displace Witness A if Witness B gets enough votes. Voting is quantified by the amount of stake behind those votes.
While great in theory, in practice, the security and integrity of dPoS is observably fragile because the original distribution of most dPoS coins is typically highly unfair. For example, practically all STEEM was mined over a one week period under conditions that some have described as dubious. Most of this STEEM was mined by the Steemit founders, and this stake has always had the potential to be the “de facto community”. Meaning the stake owned by Steemit could be utilized to completely control the blockchain by electing Steemit’s own set of witnesses, to the exclusion of the preferences of the greater community.
To summarize the events, Steemit, the company that founded and supported the Steem blockchain was recently sold to Justin Sun of the cryptocurrency Tron. This sale included Steemit’s stake in the Steem blockchain, mined under the conditions described above. For reasons that are not completely clear to me, the existing witnesses coordinated to attempt to lock Tron from its stake in the blockchain. In the video, I use the term “collude”, but this highly loaded word may be unfair to the witnesses.
In response, Justin Sun used Steemit’s stake as well as the stake of coin holders at various exchanges to vote in its own witnesses and completely take control of the blockchain.
To me, such whipsaw control, or even any control at all by a central entity or cabal of users, evidences fundamental and irresolvable flaws in dPoS. In the video, I discuss how competition helps an economy, but I don’t explain how the importance of competition relates to the flaws in dPoS.
The connection is that an unfair stake acquisition (such as the so-called Steemit “ninjamine”) is not competitive. This absence of competition means that selection of block validators is not competitive, and, as a result, reduces or even eliminates the integrity of block validation. In the present example block validators can be swapped according to the whims of founders and/or insiders to control a blockchain and even seize stake.
Additionally, and perhaps more disturbing, is that stake can be co-opted by exchanges. The impermanent nature of the dPoS vesting mechanism allows this to happen. Essentially, these exchanges have borrowed user funds (without asking the users) and presumably will pay those funds back by divesting them from the Steem blockchain vesting mechanism.
The Stealth Junaeth protocol avoids both problems of (1) lack of competition and (2) impermanence of vesting. First, potential validators of the Stealth blockchain engage in economic competition by purchasing Stealth on the market. The forthcoming switch to Junaeth has been broadcast by the team for a long time. Potential validators have had all that time to purchase Stealth in preparation to acquire a StealthNode from the blockchain. Also, Stealth was mined fairly according to the standards when it was launched. Second, vesting in Junaeth is permanent, taking the form of purchasing a StealthNode token from the blockchain. This purchase permanently destroys the expended XST. If exchanges want to control the Stealth blockchain under Junaeth, they will have to irreversibly take user funds, with no surefire way to pay those funds back. The need for exchanges to essentially steal from their own users greatly reduces the chances any exchange would dare to seize control of the Stealth blockchain with user funds.
— — — — — — —
Hondo
— — — — — — —
