Testing the Ultra blockchain

Ultra’s blockchain team has had an extremely busy and exciting couple of months, with our core focus being on benchmarking and optimizing our blockchain code base.

EOSIO is excellent open-source software that we are privileged to use as a foundation for Ultra’s blockchain.

As we discussed in our previous article, there have been many improvements and updates made on top of the vanilla version of EOSIO by our team. These changes needed to be validated as they were made with the intent of providing mass-market ready solutions that EOSIO can’t provide such as: free account creation and a powerful user resource management capable of providing hassle-free truly-free blockchain transactions.

We are aiming for rock solid and this has required in depth research and benchmarking.

Providing the best tech through rigorous testing

Ensuring that these features are correctly implemented and well-integrated requires that we perform extensive internal testing. This benchmarking provides us with validation that what we end up using is bulletproof and it gives us the confidence to deploy a platform where the value of your assets can be trusted.

The usual methodologies for benchmarking EOSIO deployments proved to be inadequate for our needs. The tools available to us simply did not allow us to push enough transactions to the network to fill our unique queuing mechanism and saturate the network. We needed bigger guns to test with, so we built them.

The first of these innovations is a custom EOSIO plugin for benchmarking that allows us to perform transactions at an unprecedented rate. We can push data to our chain faster than any competitor has tried so far, resulting in unique stressors that allow us to simulate real world applications. We ported this plugin to vanilla EOSIO allowing us to make an apples to apples comparison between the two deployments. Our goal has been to run the same benchmarks on the same hardware for both EOSIO and Ultra’s blockchain so that we can accurately gauge how our code stacks up to the competition.

To support the benchmarking research, we have also implemented custom APIs which allow us to prepare and then push transactions that automatically generate test accounts, set balances for them and manage account resource staking (CPU/NET vs. Ultra Power). Automating these features accurately provides a foundation that we can test on which reflects what we expect our network will encounter as we roll out our gaming platform towards the end of 2020.

Beyond this, we also built a way to generate reports based on the benchmarks that we ran for different feature branches. This allowed us to share this data internally and make intelligent decisions about what technical improvements we should make. This has been exceptionally useful for debugging unexpected bumps in the road that we encountered during the testing process.

With many of these benchmarks completed, we’ve been able to scientifically select the optimal requirements for our Block Producer’s infrastructure that ensures we can push an enormous amount of transactions through the Ultra blockchain network with confidence. It also has allowed us to deeply examine the changes we have made that there are no unexpected surprises which may have slipped under our radar due to the extensive changes we have made to the original system protocols.

Benchmarking methodology

For each test we raise a new, fresh chain on a series of bare metal servers directly under Ultra’s control. Then, using our custom APIs to create and manage them, we set up 100,000 accounts with selected resource staking distributions.

There are three resource distribution types that we have tested: normal, uniform, and exponential, which covers the expected range of network configurations that we are likely to experience plus possible edge cases.

Each distribution has three versions of mismatch between staking and transaction demand. This gives us nine cycles of testing (three distributions with three versions), which we prepare transactions for.

During a cycle there are 1.7 million transactions that are sent, which are split across the 100,000 accounts. These transactions are sent at a static rate, e.g. 10k transactions every 500ms. We use eosio.token contract and transfer transactions as they are the industry standard.

The results are logged and we share the reports internally as part of our larger discussion regarding quality, preparedness, and implementation.

We are trying to take a holistic approach to our benchmarking so that we can confirm the stability and integrity of the release that we’d like to build our public Testnet (and Mainnet!) on.

Benchmarking achievements

The EOS Mainnet is currently the largest deployment of EOSIO. There are dozens of high-quality Block Producers who have developed excellent standards for infrastructure and network schema. The EOS Mainnet currently maxes out at around 4000 transactions per second.

There was an enterprising group of BPs who pushed the Jungle Testnet (EOS) to over 9100 TPS, but had micro forking problems. The same team did achieve a stable peak of 6977 TPS on Jungle.

Ultra’s blockchain does better.

EOSIO Vanilla Results

As of today we are currently managing to push through in excess of 10,636 peak TPS on a vanilla deployment of EOSIO using our powerful EOSIO benchmarking plugin, which we’ve back-ported, and using our specialized infrastructure.

Ultra Blockchain Results

For the Ultra blockchain, with our new queuing mechanism and all the other bells and whistles we have developed, we are pushing more than 11,610 peak TPS. We are currently experimenting with alternative configurations of settings and hardware and we hope to increase this threshold even further.

Further Testing

Of course, peak TPS is not the only thing that we’re interested in as average TPS speaks volumes about how consistent the throughput is. This directly translates to how well the network handles its capacity and ensures that your transaction gets through in a timely fashion regardless of the network situation. For an enterprise-grade network, we understand that consistency is king.

Here you can see the sustained number of transactions that can be continually pushed through for both Vanilla EOSIO and the Ultra Blockchain. The Ultra blockchain is highly consistent and excellent at pushing through as many transactions as the network can handle.

Average TPS Results

For the vanilla EOSIO deployment, we have benchmarked 6930 average TPS. Ultra pushes a substantially higher 9514 average TPS, in other words in our tests, Ultra executes on average 31% more transactions than EOSIO.

Ensuring Transaction Success

One of the other things that we have been working on is stress testing the network over time, and checking the failed number of transactions depending on the mismatch between an account’s transaction demand and its remaining staked resources.

This last set of tests is particularly important as any transaction that is failed directly impacts users and developers. Ultra’s variant currently shows between 1–2% failed transactions, where the vanilla EOSIO shows significantly more with between 30–50% failure rates. Our queuing mechanism works very well.

Our Conclusions

We’ve successfully extended the EOSIO software stack according to our business requirements. They are:

1. Free transactions for users that need them
2. Fast transactions for our developers
3. A secure queuing mechanism that can prevent spamming of the network

Our extensive benchmarking and testing has shown that our improvements have not had a negative impact on the stability or the speed of the network.

The Ultra’s blockchain has a higher peak TPS, a higher average TPS, and a more consistent throughput even with an enormous amount of transactions thrown at it.

Our vision for Ultra is one of a mass-market ready, enterprise grade network that developers can trust to deploy to and that users can easily use. We’re genuinely excited by the progress that we have made, and look forward to the launch of the Mainnet .

If you are a game developer and want to receive access to the Ultra blockchain, please register here, we’ll be in touch.

About Ultra

Ultra is the first entertainment platform providing all key games industry services under a single roof, accessible through a single login.

Built around our PC games distribution store, Ultra Games, our platform will provide access to countless centralized and decentralized services: Discover, buy, play and sell your games and in-game items, watch live-streaming feeds, interact with your favorite influencers, participate in contests, compete in tournaments, and much more.

Ultra has been built to provide endless value for players, a fair playing ground for developers, and a whole new world of opportunities for the games industry.

For more information, visit ultra.io and onultra.io and follow along on Twitter, YouTube, Telegram, and Discord.