That’s how we broke Jungle Testnet twice in a row, eleven days before 1.0 #EOSIO Release

EOS Metal
Coinmonks
6 min readJun 18, 2018

--

FIRST: This Article doesn´t have any relation with the Pause of EOS Mainnet on 6/16/2018. This Article covers the controlled Stress Test in Jungle Testnet on 5/20/2018.

May 20 2018 — Dawn 4.0 EOS Version (11 days before releasing 1.0 version)

team, an EOS BP Candidate, has actively participated in Jungle Testnet (the most important testnet before EOS Mainnet Launch) with our node “mosquitometa” (64GB of RAM and 8 cores). Our purpose was to keep up to date with all the news related to the software, and of course we wanted to push the network to the limit to know how far it could withstand.

We knew that with our more than 20 years of experience in critical systems, we could do something good for the community.

Jungle Testnet is a wonderful place for experimenting and learn about EOSIO Software. This testnet is administered by Cryptolions, mainly Bohdan as System Administrator, with a very active Telegram Channel and a lot of very goodies for EOSIO Software in their Github.

Our first goal was to know, how many transactions per second could be processed by Testnet nodes, and monitoring in real time the status of all nodes, using Testnet Monitor (another great tool from Cryptolions). Todo it, we used a simple Linux script that spammed random transactions to the network, using HTTP endpoints of every node. To start the stress test, we first needed staked tokens to use the resources of the Jungle testnet, so we requested it to Bohdan:

Bohdan transferred 10Millions to our account
With the staked tokens, we have plenty of CPU and Net Bandwidth to execute transactions in Jungle Testnet
Some explanations from about how limits works

Of course! other BPs in Jungle testnet joined the TRX party

and the party started!

We used https://github.com/CryptoLions/TX-test-sender

a few minutes after start, we got the first results (tons of red)

most nodes were flooded, and other Bps wanted join to the party too

And the Nightmare, started…

The effect was nodeos service consuming 100% CPU
We informed about our test results

Some explanations about resource limits

In this case, was possible recovered Jungle Testnet, and We started again but with a heavier Stress Test issuing transactions with a long memo of 20k

and the Nightmare started again….

A few minutes later, all nodes were flooded and finally Jungle Testnet crashed

Alfter request of Bohdan, We stopped the flood

Jungle Testnet died, without possibilities of recover

and

helped us

New tools for recovery were created

Next day, 24 hours since Jungle Testnet death and after many recovering attempts, Blockone analized the block logs and suggested to restart the chain

and super Bohdan started a new chain

All BPs had to rejoin

Of course, next step after Jungle Testnet was launched again, and our node mosquitometa was back in business and producing blocks, We wanted to verify that what happened the previous day was not fortuitous so we started a new stress test :)

First, only my node was overloaded

and after about two hours, the nightmare appears again…

Send TRX with Long Memo is confirmed as the root case of the problems
All nodes were unsynced

and Daniel came up with new Improvements

Blockone, adds a new P2P protocol called bnet for better performance

and a fix for MEMO size bug in Transactions

limit memo from 40k to 256 bytes

A few hours later, after relaunch again Jungle Testnet from scratch

And we detected another issue in the chain

New tag was made and all nodes were updated fixing different issues (same day!)

And we started again Stresstesting with the new Tag, with very good results:

662 Transactions per second, without problems

All was fine, and we learned some lessons

and some work to do

Thanks to all people participating in Jungle Testnet and their patience and hard work in those two days (Without all these professionals, we would never have been able to test it), and thanks to B1 for fastest support that I’ve seen in my 20 years of experience.

After this intense work without a rest, we was very happy for contribute to this very important cause.

It’s very important that all those small teams that have been working hard in testnets prior to the mainnet launch get enough votes to be able to survive. Those teams don’t have huge marketing machines behind them but they have really good professionals in the technical field.

We want to keep contributing to the community, helping to create a better and more secure mainnet for EOS, but for that we need your vote. With your vote we can get rewards to keep collaborating, testing real dapps in Mainnet, making disaster recovery procedures, and make EOS mainnet bigger and better.

We don´t have a Marketing team, We are Technical System administrators, and we work in real production non-stop environments everyday. We can contribute a lot as we proved in Jungle. If you have questions, you can drop by our Telegram Channel: https://t.me/EOSMetal and EOS Spanish Channel: https://t.me/eosEs.

And of course, you can vote for us! to our BP Candidature “eosmetaliobp”

Sincerely, your

Team

--

--

EOS Metal
Coinmonks

We are a Block Producer Candidate for #EOSIO Software