PERL Clicker Maintenance
16,987 smart contract calls per second, 487 concurrent players — game and challenge down for maintenance, 5,555 PERLs to be distributed.
Background
The initial demo launch on 28th August 2019 gave the community a chance to try out PERL Clicker.
PERL Clicker is a game that was developed to help us stress-test our network under extreme load; allowing just a small number of users to create a large amount of tracks.
With just 100 users, PERL Clicker could generate up to 10,000 transactions per second.
Following the launch, we collected extremely valuable feedback which has allowed us to make adjustments/improvements to PERL Clicker’s gameplay, alongside initial feedback as to how we could better cater Wavelet to scale with respect to transaction-heavy DApps.
Given the positive responses from our players, we then decided to take a step further to introduce an in-game challenge where players may earn a chance to win real $PERLs competing against other players.
Thus, the #PERLClickerChallenge was born on September 6th, 2019.
Scenario
Within the span of the first few hours the #PERLClickerChallenge was launched, we were taken aback:
16,987 smart contract calls were happening per second, with 487 players concurrently tapping all at once!
With the Wavelet testnet running on full-gear with each node encompassing just 4GB of RAM and 2vCPUs, this was shocking:
In no way throughout the challenge did the testnet go down for even a second in spite of all of this transactional load produced by PERL Clicker.
However, one thing did bust, which we did not expect.
As our team tightly monitored the internal state of the smart contract governing PERL Clicker, it appears that the PERL Clicker smart contract used 86x more disk space than needed for storing every player's state!
Tracing back to the root of the problem, we noticed:
The wee_alloc memory allocator, which is greatly trusted and used for a large proportion of all Rust (WebAssembly) programs, allocated significantly more memory than necessary.
In particular, this implicated that Rust (WebAssembly) Wavelet smart contracts which trustily utilized wee_alloc, under extremely high loads, allocated significantly more memory than necessary.
To prevent any possibly-exponential consumption of resources within our testnet, we immediately went to action and have taken down the PERL Clicker game and the challenge.
We had manually firewalled all of Wavelet’s testnet nodes from accepting further smart contract calls to PERL Clicker on September 6th, 2019 at 12:37 AM SGT while we worked away on the problem.
Technical Report
To address a quick technical fix, we have made by default that all WebAssembly (Rust) smart contracts on Wavelet will no longer use the wee_alloc memory allocator.
In the following commit, we have swapped away from utilizing the wee_alloc memory allocator to the standardized dlmalloc memory allocator: https://github.com/perlin-network/smart-contract-rs/commit/f2a070f46ab292410f1229b6e929cb25fa5e73dc
For those interested to still utilize wee_alloc, a simple addition of the code provided above would configure your Rust smart contract to use the very same memory allocation model.
An issue was referenced as well to the wee_alloc repository in regards to the extraneous memory allocations done.
Lastly, two members from our core development team have been assigned to figuring out the root cause for wee_alloc’s extraneous allocation behavior with respect to PERL Clicker’s smart contract.
All fixes we make will be contributed back to wee_alloc, given that it is an open-source project.
Reparations
First and foremost, we apologize for the inconvenience and the delay in releasing this announcement. We wanted to be absolutely sure of the technical root cause of the problem, and of our solutions to the problem.
That being said, we don’t want to let all of those passionate clicks that were already made to go to waste.
To show our sincere gratitude for all of the efforts made for the bugs found and feedback provided by the community during the launch:
We will be distributing 5,555 PERLs to the top 300 players during the #PERLClickerChallenge that submit their info!
Upon the halting of both the game and competition, a snapshot of all player's data was made on the 6th of September 2019 at 12:37 am SGT.
All we need from you are two things:
- Your Binance Referral ID.
- Your Wavelet TESTNET Private Key.
As shown below, your Binance Referral ID is an 8-digit numerical ID that can be found by logging in to: https://www.binance.com/invite.html
As shown below, your Wavelet TESTNET private key can be found directly within the login UI of the game: https://perlin-network.github.io/clicker
With your Referral ID and Private Key, fill out the form here: https://forms.gle/97jLZXjmn3NcoE987
There will be hard cutoff date on September 22nd, 2019 12:00AM SGT in which the form will be closed off. All the rewardees will proceed to be verified, and the 5,555 PERLs will then be distributed.
The snapshot dataset will also be released out to the community once the cutoff date is reached.
’Til next time,
Kenta Iwasaki
