On October 10th, 2017, we launched the AirSwap Token (AST). While many Ethereum-based token sales are driven entirely by smart contracts, we did things differently. AirSwap itself is an Ethereum-based decentralized global marketplace, and a token purchase could be a trade on the marketplace. Launching the token this way gave us several benefits, but also meant that we had to build and operate our core platform to support the launch itself.
Our token launcher is an implementation of the Swap Peer Protocol that provides each buyer an order, which is in turn submitted to our exchange contract on the Ethereum blockchain. The exchange contract, which you can see deployed via Etherscan and source code on GitHub, is also an implementation smart contract found in the Swap whitepaper.
It’s been shown time and time again that performing token launches through crowd sale contracts creates pressure and congestion on the Ethereum network. We hoped to relieve this during the sale by lifting the token purchase flow off of the blockchain and onto the token launcher. Additionally, we guaranteed each buyer in the main sale an opportunity to purchase tokens over the course of 23 hours, demand was smoothed over that time. With an order in hand, buyers could then complete their purchase at their leisure using an Ethereum browser extension like MetaMask.
We were able to harden core platform technology and immediately operate the platform at scale, which did well. The team worked to minimize latency while providing 100% uptime for persistent connections, even during code upgrades. This meant redundancy, fault tolerance, and active monitoring. During the 48 hours of the public and private sales, we deployed frontend code 4 times and hot upgrades to the backend 3 times. Our continuous integration (CI) process was designed and scripted to allow us to deploy confidently.
Preparation
To set up for the sale, we transferred 45.2M AST to our hot wallet, the private key to which was very well protected. Prior to each sale, we approved the exchange contract to only transfer the amount of that sale so there was no possibility of selling beyond the per-sale amount.
Approaching the main sale, we iterated through a sequence of load, test, fix, and deploy an average of once per hour. Our work the day before launch centered around edge case checking and fixing minor issues. We ran a core suite of automated tests against our code that we had been developing for months. We distributed load testing through Tsung and AWS, hammering our boxes with 50 connection requests per second. This was thought to be a ridiculously high number. We later learned differently.
The Main Sale
Our main sale enforced a maximum (cap) for individual order size based on the number of participants. The total number of tokens sold in the main sale was 42M AST, among 12,719 white listed registrants, making for a 3,300 AST individual cap. The sale was to be run over 23 hours from 10:10:10am to 9am the next morning, so every participant would have the opportunity to purchase at their convenience.
The morning of the launch, we were up early for final preparations. We deployed the latest frontend code and used our beta environment to conduct multiple complete end-to-end purchases on multiple computers using multiple wallets. We ran the beta tester sale, and our betas were supportive and patient as we finalized the deployment. About 30 minutes prior the sale, we began reducing the TTL on our DNS records in preparation for a DNS cutover to our production token launcher code. Prior to the sale opening, buyers would see a countdown on the UI, which would switch to the purchase flow once the sale opened.
What we didn’t expect was the large number of connections even before the sale opened at 10:10:10AM. At launch, every one of our charts registered a massive spike as users began requesting orders. Over the course of the day, filling orders on our exchange contract was the second highest use of gas on the Ethereum network. The launcher stayed strong during the day, despite repeated attempts to access bogus URLs or hit our API with incorrect data. It was apparent from the beginning that we would have buyers attempting to bypass the UI and hit the API directly.
All was humming along nicely until approximately 2:30PM, we encountered a race condition in our order signer that allowed some buyers to programmatically place multiple orders. We had monitors set up to notify us immediately whenever one of the invariants of the sale, such as the individual cap, was violated. We were notified within seconds of the first buyer who was able to programmatically place multiple orders, and moved quickly to ban all accounts taking advantage of this mechanism. As a precaution, we also removed from the whitelist addresses that were registered with @protonmail.com email addresses, an anonymous email service that several of the offenders had used. We were able to quickly patch, test, and deploy a fix to this issue, within an hour of our first being alerted to it and restore the protonmail accounts that were temporarily banned as a precaution. Throughout the day, traffic slowed and then picked back up as our overseas buyers woke up and purchased. By the end of main sale, we had sold 30M AST for 30K ether to over 9,400 customers.
The L(AST) Chance Sale
The tech team went to sleep on the 10th knowing that the 11th would be the true test of our system. We were running an uncapped, first-come, first-served sale for the remaining unpurchased portion of our AST tokens. We knew it was likely that the sale would not last longer than 2 hours as people would be buying large quantities quickly. We did not use any kind of queueing system in front of the launcher, as we had confidence in its capacity to scale.
At 10:10:10 AM, the uncapped sale opened. Immediately, we saw a spike of 16,695 concurrent connections. Our database was hammered with inserts and spiked to 8% CPU usage. Looking at the data after the fact, we had multiple orders placed in the first 30 seconds for over 8,600 ETH. Based on the speed of order, it became obvious that there were a select few buyers who had programmed scripts to place orders on our API and submit them to the blockchain.
By 10:19:56, the last block from the sale was settled and we had sold approximately 12M AST. We shut down access to our systems and waited for the orders placed to clear on the chain. We had sold 70% of our allocation in just over 30 seconds and the rest was purchased as orders cleared. Counting in the network performance lag of processing the blocks, we estimate that the total L(AST) chance sale time to be around 9 minutes from the first order being placed to the last order being served.
On a memory and hardware side, our monitoring solution showed us using resources efficiently. Running on c4.xlarge boxes, we had plenty of capacity and memory to spare. We did have a problem with one of the boxes we used during the day, resulting in more traffic flowing to another production box. Even under this additional load, the production API did not have any noticeable issues nor unusual resource consumption. The spike on the 11th is due to the incredible amount of traffic we got over a very short period of time.
Wrapping up
We spent the rest of the day cleaning up from the sale, generating reports and charts showing the velocity of sales, sizing down hardware and finally getting a night of sleep. While there was certainly a cost in time and effort to building out this design, the launch was a success and our core platform and ops playbooks have been tested.
In the time since the sale, it’s been fascinating to crunch the numbers from our logs and piece together a story of the actors and forces at play in this new kind of token launch. We’ve learned a lot and we’re excited to share what we can with the community. If you’re looking to perform a token launch of your own, please be sure to reach out to us at team@airswap.io.
About AirSwap
AirSwap enables peer-to-peer trading on the Ethereum blockchain. Built on a decentralized protocol, traders can add or remove liquidity through a suite of trustless products that are easy to use and free. Our mission is to empower the world with frictionless trade.
Blog | Twitter | Discord | Developers | Makers | Reddit | Facebook | Linkedin | Subscribe | Support | Request a Feature | FAQ | Trade Now
