Technical problems, or what to consider if you have decided to run your own ICO
Actively promoted by us Polybius’ ICO succeeded stunningly — and this time in a literal sense. Stunned by attention of 14,000 people … Polybius’ website went down. And that was just the beginning. As promised, what happened and what could be done better — in today’s article.
Sure enough at the beginning of the ICO, we were ready for DDoS attacks and server overloads. However, frankly speaking, we underestimated not only abilities of nasty hackers and degree of interest in the project but also the potential for emergency situations.
The technical platform (user’s dashboard, all infrastructure) for Polybius’ ICO was provided by Ambisafe. Starting with the development of blockchain technology products company eventually expanded their field of interests towards providing the set of tools and services needed for running an ICO. After choosing an experienced contractor, we entirely counted on their knowledge and skills without paying additional attention to those things we should (and even must!) have checked. And that was our first mistake.
Too weak hardware
Powerful servers are essential if you want your website to run at the start of the sales. Predicting the potential server load at the very first hours of the ICO, our contractor provided us power requirements, but even then we assumed that those measures might be insufficient and we doubled the characteristics. To avoid server overload, we’ve located the user’s dashboard and the Polybius’ website itself on different servers. However, database and dashboard main files remained undivided. And that was the mistake number two.
The database suffered each time there were too many requests to the dashboard (and vice versa), and the server went down being periodically unavailable. When the problem became apparent, we increased the servers’ power almost ten times in comparison with original characteristics. However, it took some time to change the configurations of the dashboard and upload all files to new servers.
The furor around the start of the ICO was so high that once the counter on polybius.io showed zero, crowds rushed to buy tokens. A part of the users even managed to make a false start. The rush of real users itself was not a huge problem. The main trouble was caused by high attention from bots — the DDoS attacks had begun. The dashboard was overwhelmed by 30 million requests sent by DDoS’ers within 6 hours. In the first couple of hours, the situation escalated — it was twice as hard to fight against malicious traffic as it wasn’t always clear whether it was a real person hitting “Refresh” in hope to access the account, or it was a bot that acts in just the same way.
Considering that there were lots of those wishing to participate — they insistently tried to access the website regardless of the DDoS and continued to buy tokens in moments of relief. As a result, system accumulated numerous amount of unaccomplished transactions that got stuck. And here we smoothly shift to the problem which for obvious reasons worried the participants of the crowdfunding the most.
Since the system where transactions took place is complex and multi-module, each operation had to pass through a particular way to be completed. The DDoS attack caused several modules to lag, and we had to confirm stuck transactions along with traffic filtering manually. But even after the manual outthrust of stuck transactions the users still couldn’t get their tokens — now due to obstructions that appeared in the Ethereum network. That was an emergency which caused panic, and a blast of comments in the sort of “My funds have been deducted, but I haven’t got any tokens. Half an hour/hour/5 hours… had already passed, what’s going on?” Now, as all of the tokens are accounted, we can only apologize once more for the inconveniences and explain the details behind the problem.
Along with the start of Polybius’ ICO, another large ICO (BAT — BasicAttentionToken) took place on Ethereum blockchain, and competing investors led the network to an overload. To understand the basis of the issue we had, it is necessary to comprehend how Ethereum works. To send a transaction to the Ethereum network you need to pay a commission fee — the higher the commission fee is, the faster the transaction reaches the network. The participants of the second ICO increased their commission fees to speed up their transactions. With such “competition,” Polybius transactions (having an optimal cost-time ratio) couldn’t get through and formed a queue in Ethereum network, hanging somewhere between our system and the blockchain.
To make not a single but multiple transactions in Ethereum, as it was in our case, we had to form a queue of transactions, giving a conditional number to each of them. These numbers have to be in an absolute order of increment which means that each next transaction must have a higher number than the previous one. That means that the transaction number 10 cannot be sent to the network if the transaction number 9 is still undispatched. To avoid network overload and multiplication of stuck transactions we were forced to raise commission for all new orders — new transactions started to get normally confirmed, now in acceptable time. The very first transactions (that is a huge amount of orders — more than a thousand of them) we had to leave “as is,” so our first participants weren’t quite the first to receive the tokens.
What have we learned
The most important (and obvious) knowledge we obtained is related to problem-solving in general — when you’ve got a problem, the best you can do is to start solving it with maximum speed and efficiency. The first thing we did was increasing the server capabilities, switching to more powerful hardware, dividing the database among multiple servers, optimizing the firewall and filtering the DDoS attacks. And only that took us about 24 hours. Tuning the process of carrying out transactions through Ethereum took additional time.
Now, with a substantial ICO experience, we can surely say that you cannot be too prudent with that sort of things. It is always better to be overly cautious — that’s why if you ever decide to have an ICO, you should consider the following points:
– During the first hours the server load can be sky high and leave behind the bravest expectations;
– If your project (or the company behind it) is known, attention from DDoS’ers is guaranteed — be ready to be attacked;
– If other ICOs would happen at the same time when yours, it is better to think about your strategy in advance. You can either wait a couple of days until transactions eventually go through or you can invest additional resources and force orders through the network;
– It is also important to inform your participants about such issues. That will decrease the number of similar queries to technical support, allowing them to focus on more relevant issues.
Incidentally, it is impossible to foresee all the potential problems, so you have to be ready for all kinds of emergencies. You can spend a lot of time on counting the number of mistakes you’ve done inadvertently, but learning from mistakes is just another way of improvement. After all, we are glad that our technical issues haven’t opposed the successful accomplishment of our campaign.