Post-Mortem Disclosure
How We Beat Our First Attackers
IT security learnings from the early days of the CIV community
Disrupting traditional finance apparently comes with a price. Hackers, haters and exploiters of all sorts pay unwanted visits and disrupt daily operations in creatively destructive ways, to try and extract rent.
During the early days of our Civilization $CIV community, we have already received two targeted attacks, likely originating from the same group of attackers:
- Telegram bot attack
- Website DDOS (Distributed Denial Of Service) attack
Both these attacks exploited relatively well documented security flaws of the underlying platform, Telegram and the HTTP protocol respectively. This post outlines the anatomy of our DDOS attack.
Before the Attack
On July 31st 2021 the crypto token $CIV was soft-launched with zero marketing budget. Around that time, we were just ~20–30 people plus a few sniping bots, frequents companions to a lot of new token listings.
Since then, we grew organically, to just few hundred holders and a similar number of subscribers to our Telegram chat. For a token that early and that small, we were punching well beyond our weight, with a Market Cap of over $10 million dollars reached relatively quickly. Cool.
The less cool part of that was the unwanted attention we attracted. Since our project is to launch the world’s first 100% Decentralized (DEX) Fund, we do not have a full-time dedicated team of experts who all know each other and work together in-person, unlike traditional funds. But much like traditional funds, we are exposed to vulnerabilities and hack attempts to a greater extents than other crypto projects and memes. Everyone loves the money.
Even if initially this approach may be slower and cause complexities, I am a passionate believer in the power of community. When you set out to do something, you may fail. But when a large enough group of people sets out to do something, it will essentially have already happened! Such is the power of a laser-like-focussed community. If you want to go fast, go alone. If you want to go far, go together.
People who volunteer work with greater passion, focus and love. Of course we all aim to gain from our contributions, but when the reward is purely monetary, I personally lack purpose and clarity. I want to belong to something bigger, with a purpose, and allowing everyone to access the benefits of decentralized crypto investment ticks all my boxes.
And the example of how we responded to our first attackers, just makes me very very proud: it confirms the power of our wonderful community, grounding its performance into observable facts.
The First Warning
On August 11th 2021, I received the first threat warning via Telegram.
Now, those of you who are not familiar with malicious attacks, may be wondering:
This sounds like a friendly and legitimate — if slightly technically worded — warning, right?
Not really. Do not be fooled so easily. Have you ever seen an attacker come and tell you clearly how they are attacking you?
Our friends from CloudFlare warn you, on their own authoritative support website:
Ransomers tend to introduce themselves as security researchers who have found a vulnerability. This will, understandably, increase the response rate of website owners, as it is not immediately clear that they are about to be ransomed. If at all possible, you should not respond.
In fact, we did not respond. And nothing happened.
The First Flood Attack: Telegram
Shortly after I ignored the message above, starting on August 12th 2021 and thereafter intermittently for the following 2 days, our Telegram group gets flooded by over 40,000 fake users. Just 1 day later, now would that be a puzzling coincidence?
For those of you wondering, Telegram is an insecure platform, subject to a number of well documented security weaknesses: like for example those reported in 2021, 2019 and 2018 among others. Basically anyone can join public Telegram groups, harvest their user lists, send spam messages to users without strong privacy settings, and more relevant to our case, add masses of users to groups to flood them.
Before getting into the DDOS attack, please read this excellent piece from P. Alessio to learn more about the Telegram flood attack and the brilliant solution that our community developed to address it, gifting valuable code to the whole open-source world:
So this first attack was successfully mitigated thanks to the prompt, generous contribution of our young but powerful community.
However, our malicious friends were still lurking in the darkness of our anonymous Telegram group, waiting to launch their next attack.
The Second Flood Attack: DDOS
On August 23rd 2021, Civilization launched its official new (and very professional) Website, CivPaper, 25 FAQs page and NFT Treasure Hunt. We demonstrated again how powerful a community can be, when setting its common mind to something: and just prior to that date, we had been listed by Coingecko and CoinMarketCap in record time — only adding to our credibility and comparative advantage towards other project taking months and months (and millions of $ in funding) before achieving comparable objectives.
Punctual, like a British Butler, the next anonymous message is delivered to my Telegram inbox:
Reminds you of anyone?
Now this guy really got on my nerves, but in spite of the hard work prior to the launch date having brought everyone to a state of mental exhaustion, I held steady and again did not reply. However, DexMan received the same message at the same time, and had the bad idea of banning this hacker. The hacker created another fake account, re-sent the message, and DexMan banned him again (it’s likely to be a male).
Enter the hacker temper tantrum.
This a screenshot from our main Telegram group.
The attacker joined on purpose with this account (likely to be one of his many personal accounts… and personalities) to post the message, then left, here are the logs from the main group: literally joined and left right after posting the message.
So what did the attacker do?
Around 10pm cet, flooded our web server with a lot of requests. How many?
At 4pm cet, time of our launch, we received 32.4 thousand requests; at 10pm cet, time of the attack, we received 5.39 MILLION requests.
Now this is not an especially high number, as DDOS can reach millions of requests PER SECOND — but it was high enough to cause the following problem:
In essence, our brand-new, wonderful website became unreachable. Boo.
The Immediate Response To The DDOS
Within 20 minutes, DexMan (dev) and GigaYoshi (sysadmin) had mitigated the attack by activating the “I am under Attack!” mode in Cloudflare, which blocks malicious requests using aggressive techniques that also degrade the real user performance (and therefore are not appropriate to other scenarios).
This solved the first issue — stopping the attack — but did not bring the site back up. The hosting provider, having received all that traffic, had disabled our server. All in all, the attack itself only lasted a few minutes and was very effectively mitigated by our team through the appropriate emergency patch.
Therefore, within the first 60 minutes from the attack, our team joined forces with Kikai (web-dev) to bring back up the previous version of the website, hosted on Squarespace. We already had a clone prepared and ready to deploy, but we didn’t want to expose it to the attacker without further investigation.
Bringing the old site back solved the second issue — showing some of our content to real users — but still did not let us bring the new site back up, which is frankly so much better than the old one!
Next, we had to contact the support service of our hosting provider, to ask them to get the new site server online.
Finally, around 3.15am cet, about 5 hours after the attack, the old website was back online. No monetary damage caused, but loss of sleep for sure — much like with the Telegram bot development, as you can read in that brilliant piece, again here is the link to it in case you missed it above!
The hosting support replied and cleared the incident, so we solved the third problem — bringing our new site back up — but we still had to close any security shortfalls leading to other potential attacks.
Root Cause Analysis
While setting out to bring our new site online, DexMan GigaYoshi and Kikai were also running some analytics to understand what had happened.
This was the legitimate traffic that had developed in the first few minutes after the new website launch at 4pm cet:
Quite a global audience, relatively focused on the EU.
This was the malicious traffic we received during the attack around 10pm cet:
See any difference?
Not only much more traffic, but also geographically sourced from very different origins. So we could immediately conclude the first root cause was a flood / denial of service attack — this was obvious pretty much instantly to everyone involved.
The second question we had to address was if our Wordpress hosting set up had any security flaws. GigaYoshi ran a WP Scan and did not find any trace of security issues.
We had been under an extremely tight deadline to deliver the site on time for the launch, therefore our team was essentially performing initial security audits that were scheduled anyways for this week — just a lot faster, under more pressure, and with less sleep than planned.
So if WordPress wasn’t the issue, then what could be? The team upgraded the CloudFlare subscription and switched on a few extra security measures called Web Application Firewall (WAF).
However, the security gap was still unclear at that stage. Why would CloudFlare not filter out all such obviously malicious traffic through its own static cache, but let it pierce through and flood our origin server?
After all, CloudFlare should theoretically be designed to filter out attacks like this one, but in our case it only filtered away the attackers labelled as Bad Browser, leaving a large proportion of the offending requests as Unclassified and therefore undetected as threats, or to be more accurate, unfiltered.
As it turned out, DexMan noticed what turned out to be a vital clue.
Each attack was appending a random string to our website address. The Cloudflare protection was pierced due to the use of these useless query strings designed to bypass the cache protection; since we do not have query strings on our static website, we activated Ignore Query String caching level, and this will prevent any similar attack from being passed onto our origin server.
A single switch, and the incident was closed.
Further security audits and stress testing will be commissioned to prevent future risks, especially as we prepare to host a money-sensitive piece of technology: due care is never enough. Launching a website is relatively low-risk, but our staking will be a whole different ballpark!
Please donate ETH to our multi-signature wallet, so we can afford to invest in high-quality paid security audits from third parties that do not belong to our community:
0x51D377b4CD1585C460c75f9aD806b8d63db7A09E
Thank you for your time reading through this technical piece. If you want to learn more about Civilization please read:
