Terra DDOS Emergency Response
Over the last 72 hours it has become widely understood Terraform Labs Was the victim of a DDOS attack on Friday September 3rd, 2021. Terraform Labs swiftly shut down their full nodes which were targeted and effectively went into “safe mode” until the attacker could be subdued. At the time of writing, Terra is the 15th largest blockchain and it should be expected this will not be the last aggressor against the network.
Of note the attack occurred after U.S. business hours on a Friday heading into Labor Day weekend (U.S. Federal Holiday). The attacker seems to have been attempting to exploit variabilities in both network reliance on centralized Terraform Labs infrastructure, as well as a time when staff may have had their guard down. Fortunately, Terraform Labs’ response was both swift and effective. The attacker was subdued within hours and the network proved extremely resilient. The extraordinary effort and success of Terraform Labs’ response to the attack was consistent with their handling of previous malicious actors, perhaps most notably the Anchor release bot attack which included a DDOS component.
Despite the outstanding effort by Terraform Labs during this test there are deep learnings and new awareness’s for the community which we should take to heart.
Prerequisite: It is important to understand the DDOS attack effectively shut down Terraform Labs full nodes. Full nodes broadcast transactions to the network which are executed by validators. At no time was the Terra blockchain compromised or even stopped. End users simply were not able to use Terraform Labs provided Graphical User Interfaces (GUI) or third-party GUI’s which relied on Terraform Labs nodes to interact with the network.
First, the attack highlighted Terra community reliance of Terraform Labs Infrastructure. We learned most projects and utilities to date (Anchor, Mirror, Pylon, Station, etc.) require public Terraform Labs nodes’ resources to operate. An interesting exception to this was Extraterrestrial Finder which was unphased by the attack. The community takeaway should be that we have an urgent need to decentralize reliance on Terraforms Labs infrastructure. Validator Watchtower suspects Terraform Labs already had this revelation prior to this most recent attack. This is evidenced by public measures to specifically decentralize front ends such as the most recent partnership with Wall Street Bets to host a Mirror frontend.
Second, the attack highlighted certain Terra Validator’s reliance on Terraform Labs Infrastructure to operate. For Terra validators to function properly they must be able to interact with the blockchain. Much of this interaction is facilitated by the Terra “Light Client Daemon” (LCD). Concerningly, it turns out, many community validators were likely relying on Terraform Labs hosted LCD as opposed to self-hosted versions.
We can estimate which validators were relying on Terraform Labs LCD infrastructure based on which validators have sub 100% uptime following the attack.
Concerningly:
· 2 of these validators have over 10MM LUNA staked.
· 5 of these validators have over 5MM LUNA staked.
· 14 of these validators have over 1MM LUNA staked.
· 2 of these validators are LIDO validators.
We are perhaps most concerned by the seeming reliance of LIDO validators on Terraform Labs Infrastructure. The reality is LIDO validator rewards are the primary source of revenue to pay Anchor depositors. If LIDO bLUNA nodes miss votes Anchor loses revenue and depositors are put at risk. Theoretically, LIDO was selected to be in charge of choosing bLUNA nodes because they had a reputation as the organization running the most elite Proof of Stake validators across all networks. The sub-optimal performance of LIDO validators on Terra should throw this reputation into question for everyone in the community (not exclusive to Terra).
We want to conclude this piece on a positive note. The list below are validators which did not experience downtime during the most recent attack.
Please consider delegating to validators on the list below. It is imperative to the future of Terra.
