Ethereum Name Service Launch Postmortem

The first Ethereum Name Service (ENS) launch happened on March 14, 2017. It was pulled back and rescheduled due to bugs discovered upon launch.

A note about postmortems: Postmortems exist to document an event, to find all the root causes of an issue, and to ensure changes are put in place to prevent such issues from happening again. They are focused on the technical issues behind the event, and on technical measures that can be put in place to prevent recurrence, not on assigning culpability or blame. Technical solutions scale and persist; blaming people does not.

Note: This post-mortem was originally written and published here by Nick Johnson. It has been republished here with his permission. Action statuses have been updated in this version.

Overview

The Ethereum Name Service (ENS) is a set of Ethereum contracts that together provide name resolution on the Ethereum blockchain. Launch of a ‘default’ resolver was scheduled for Pi day, March 14.

ENS consists of three main components, the ENS registry, ENS resolvers, and ENS registrars. Only the registry is a core immutable part of the system; resolvers are implemented by end users, and registrars are contracts that own names in ENS and are responsible for assigning subdomains based on a set of rules.

The initial ENS deployment consisted of the ENS registry, a registrar for the “.eth” subdomain that uses an auction system to allocate names, and a registrar for reverse resolution of Ethereum addresses.

All three components were deployed ahead of time, with the .eth registrar configured to go live at Unix timestamp 1489431415 (2017–03–13 18:56:55 UTC). The registrar went live at the expected time, but the registration DApp was deliberately left disabled for a while as ENS team members checked everything was working as expected.

Shortly after it went live, a community member informed the ENS team of a potential issue with the registrar, which allowed users to bid during the reveal period of auctions. This would remove some important protections of the Vickrey auction process and allow some participants an unfair advantage. This issue was quickly verified, and a fix was devised, along with providing a process for anyone who had bid already to recover their deposits.

A war room was convened on Skype to review the proposed fix and recovery path, and several Ethereum members and community developers were brought in to audit the fix. All agreed that the fix appeared to resolve the issue, and that the recovery process would work.

One party, dapphub, stated that although they believed the patch to be sound, and a last minute review of the contract hadn’t turned up any other obvious issues, they could not officially endorse continuing the launch, and recommended postponing the launch until further review could be undertaken.

The ENS team examined the nature of the bug, the feedback received from reviewers, and their knowledge of the code. Considering this and the relative ease of deploying a fix, the ENS team decided to continue with the launch, and a proposal was sent to the ENS trustees to replace the original .eth registrar with the amended one. This proposal was quickly approved, with the launch date of the replacement registrar set to unix time 1489535999 (2017–03–14 23:59:59 UTC). The new launch date, along with a description of the first bug and its remediation, was announced publicly.

Shortly thereafter, a Github Issue was submitted demonstrating a second issue with the auction registrar, which allowed participants to win auctions despite underfunding their bids. After reviewing this issue determining that it was valid, the launch was officially called off.

Timeline

All times in UTC.

  • 2017–01–10: Nick Johnson submits PR 35, a major refactoring that accidentally introduces both issues. Neither issue is caught by unit tests.
  • 2017–03–11: ENS contracts deployed and configured.
  • 2017–03–13 19:09: Community member ‘EthHead’ alerts the ENS team of the bid-during-reveal issue via the Gitter channel.
  • 2017–03–13 19:10 to 2017–03–14 02:09: Issue discussed and solution implemented by ENS team.
  • 2017–03–14 02:09: War room convened to review and approve fix.
  • 2017–03–14 05:31: Fix approved and deployed with activation timestamp of 1489535999 (2017–03–14 23:59:59 UTC).
  • 2017–03–14 07:42: ENS keyholders emailed and requested to approve switchover of registrar to new contract.
  • 2017–03–14 10:10: Switchover approved by fourth keyholder, and takes effect.
  • 2017–03–14 12:55: Issue #49 submitted, detailing the underfunded-bid bug.
  • 2017–03–14 13:48: ENS team decides to postpone launch. ENS keyholders emailed and requested to approve proposal to deactivate the new registrar.
  • 2017–03–14 02:40: Proposal approved by fourth keyholder, and registrar is disabled.

Impact

Approximately eight (8) ether in deposits was locked in the original .eth registrar by users submitting bids directly. This figure was very low because the bidding DApp was never enabled. The majority of this figure will be recovered as soon as the users reveal their bids during the original reveal period, as the ENS team have deliberately outbid all bids they could identify in order to enable this. Remaining funds will be recoverable after ENS relaunch or after one (1) year.

Approximately twenty-three (23) ether was spent on gas fees calling the .eth registrar contract.

The ENS launch was postponed.

Root Causes

1 — Lack of unit test coverage

Although the registrar had a suite of unittests, this testing was not comprehensive. The unit tests were written after the contract by a different author. No concerted effort was made to ensure that all code paths or use cases were thoroughly covered.

Unit tests existed that could have caught the bugs, but they made insufficient assertions about the post-state of the contract, resulting in a missed opportunity.

2 — Insufficient code review

The changes to the contract that introduced the bugs were subject to review in a Pull Request, but the review did not catch the bugs. In future, more care is needed when reviewing changes to critical code to ensure reversions aren’t accidentally introduced.

3 — Lack of formal, independent audit

The registrar contract was reviewed informally by a number of community members during its deployment on the Ropsten testnet, resulting in the identification and remediation of several bugs. A review was also performed by Martin Swende shortly prior to launch, but on an informal basis with time constraints. No independent, comprehensive audit was done.

4 — No clear indication of bug bounty status

No clear indication was provided to the community as to whether ENS was covered by the Ethereum Foundation’s bug bounty process. Had it been announced that ENS was covered, more attention would likely have been directed towards the code at an earlier stage in the process, resulting in the bugs being found at a stage where they could be easily remedied without affecting the launch.

5 — Reluctance to cancel a launch

The launch should have been postponed after the first bug was discovered, to offer more time to assess the contract and determine if other related bugs existed. The decision to proceed was made in part due to the mistaken belief that the first bug was a one-off one, and in part due to a desire to meet the original stated launch day.

In future, this can be mitigated by ensuring there are independent stakeholders with the ability to call a launch off if, in their opinion, such an action is warranted.

6 — ‘Attention amplification’ effect of launch

Attention should also be drawn to the ‘attention amplification’ effect of launch. We believe that a significant reason the bugs were found so close to launch was due to the large amount of additional attention the launch directed at the code. This attention could have been harnessed for better use had there been an ENS bug bounty offered.

The amplified attention on the code resulting from the public launch, while now requiring postponement, has a silver lining in that it’s possible a ‘softer’ launch would have allowed the bugs to go undetected for much longer, at which point they could have done more damage.

7 — Difficulty of live upgrades

During the development of the registrar, a conscious choice was made to require positive action from users in order to migrate from one registrar to another. This was done because the registrar has total control over what happens to the user’s deposit; by requiring user input, it’s guaranteed that they’re able to make an informed choice if the code change.

However, this complicates deployment of on-the-fly upgrades by precluding a simple automatic transfer to new rules. Future efforts should investigate the practicality of allowing changes to registration logic that does not affect users’ deposits to proceed without explicit user action.

Mitigating Factors

Fund Isolation

Because the registrar was authored specifically to segregate deposited funds in separate ‘deed’ contracts with very limited APIs, there was no prospect of either issue resulting in the compromise or loss of user funds.

DApp Disablement

The fact that the DApp was not enabled for bidding meant that the only users who placed deposits were those who interacted directly with the registrar via the console or their own code. This enormously limited the amount of deposited funds.

Conversely, disabling opening auctions would have resulted in less wasted gas in opening auctions on a registrar that was not able to accept bids.

Keyholder Responsiveness

The prompt action by ENS keyholders when issues were identified made it possible to execute alterations to ENS on a limited timeline, substantially reducing the impact of the bugs.

Actions

  • Done — Commission formal independent audit by dapphub and internal audit by Martin Swende. Owner: Nick Johnson, Status: Complete
  • Done — Announce that ENS will be covered by the bug bounty for any bugs not found in the audits. Owner: Nick Johnson
  • Done — Determine amounts to pay bug bounties to the finders of the two ENS launch bugs. Owners: Nick Johnson, Martin Swende
  • Done — Assess the viability of structural changes to the registrar to permit easier upgrades in the event of future bugs. Owner: Ethersum Dev Team
  • In progress — Discuss implementing and documenting a standard for what constitutes a ‘showstopper’ launch bug. Owner: Ethereum Dev Team
  • In progress — Enumerate use-cases and edge cases for auction registrar, and verify each is covered by a unit test. Owner: Nick Johnson, Status: In Progress

Other Perspectives