On the 9th of November 2017 we experienced a sever outage of all our EU stratum nodes that caused mining via that nodes to become unavailable. On top of that our main and backup ethermine.org servers were affected by this outage as well.
The reason for this issue was a simultaneous electricity and networking outage in two data centers at our hosting provider OVH. At we lost access to all our servers that are hosted in the first data center and approximately one hour later we also lost access to all our servers in the second, backup, data center. In general such an event is highly unlikely and unprecedented in the history of our hosting provider. While it is common (common as in once every few years) to experience issues with one complete data center, it is extremely uncommon for such a dramatic issue to occur in two data centers at the same time.
During the outage our US and ASIA nodes were working fine and available for mining. As both our main and backup ethermine.org servers were hosted in the affected data centers, block accounting and distribution was paused until the issue has been resolved and then immediately re-enabled. Blocks that were mined during the outage will be distributed according the last recorded hashrate at the pool at the time the block has been found. This means all Ethereum blocks mined during the outage will be distributed according to the hashrate on the pool immediately before the outage.
In order to make our systems resilient against such multi-data center outages we will continue to expand our stratum mining infrastructure into a multi data center — multi hosting provider approach.
We are very sorry for the inconvenience this issue has caused.
