Note: All times shown below are in Universal Coordinated Time (UTC).
The Outage
As most, if not all of you know by now, last Friday (October 21st) Dyn was hit by a massive DDoS attack that disrupted services for Twitter, Spotify, Reddit. Netflix, TNYT, and Pingdom along with a host of other core Internet services.
Dyn stated on their site that “at 11:10 UTC on October 21st-Friday 2016 we began monitoring and mitigating a DDoS attack against our Dyn Managed DNS infrastructure. Some customers may experience increased DNS query latency and delayed zone propagation during this time.”
Effect on Pingdom services
Shortly after this, our monitoring services noticed a spike in outages globally, particularly on the Eastern US seaboard. At 16:33 delayed responses from our probe servers had escalated to the point that our customers were receiving delays in alerts.
As a result of the DDoS attack and our severely degraded monitoring service, our alerting service could not handle the sheer number of alerts being raised. This resulted in large queues of hour-old alerts waiting to be sent to Pingdom customers. Despite our Ops team purging the queues, the performance and reliability of alerts degraded until, at 19:20 we went to DEFCON 1 and suspended the alerting service altogether.
To add to the frustration of the issue, Zendesk and our helpline suppliers also experienced issues due to the attack. This meant that our support ticketing system and phone lines were also down, meaning that we were only contactable via email. By 21:10 we were able to reinstate the service and report that Pingdom services had returned to normal service levels. All updates on how events unfolded for us during the DDoS attack are available on our status page.
Avoiding disruption in the future
We would like to apologize to our customers for the loss of service during the incident and in order to avoid similar incidents in the future, we have put together a plan of how we can mitigate the impact of DDoS attacks in the future.
In the short term, by caching IPs internally, we can maintain the operability of our services even when DNS go down. We will improve our notification system to better handle spikes and network issues, like the one faced on Friday.
In the long term, we will work on improving our availability monitoring service to better deal with regional outages. We will also endeavour to improve our presence online, such as the availability of pingdom.com, during a crisis.
