Puerto Rico’s Slow Internet Recovery
While most of the BGP routes for Puerto Rico have returned, DNS query volumes from the island are still only a fraction of what they were on September 19th — the day before the storm hit. DNS activity is a better indicator of actual internet use (or lack thereof) than the simple announcements of BGP routes.
We have been analyzing the impacts of natural disasters such as hurricanes and earthquakes going back to Hurricane Katrina in 2005. Compared to the earthquake near Japan in 2011, Hurricane Sandy in 2012, or the earthquake in Nepal in 2015, Puerto Rico’s disaster stands alone with respect to its prolonged and widespread impact on internet access. The following analysis tells that story.
DNS statistics
Queries from Puerto Rico to our Internet Guide recursive DNS service have still not recovered to pre-hurricane levels as illustrated in the plot below. Earlier this week, on December 4th, we handled only 53% of the query volume from Puerto Rico that was received on September 18th, just before the hurricane. Both dates are Mondays, hopefully ruling out possible day-of-the-week effects.
Queries from Puerto Rico to our authoritative DNS services are also reduced from prior to the hurricane, but not as much as to our recursive DNS service. This may be because caching effects are more pronounced with our authoritative DNS services, since they handle queries for a smaller set of domains than our recursive DNS service. Additionally, we may have lost some Internet Guide clients if those computers reverted to a default DNS configuration upon returning to service. Regardless, the volume is still lower than pre-hurricane levels for authoritative DNS. On December 4th, we handled 71% of the query volume from Puerto Rico as compared to September 18th.
Based on these two figures (53% and 71%), we estimate that internet service in Puerto Rico is only a little more than half of where it was before the hurricane.
BGP and Traceroute measurement statistics
The graphic below shows the impact of the hurricane on the routed networks of Puerto Rico colored by the major providers. Many of these BGP routes were withdrawn as the hurricane came ashore and the island suffered what has been labeled the largest power outage in U.S. history. By early November, most of these routes were once again being announced in the global routing table. However, damage to last-mile infrastructure meant that many Puerto Ricans were still unable to obtain internet access.
Our traceroute measurements to Puerto Rico, illustrated below, tell a similar story — a steep drop-off on 20 September 2017, followed by a long slow recovery that appears to come incrementally as different pieces of Puerto Rican infrastructure come back online. Despite an island-wide power outage, some networks in Puerto Rico (like Critical Hub Networks) continued to be reachable throughout the period of analysis. While the plot below shows a steeper dip than the BGP-based plot above, the responding hosts that we measured to are often part of the core infrastructure. These hosts are more likely to be connected to backup power than access layer networks and could, like the BGP routes above, overstate the degree of recovery.
Submarine cable impact
Perhaps less appreciated about this incident is Hurricane Maria’s impact on connectivity in several South American countries. Puerto Rico is an important landing site for several submarine cables that link South America to the global internet. The cable landing station serving Telecom Italia’s Seabone network had to be powered down due to flooding. A statement from Seabone read:
We must inform you that Hurricane Maria (Category 5) has impacted Puerto Rico causing serious damage and flooding on the island. We had to de-energize our nodes at the station to avoid serious damage to the equipment.
As a result, in the early afternoon on 21 September 2017, we observed traffic shifts away from Telecom Italia in multiple South American countries as the submarine cable became unavailable. To illustrate the impact, below are four South American ASNs that experienced a loss of one of their transit providers at this moment in time. Cablevision Argentina (AS10481) is from Argentina, while the other three are from Brazil. Brazilian provider Citta Telecom AS27720) lost service from Eweka Internet, while the others lost Telecom Italia transit.
Additionally, following the hurricane, Venezuelan incumbent CANTV announced that their international capacity had been cut by 50% due to storm-related submarine cable damage. The announcement was met with skepticism from a citizenry increasingly subjected to censorship and surveillance by their government.
¡SÍ CLARO OTRO MENTIRAS MAS! Cantv informa que Huracán María ocasiona fallas de internet en Venezuela pic.twitter.com/eH2kMEUfwk
— NoticiasSimonBolivar (@NoticiasSB1) September 22, 2017
However, our data shows that impairment of CANTV’s international links aligns with other outages in the region due to the effects of the hurricane. The plots below show latencies to CANTV from several cities around the world spiking on 21 September 2017 after the submarine cable station in Puerto Rico was flooded.
Conclusion
Immediately following Hurricane Maria’s arrival in Puerto Rico, Sean Donelan, Principal Security Architect of Qmulos, began dutifully posting status updates he had collected to the NANOG email list about the connectivity situation on the island. In addition, the website http://status.pr/ was setup to collect various metrics about the status of the recovery.
Now with over two months of hindsight, we can truly appreciate just how devastating the hurricane was in many respects, other than simply internet impairment. Puerto Rico may no longer be in the headlines as it was just after the storm, but the resources required to get this part of the United States back on its feet are truly extensive.
Here is more information about how you can help:
Puerto Rico hurricane victims still need help. Here’s what you can do https://t.co/lSmulxJ0Kv
— PBS NewsHour (@NewsHour) November 10, 2017
Originally published at blogs.oracle.com on December 7, 2017.
