Hurricane Irma: U.S. Virgin Islands and Puerto Rico
5 fatalities in U.S. Virgin Islands and Puerto Rico FCC doesn't have reliable reporting on Wireline and Cable service providers in either USVI or Puerto Rico. The assumption is large numbers of customers are without cable or wireline service. 870,000 customers (55% of the island) without power in Puerto Rico 22,900 customers without power U.S. Vigin Islands 1 hospital closed, 25 hospitals on backup generators 29.3% of cell sites in Puerto Rico and 60.7% of cell sites in the U.S. Virgin Islands are out of service. 1 radio station and 2 TV stations in USVI are out of service 4 radio stations in Puerto Rico are out of service
I have talked(briefly sms/fb) to several friends of mine who operate critical datacenter infrastructure. What is not shot down mostlly on diesel generators but actual impact if power isnt restored in next 5-7 days is yet to come. USVI and Puerto Rico will get the restoration it needs in critical infrastrucure, no doubt I am more concerned about othet caribbean who were impacted. Having lived in Caribbean for years and been on many islands, i can only imagine recovery Of services on smaller islands taking long time. There is an effort by Martin hannigan announced on twitter and a coordination call. Martin is active member of NANOG, Martin how can we support/join this great initivative? On Sat, Sep 9, 2017 at 7:03 PM Sean Donelan <sean@donelan.com> wrote:
5 fatalities in U.S. Virgin Islands and Puerto Rico
FCC doesn't have reliable reporting on Wireline and Cable service providers in either USVI or Puerto Rico. The assumption is large numbers of customers are without cable or wireline service.
870,000 customers (55% of the island) without power in Puerto Rico
22,900 customers without power U.S. Vigin Islands
1 hospital closed, 25 hospitals on backup generators
29.3% of cell sites in Puerto Rico and 60.7% of cell sites in the U.S. Virgin Islands are out of service.
1 radio station and 2 TV stations in USVI are out of service
4 radio stations in Puerto Rico are out of service
Electric Power Florida: 6,117,0247 customer outages (59% of state) Puerto Rico: 368,682 customer outages (23.5% of territory) U.S. Virgin Islands: St. John: 100% customers without power St. Thomas: 99% customers without power (airport, hospital, and some areas on those substations re-energized) St. Croix: 31% customers without power Georgia: 866,682 customer outages (9% of state) South Carolina: 184,471 customer outages (7.2% of state Public Safety: 29 Public Safety Answering Points (9-1-1 centers) impacted Florida 14 PSAPs out-of-service, no re-route 11 PSAPs out-of-service, rerouted to other answering points 2 PSAPs with partial service, no automatic location U.S. Virgin Islands 2 PSAPs (out of 2) with partial service, no automatic location Wireless Services: Florida 27.4% cell sites out of service (6 counties with more than 50% out of service) Puerto Rico 19.4% cell sites out of service (8 areas with more than 50% out of service) U.S. Virgin Islands 55% cell sites out of service (90% out of service St. John, 72% out of service St. Thomas, 22.5% out of service St. Croix) Cable and Wireline Florida 7,597,945 customers out of service 390 switching centers (central offices) out of service Puerto Rico and U.S. Virgin Islands Reporting still spotty, assume customers out of service in areas without electric power. Some VSAT service restored on U.S. VI. Broadcast 8 TV stations out of service Florida: 2 stations Puerto Rico: 4 stations U.S. Vigin Islands: 2 stations 26 radio stations out of service Florida: 25 stations U.S. Virgin Islands: 1 station reporting (but other stations are likely out of service, but can't report)
A couple of additional items Telecommunications - Free WiFi hotspots (where power available) Comcast Xfinity WiFi Charter Spectrum WiFi I expect other commercial WiFi hotspot service providers in Florida have (or will) have free access in the disaster areas. AT&T and Verizon are bringing temporary Cells on Wheels trucks to the disaster areas. AT&T, Sprint and Verizon are providing extra mobile data or waiving usage surcharges in the disaster areas. Check the AT&T, Comcast, Charter, Sprint and Verizon corporate sites for details. T-Mobile is probably doing something too, but the T-mobile web-site makes it very difficult to find disaster information.
T-Mobile is providing free calling text and data.
On Sep 11, 2017, at 21:11, Sean Donelan <sean@donelan.com> wrote:
A couple of additional items
Telecommunications - Free WiFi hotspots (where power available)
Comcast Xfinity WiFi Charter Spectrum WiFi
I expect other commercial WiFi hotspot service providers in Florida have (or will) have free access in the disaster areas.
AT&T and Verizon are bringing temporary Cells on Wheels trucks to the disaster areas. AT&T, Sprint and Verizon are providing extra mobile data or waiving usage surcharges in the disaster areas.
Check the AT&T, Comcast, Charter, Sprint and Verizon corporate sites for details. T-Mobile is probably doing something too, but the T-mobile web-site makes it very difficult to find disaster information.
A couple of folks asked about commercial colocation and data centers... There are about 100 major colocation and datacenters in Florida and southern Georgia. I've check the usual suspects, and none have posted outages. Directv and Dishtv "local-in-local" TV stations appear to be uninterrupted, which indicates the major fiber hubs they use for TV signal aggregation are working in the local areas. I can't tell how many data centers are on backup generators. Today's update from various official sources (FEMA, Dept. of Energy, FCC, NOAA, etc). Electric Power (DOE) Florida: 4,788,277 customer outages (48% of total state customers) Georgia: 932,587 customer outages (22% of total state customers) South Carolina: 140,759 customer outages (6% of total state customers) North Carolina: 56,834 customer outages (1% of total state customers) Alabama: 20,050 customers outages (1% of total state customers) Puerto Rico: 303,998 customers (19% of total customers) U.S. Virgin Islands Water and Power Authority (WAPA) reported several feeders on St. Thomas are re-energized. There are currently two generators (#14 and #25) online with a maximum capacity of 39 MW. An 800 kW generator has arrived on St. Thomas September 12. Water (FEMA) U.S. Virgin Islands: 341,000 people without potable water Puerto Rico: 61,980 people without potable water Public Safety Hospitals (FEMA) Florida: 33 closed, 204 healthcare facilities evacuated Puerto Rico: 1 closed, 6 on generator power U.S. VI: 1 closed and evacuated NOAA Weather Radio (NOAA) Florida: 9 out of 32 stations (28%) out of service Georgia: 4 out of 29 stations (13%) out of service U.S. VI: 1 out of 1 station (100%) out of service Public Safety Answering Points (9-1-1 centers) (FCC) Florida: 27 impacted (3 out of service, 9 partial service, 9 re-routed with ALI, 8 re-routed without ALI) U.S. VI: 2 impacted, without ALI/ANI Cable and Wireline systems (FCC) 819 switching centers (cable headends and central offices) out of service. Unknown how many are isolated, damaged or just without power. 7,184,909 subscribers out of service in Alabama, Florida and Georgia; not including Puerto Rico and U.S. Virgin Islands. Because of the limited number of service providers, I think FCC is not releasing the information.
From other reporting, Puerto Rico has about 50% out of service and U.S. VI has about 33% out of service not including St. John and St. Thomas.
Wireless Service (FCC) Alabama: less 1% cell sites out of service Florida: 24.6% cell sites out of service (5 counties over 50% OOS) Georgia: 10.5% cell sites out of service (1 county over 50% OOS) Puerto Rico: 14.5% cell sites out of service (3 counties over 50% OOS) U.S. VI: 53% cell sites out of service (St. John - 9 out of 10 OOS, St. Thomas 38 out of 57 OOS) Broadcast (FCC) Television: 9 stations out of service Radio: 51 stations out of service
On Tue, 12 Sep 2017, Sean Donelan wrote:
A couple of folks asked about commercial colocation and data centers...
There are about 100 major colocation and datacenters in Florida and southern Georgia. I've check the usual suspects, and none have posted outages. Directv and Dishtv "local-in-local" TV stations appear to be uninterrupted, which indicates the major fiber hubs they use for TV signal aggregation are working in the local areas.
I can't tell how many data centers are on backup generators.
Just to clarify, since several people wanted corrections. There is substantial outside plant damage in Florida, and many local loop circuits are out of service. I was only referring to data services inside major colocation and data centers, not in offices and other locations. One person did report they had servers reboot in a Florida colocation facility during the hurricane, likely due to power distubances. The power came back quickly, I assume an issue switching to backup generators or a glitchy UPS. Another person reported they lost connectivity to their systems in a colocation facility for 24 hours. I'm still checking on that one.
Disclosure note: AT&T and Comcast public relations folks have been sending information about what they are doing for disaster recovery. I've included some of their information.
From various official sources (FEMA, Dept. of Energy, FCC, NOAA, etc).
Fatalities (FEMA) Georgia: 2 Florida: 12 South Carolina: 2 Puerto Rico: 3 U.S. Virgin Islands: 4 Note: FEMA is slower than media reports about U.S. fatalities Non-US fatalities (AP/Reuters) Caribbean fatalities: Anguilla (4), Barbuda (1), British Virgin Islands (5), Cuba (10), French Territories (10), St. Maarten (4), Haiti (1) Electric Power (DOE) Florida: 3,568,499 customer outages (35% of total state customers) Georgia: 451,033 customer outages (11% of total state customers) South Carolina: 58,972 customer outages (2% of total state customers) North Carolina: 24,445 customer outages (<1% of total state customers) Puerto Rico: 117,244 customers (8% of total customers) U.S. Virgin Islands: The airport and hospital are still energized. Besides a few smaller areas, most customers on St. John and St. Thomas are without power. Restoration efforts will continue as USVI WAPA works to get critical facilities reenergized on the two islands. Water (FEMA) U.S. Virgin Islands: 341,000 people without potable water Puerto Rico: 61,980 people without potable water Public Safety Hospitals (FEMA) Florida: 11 closed, 204 healthcare facilities evacuated Puerto Rico: 1 closed, 6 on generator power U.S. VI: 1 closed and evacuated NOAA Weather Radio (NOAA) Florida: 6 out of 32 stations (18%) out of service Georgia: 7 out of 29 stations (24%) out of service U.S. VI: 1 out of 1 station (100%) out of service Public Safety Answering Points (9-1-1 centers) (FCC) Florida: 29 impacted (4 out of service, 9 partial service, 7 re-routed with ALI, 8 re-routed without ALI) Georgia: 5 impacted (1 re-routed with ALI, 3 re-routed without ALI) U.S. VI: 2 impacted, without ALI/ANI Cable and Wireline systems (FCC) 1,040 switching centers (cable headends and central offices) out of service. Unknown how many are isolated, damaged or just without power. 8,190,407 subscribers out of service in Alabama, Florida and Georgia; not including Puerto Rico and U.S. Virgin Islands. According to Comcast: All comcast's miami-dade and broward facilities are on generator power. Comcast is deploy portable generators in neighborhoods to re-charge outside plant. Comcast has no network access beyond Marthon in the Florida Keys, but has crews ready when the area is accessible. Wireless Service (FCC) Alabama: less 1% cell sites out of service Florida: 18.1% cell sites out of service (3 counties over 50% OOS) Georgia: 5.3% cell sites out of service Puerto Rico: 10.1% cell sites out of service U.S. VI: 55% cell sites out of service (St. John - 9 out of 10 OOS, St. Thomas 38 out of 57 OOS) According to AT&T: deployed 6 portable, satellite connected cell on trucks in the Florida Keys (Stock Island, Key West and Marathon and 2 satellite connected cell on trucks in Naples, Florida. The AT&T National Disaster Recovery Team has over 20 units deployed throughout Florida (don't know what that means, but sounded good). Broadcast (FCC) Television: 10 stations out of service Radio: 39 stations out of service
Sean - I think I speak for all of us when I say thank you very much for these updates! The concise nature of them is super helpful. -Dave On Wed, Sep 13, 2017 at 8:58 PM, Sean Donelan <sean@donelan.com> wrote:
Disclosure note: AT&T and Comcast public relations folks have been sending information about what they are doing for disaster recovery. I've included some of their information.
From various official sources (FEMA, Dept. of Energy, FCC, NOAA, etc).
Fatalities (FEMA) Georgia: 2 Florida: 12 South Carolina: 2 Puerto Rico: 3 U.S. Virgin Islands: 4 Note: FEMA is slower than media reports about U.S. fatalities
Non-US fatalities (AP/Reuters) Caribbean fatalities: Anguilla (4), Barbuda (1), British Virgin Islands (5), Cuba (10), French Territories (10), St. Maarten (4), Haiti (1)
Electric Power (DOE)
Florida: 3,568,499 customer outages (35% of total state customers) Georgia: 451,033 customer outages (11% of total state customers) South Carolina: 58,972 customer outages (2% of total state customers) North Carolina: 24,445 customer outages (<1% of total state customers) Puerto Rico: 117,244 customers (8% of total customers) U.S. Virgin Islands: The airport and hospital are still energized. Besides a few smaller areas, most customers on St. John and St. Thomas are without power. Restoration efforts will continue as USVI WAPA works to get critical facilities reenergized on the two islands.
Water (FEMA)
U.S. Virgin Islands: 341,000 people without potable water Puerto Rico: 61,980 people without potable water
Public Safety Hospitals (FEMA) Florida: 11 closed, 204 healthcare facilities evacuated Puerto Rico: 1 closed, 6 on generator power U.S. VI: 1 closed and evacuated
NOAA Weather Radio (NOAA) Florida: 6 out of 32 stations (18%) out of service Georgia: 7 out of 29 stations (24%) out of service U.S. VI: 1 out of 1 station (100%) out of service
Public Safety Answering Points (9-1-1 centers) (FCC) Florida: 29 impacted (4 out of service, 9 partial service, 7 re-routed with ALI, 8 re-routed without ALI) Georgia: 5 impacted (1 re-routed with ALI, 3 re-routed without ALI) U.S. VI: 2 impacted, without ALI/ANI
Cable and Wireline systems (FCC)
1,040 switching centers (cable headends and central offices) out of service. Unknown how many are isolated, damaged or just without power.
8,190,407 subscribers out of service in Alabama, Florida and Georgia; not including Puerto Rico and U.S. Virgin Islands.
According to Comcast: All comcast's miami-dade and broward facilities are on generator power. Comcast is deploy portable generators in neighborhoods to re-charge outside plant. Comcast has no network access beyond Marthon in the Florida Keys, but has crews ready when the area is accessible.
Wireless Service (FCC)
Alabama: less 1% cell sites out of service Florida: 18.1% cell sites out of service (3 counties over 50% OOS) Georgia: 5.3% cell sites out of service Puerto Rico: 10.1% cell sites out of service U.S. VI: 55% cell sites out of service (St. John - 9 out of 10 OOS, St. Thomas 38 out of 57 OOS)
According to AT&T: deployed 6 portable, satellite connected cell on trucks in the Florida Keys (Stock Island, Key West and Marathon and 2 satellite connected cell on trucks in Naples, Florida. The AT&T National Disaster Recovery Team has over 20 units deployed throughout Florida (don't know what that means, but sounded good).
Broadcast (FCC)
Television: 10 stations out of service Radio: 39 stations out of service
In my last summary, I made a comment I didn't know what the network disaster recovery team meant. AT&T recovery efforts 3000 recovery team members 14 Satellite Cell on Light Truck (SatCOLT) - 1 in Tallahassee, 2 in Naples, 4 in Florida Keys - 1 portable cell site St. Thomas, U.S. Virgin Islands 3 Emergency Communications Vehicles 50 Drones Command trailers, hazardous materials response equipment Verizon recovery efforts 2 Satellite Picocels on Trailers (SPOTs) in the Florida Keys Refueling and generators at cell towers througout Florida Hundreds of portable generators 2 Wireless Emergency Communication Centers in Naples (provide charging stations, phones and computers for the public to contact family and friends) Sprint recovery efforts Sprint reports over 75% of its network is repaired in the southeast and Puerto Rico. AT&T, Sprint, T-Mobile, Verizon continue to extend their wireless data, text and voice plans waiving overage charges in the disaster areas. Details are different for each carrier, so check their web sites or customer service. Federal Aviation Administration recovery efforts Deployed emergency mobile air traffic control tower to the international airport on St. Thomas, U.S. Virgin Islands Drone authorizations 138 commercial drone operator authorizations for Hurricane Harvey 80 commercial drone operator authorizations for Hurricane Irma Federal Communicatiosn Commission recovery efforts Reports at least 8,258,789 cable and wireline subscribers out of service in Alabama, Florida and Georgia. Strangely, the number of non-mobile switching centers increased from 1,040 (Sept 13) to 2,188 (Sept 14). I believe this exceeds the number of cable headends and central offices in Florida. So I don't know what this number represents or why it increased so much. The situation is still dire in some locations, but generally the disaster operations have moved to recovery and restoration. Unless something significant happens, this is the last summary report about Hurricane Irma from me.
any word on cuba? randy
On Fri, 15 Sep 2017, Randy Bush wrote:
any word on cuba?
Cuba status: (United Nations, Office of the Resident Coordinator) 10 fatalities. The country has recovered 70% of its power service. The seven provinces most affected are being assisted by other territories. It is expected that full services could take up to a month to be restored. 62% of the 320 Wi-Fi areas affected have been restored. Cellular coverage remains and 43% of the service was recovered, even though there were 176 transmitters affected. 50% of telephone services (fixed lines, collective phones, public pay phones and internet lines) have been restored, with 79,000 cases pending. https://reliefweb.int/report/cuba/response-hurricane-irma-cuba-situation-rep...
any word on cuba? ... https://reliefweb.int/report/cuba/response-hurricane-irma-cuba-situation-rep...
bad, but more like florida than their neighbors to the east, some of whom are almost standing on bedrock. thanks for the info. randy
On Fri, 15 Sep 2017, Sean Donelan wrote:
The situation is still dire in some locations, but generally the disaster operations have moved to recovery and restoration. Unless something significant happens, this is the last summary report about Hurricane Irma from me.
I said it would be the last summary report, but an important correction. FCC found a problem with the data statistics: [NOTE: The number of cable systems and wireline subscribers out of service is reduced significantly from yesterday because, in addition to removing from the total states/territories deactivated in DIRS, the FCC found that one or more providers had previously filed multiple entries, resulting in double/triple counting number of subscribers and inaccurate percentages per day. Those duplicative entries have now been removed.] There are at least 1,691,484 subscribers out of service in the affected areas in Florida, down from 8,258,789.
participants (5)
-
Dave Temkin -
Matt Hoppes -
Mehmet Akcin -
Randy Bush -
Sean Donelan