T-Mobile PR on twitter just posted that two of it's submarine cables are out of service. Claro PR Wireless (this is the ILEC in PR) website can't even be reached. I am assuming this is due to power and submarine cable issues since I'm sure t-mobile and many other providers are using the same cables. Link to the post on twitter: https://twitter.com/tmobilepr/status/911644083155869697 - Javier On Fri, Sep 22, 2017 at 10:43 AM, Sean Donelan <sean@donelan.com> wrote:

Following up - there are three cable landing stations and 9 submarine cable systems connecting Puerto Rico.

One of the cable landing stations experienced flooding, and shutdown its power system affecting some circuits. I haven't been able to determine how many submarine cable systems are affected, since they share cable landing stations.

According to PREPA Net, the fiber subsidary of the Electric Power Authority, the power system for the Punta las Marías submarine cable station is back in service after flooding. I think Isla Verde and Punta las Marias refer to the same landing point. I don't know the status of individual submarine cable systems using that landing station. The Miramar and San Juan cable landing stations are also in service.

From one of my colleages that has a decent sized WISP in Puerto Rico.

" Guys, we are ok, network hurt pretty bad… will need help " There are a bunch of WISPs waiting to go rebuild, but waiting for the clearance to do so. https://radar.qrator.net/as14979/providers#startDate=2017-08-09&endDate=2017-09-23&tab=current It looks like they're still online via Critical Hub Networks and Columbus Networks, but not Liberty. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Sean Donelan" <sean@donelan.com> To: nanog@nanog.org Sent: Sunday, September 24, 2017 2:28:35 PM Subject: Re: Hurricane Maria: Summary of communication status - and lack of The ILEC, Claro, is reporting all 24 central offices in Puerto Rico are now operating on generators, and maintaining re-fueling operations. The CO's in the (San Juan?) metro area have voice, data and long distance service including to the mainland. The CO's elsewhere in Puerto Rico have only local voice service. The offices are isolated, with no long distance or inter-office data service. Although the CO's are operational, substantial outside plant is damaged. Which means most subscribers do not have service. Inter-office facilities outside the (San Juan) metro area are damaged, which means people with service in those areas can only make local calls. Wireless sites are still being evaluated. The Puerto Rico Transportation Department is providing road crews to clear/rebuild roads and escort cellular providers repair convoys to remote cell sites. The Puerto Rican government has not re-established communications with officials in the following municipalities: Aibonito, Jayuya, Lajas, Mayaguez, Quebradillas, Rincón, Sabana Grande, Vieques and Villalba.

Sorry, WISPs in the US48 to go to PR to help rebuild downed WISPs. Yes, they need to be able to get there first. Those already on the island are doing what they can until more supplies arrive. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Sean Donelan" <sean@donelan.com> To: nanog@nanog.org Sent: Sunday, September 24, 2017 4:13:33 PM Subject: Re: Hurricane Maria: Summary of communication status - and lack of On Sun, 24 Sep 2017, Mike Hammett wrote:

There are a bunch of WISPs waiting to go rebuild, but waiting for the clearance to do so.

I'm not sure what clearances they are waiting for. If they are already in Puerto Rico, self-sufficient, and respect curfews and other emergency responders, they should be able to start local restoration and recovery activities. Several local ISPs and communication providers have announced open public WiFi hotspots outside their Puerto Rico offices during non-curfew hours. I've also seen reports from individuals volunteering on the Virigin Islands setting up internet access. If they are not already on the island, most Puerto Rican airports and ports are still closed to non-military or relief activities. There is no U.S. mail or freight service. Only one airport was open for limited commercial flights. They will need to bring everything neccessary to support themselves, including food, water, shelter, etc. Managing volunteers who want to help is difficult in all disasters. Unless they have training how to survive and take care of themselves in such a situation, letting in outside well-meaning volunteers sometimes become additional people who need to rescue. WISPs already on Puerto Rico or U.S. Virigin Islands, with resources for recovery and restoration of communications; can contact the FCC Operations Center, (202) 418-1122, FCCOperationCenter@fcc.gov http://transition.fcc.gov/Daily_Releases/Daily_Business/2017/db0920/DA-17-91...

You're assuming the WISP isn't providing infrastructure to critical facilities. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Jean-Francois Mezei" <jfmezei_nanog@vaxination.ca> To: nanog@nanog.org Sent: Monday, September 25, 2017 6:55:41 PM Subject: Re: Hurricane Maria: Summary of communication status - and lack of On 2017-09-24 17:13, Sean Donelan wrote:

I'm not sure what clearances they are waiting for. If they are already in Puerto Rico, self-sufficient, and respect curfews and other emergency responders, they should be able to start local restoration and recovery activities.

Priority is to restore communications to emergency responders, restore power to hospitals and other critical infrastructure). So workers that clear roads, remove dangling electrical wires would prioritize fixing of that critical infrastructure. That road you need cleared to get to your fixed wireless antenna will wait. Similarly, I get the impression that all cargo capacity into the island is still controlled to prioritize essentials. So those spare circuit board you need to fix a router have to wait. Also, with residences overwhelmingly without power, fixing the "normal" ISP business won't do much when nobody can use it. It is best to focus on wi-fi in central locations such as shelters, and cellular for first responders and others. There are good reasons local governments work out disaster plans because they need to identify in advance what gets priority after a disaster.

This is great to hear Nicholas. On Mon, Oct 2, 2017 at 5:55 PM, Nicholas Harland <nharland@gmail.com> wrote:

Hi Sean,

Thank you for all of your updates. I am just catching up on them because I only recently got back from the virgin islands. I am one of those volunteers working in the USVI. St John specifically. We are building out a wireless network, and had our first hotspot up in Cruz Bay 4 days after Maria, with connectivity to NPS/FEMA/Red Cross/St John Rescue/Fire/Police just a few days after that.

If there are technical minded and physically able bodied people would like to join the effort on St John, even just for a 1-2 week rotation, I would be happy to discuss what we need in terms of support and can make all arrangements on the island for housing etc. Getting some relief and fresh minds in would be a great help as our team is primarily St John residents who have been on the island through both hurricanes and have had to deal with their own personal situations while also trying to get internet up where it's needed.

St John was hit directly by Irma, infrastructure was completely destroyed, but it's a very small island and so the humanitarian situation there is much more stable than Puerto Rico, but many of the resources that were assisting on STJ are now rightfully being diverted to SJU. You could expect to sleep somewhere that has a generator running overnight, have access to refrigeration/freezer (though cannot open fridge during day). Food/water situation is fine there, we have a beach volleyball game on Sundays, more generators are appearing on the island and some businesses are opening.

Regards,

Nick Harland

On Sun, Sep 24, 2017 at 2:13 PM, Sean Donelan <sean@donelan.com> wrote:

...

On Sun, 24 Sep 2017, Mike Hammett wrote:

...

There are a bunch of WISPs waiting to go rebuild, but waiting for the clearance to do so.

I'm not sure what clearances they are waiting for. If they are already in Puerto Rico, self-sufficient, and respect curfews and other emergency responders, they should be able to start local restoration and recovery activities.

Several local ISPs and communication providers have announced open public WiFi hotspots outside their Puerto Rico offices during non-curfew hours. I've also seen reports from individuals volunteering on the Virigin Islands setting up internet access.

If they are not already on the island, most Puerto Rican airports and ports are still closed to non-military or relief activities. There is no U.S. mail or freight service. Only one airport was open for limited commercial flights. They will need to bring everything neccessary to support themselves, including food, water, shelter, etc.

Managing volunteers who want to help is difficult in all disasters. Unless they have training how to survive and take care of themselves in such a situation, letting in outside well-meaning volunteers sometimes become additional people who need to rescue.

WISPs already on Puerto Rico or U.S. Virigin Islands, with resources for recovery and restoration of communications; can contact the FCC Operations Center, (202) 418-1122, FCCOperationCenter@fcc.gov

http://transition.fcc.gov/Daily_Releases/Daily_Business/2017 /db0920/DA-17-913A1.pdf

It looks like someone kicked the cellular carriers public relations people into gear. Today, instead of the normal "we care" messages; they released statements providing more concrete details about their restoration activity in PR and USVI. Overall, 91.2% cell sites out of service in Puerto Rico. 34 of 78 counties have 100% cell sites out of service. This will continue to change up and down, as sites are restored and circuits are damaged by cleanup activity. There are over 2,671 cell sites on Puerto Rico and 106 cell sites in U.S. Virgin Islands. As carriers bring in tens of generators and repair equipment at a time, gives you some idea how long restoration will take. In alphabetical order... ATT: "We continue to send aircraft with essential supplies and network resources as we help the people of Puerto Rico. These flights include portable temporary cell sites, high capacity generators to provide temporary power, and other larger network equipment on cargo planes and barges to help restore services on the island. We planning to set up a number of portable cell sites in the San Juan area as soon as possible. So far, we’ve sent multiple flights carrying the following supplies: More than 30 generators 5,000+ gallons of water We are also focused on network restoration in the U.S. Virgin Islands are bringing additional resources there." Claro (google translate from Spanish): They reported that in the metropolitan area specifically, Claro's signal was already reaching 31 percent of customers in San Juan, 22 percent in Guaynabo and 18 percent in Carolina and Bayamón. At the island level, the Claro signal is up in 14 municipalities today, covering an average of 20 percent of the clients in Aguada, Manatí, Mayaguez, San Germán, Cabo Rojo, Trujillo Alto, Dorado, Camuy, Quebradillas, Humacao, Juncos , Caguas, Aguadilla and Toa Baja. That number will increase in the coming days. Sprint: "A vessel has already arrived in Puerto Rico with the generators and parts required to begin the work. In turn, a body of over 40 Sprint engineers and technicians in the United States were sent to the Island to join the local technical staff, coordinate the delivery of the equipment received and continue work to speed up the communication. A second shipment will arrive on the island this Wednesday, September 27 with additional spare parts and materials." T-Mobile: "The damage to the infrastructure is unprecedented, but equally it is the support we are receiving from T-Mobile US. Between Saturday and Sunday, six MD11 cargo planes and one AM124 (second largest cargo plane in the world) arrived with 80 generators, 16 trucks, equipment to build 100 communication facilities. More cargo planes will arrive today with more equipment and personnel." T-Mobile also mentions while T-Mobile's field engineering crew was at the Luis Muñoz Marín Airport, they were drafted to help install a generator for the FAA Control Tower. That's one way to help get your supplies on the island. If you have information about other telecommunication providers in Puerto Rico or U.S. Virgin Islands, let me know. Due to damage to the FAA communications and guidance systems, only a dozen or so commercial flights can land during daylight hours each day. Airlines report over 20,000 people on standby lists, and nearly 1,000 people waiting at the airport for any flight. The Port of San Juan is open, daylight hours only, and receiving freight barges. While there is a plenty of fuel, food and supplies at the port; getting truck drivers to the port and damage/blocked roads is slowing distribution of supplies to the rest of the island. U.S. Mail and other express delivery companies still do not have service in Puerto Rico. Limited U.S. Mail hand-out service is available at a few post offices in U.S. Virgin Islands.

On Tue, 26 Sep 2017, Sean Donelan wrote:

It looks like someone kicked the cellular carriers public relations people into gear. Today, instead of the normal "we care" messages; they released statements providing more concrete details about their restoration activity in PR and USVI.

What is the US government role in all of this? It sounds like a few https://en.wikipedia.org/wiki/Lockheed_C-5_Galaxy could be of use here to airlift in lots of gear. -- Mikael Abrahamsson email: swmike@swm.pp.se

Keep on posting this great info Sean. It is being passed along. Just wanted you to be aware. On Tue, Sep 26, 2017 at 12:52 AM, Sean Donelan <sean@donelan.com> wrote:

It looks like someone kicked the cellular carriers public relations people into gear. Today, instead of the normal "we care" messages; they released statements providing more concrete details about their restoration activity in PR and USVI.

Overall, 91.2% cell sites out of service in Puerto Rico. 34 of 78 counties have 100% cell sites out of service. This will continue to change up and down, as sites are restored and circuits are damaged by cleanup activity.

There are over 2,671 cell sites on Puerto Rico and 106 cell sites in U.S. Virgin Islands. As carriers bring in tens of generators and repair equipment at a time, gives you some idea how long restoration will take.

In alphabetical order...

ATT: "We continue to send aircraft with essential supplies and network resources as we help the people of Puerto Rico. These flights include portable temporary cell sites, high capacity generators to provide temporary power, and other larger network equipment on cargo planes and barges to help restore services on the island. We planning to set up a number of portable cell sites in the San Juan area as soon as possible.

So far, we’ve sent multiple flights carrying the following supplies: More than 30 generators 5,000+ gallons of water We are also focused on network restoration in the U.S. Virgin Islands are bringing additional resources there."

Claro (google translate from Spanish): They reported that in the metropolitan area specifically, Claro's signal was already reaching 31 percent of customers in San Juan, 22 percent in Guaynabo and 18 percent in Carolina and Bayamón.

At the island level, the Claro signal is up in 14 municipalities today, covering an average of 20 percent of the clients in Aguada, Manatí, Mayaguez, San Germán, Cabo Rojo, Trujillo Alto, Dorado, Camuy, Quebradillas, Humacao, Juncos , Caguas, Aguadilla and Toa Baja.

That number will increase in the coming days.

Sprint: "A vessel has already arrived in Puerto Rico with the generators and parts required to begin the work. In turn, a body of over 40 Sprint engineers and technicians in the United States were sent to the Island to join the local technical staff, coordinate the delivery of the equipment received and continue work to speed up the communication. A second shipment will arrive on the island this Wednesday, September 27 with additional spare parts and materials."

T-Mobile: "The damage to the infrastructure is unprecedented, but equally it is the support we are receiving from T-Mobile US. Between Saturday and Sunday, six MD11 cargo planes and one AM124 (second largest cargo plane in the world) arrived with 80 generators, 16 trucks, equipment to build 100 communication facilities. More cargo planes will arrive today with more equipment and personnel."

T-Mobile also mentions while T-Mobile's field engineering crew was at the Luis Muñoz Marín Airport, they were drafted to help install a generator for the FAA Control Tower. That's one way to help get your supplies on the island.

If you have information about other telecommunication providers in Puerto Rico or U.S. Virgin Islands, let me know.

Due to damage to the FAA communications and guidance systems, only a dozen or so commercial flights can land during daylight hours each day. Airlines report over 20,000 people on standby lists, and nearly 1,000 people waiting at the airport for any flight.

The Port of San Juan is open, daylight hours only, and receiving freight barges. While there is a plenty of fuel, food and supplies at the port; getting truck drivers to the port and damage/blocked roads is slowing distribution of supplies to the rest of the island. U.S. Mail and other express delivery companies still do not have service in Puerto Rico. Limited U.S. Mail hand-out service is available at a few post offices in U.S. Virgin Islands.

The telecommunications damage in PR and USVI will be a good test how well the EAS works during extreme telecommunications damage.

From my brief time as a radio station tech, all you need for EAS to function properly is power to the receiver/decoder and for the station's transmitter to be alive

On Wed, 27 Sep 2017, Sean Donelan wrote:

Things are better and worse in Puerto Rico and other Caribbean islands. Help is needed, but anyone wanting to help in the field, be certain you understand what you would be doing, and whether you are actually helping or hindering on the ground efforts.

https://www.vox.com/science-and-health/2017/9/26/16365994/hurricane-maria-20... This seems to indicate that it will be 4-6 months until things get back to normal, if there indeed is a huge effort to do so. "But as first responders on the ground in Puerto Rico told Fernández Campbell, this isn’t enough. Trump should also ask Congress to pass a relief package for Puerto Rico to give FEMA and the island more money to rebuild. He could deploy more military resources to help with search and rescue operations." I hope this happens. -- Mikael Abrahamsson email: swmike@swm.pp.se

On 2017-09-27 17:44, Sean Donelan wrote:

After a week without power, all the stationary batteries throughout the telecommunications network are likely completely drained.

from the point of view of cell sites, wouldn't battery autonomy be measured in hours rather than days? I could see some site having autonomy in days due to permanent generator, and when fuel runs out so does the cell site.

I'm not ignoring the status competitive and smaller USVI and PR communication providers, its just difficult to find official statements from them. If you have status about them, let me know.

One aspect often forgotten is that people have homes (or what is left of them) families and the need to find food/water which can involve standing in line for hours in a day and they may not be able to show up for work. larger companies can usually find enough employees not so hindered, but smaller outfits may not be able to remain functional due to not enough staff able to work. Smaller outfits may not have the ability to get petrol for their trucks to go out oand fix things. (whereas the big guys have the credentials to get petrol form authorities/army.

FYI: White House announces that the US Army Corp of Engineers is in charge of power in Puerto Rico, and were given priorities to hospitals and other emergency services. No mention of telecom being part of those priorities. Initial push is installing temporary power generation. They are not yet working on fixing the electrical grid. (44 of 69 hospitals now have power). (Note: FEMA has decided to stick to road deliveries, not air drops for supplies).

The WISP I'm getting updates from is having thefts as well. Still having logistics issues. The leading idea at the moment is tower-mounted solar panels and batteries. Nothing is foolproof without armed guards, but it's better than nothing. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Sean Donelan" <sean@donelan.com> To: nanog@nanog.org Sent: Wednesday, September 27, 2017 4:44:30 PM Subject: Re: Hurricane Maria: Summary of communication status - and lack of After a week without power, all the stationary batteries throughout the telecommunications network are likely completely drained. This makes restoration even more difficult, like a dead car battery needing a jump start. I am focusing on U.S. territories, but there is also disaster response from Hurricanes Irma and Maria on Antigua and Barbuda, Cuba, Dominica, Montserrat, Saint Martin, and St. Kitts and Nevis. Fatalities, including deaths attributed to post-hurricane recovery: Hurricane Iram: 72 - Florida; 40 - Caribbean Hurricane Maria: 16 - Puerto Rico; 2 - U.S. Virigin Islands; 15 - Dominica, 3 - Haiti; 2 - Guadeloupe Department of Defense: Supporting FEMA, the Department of Defense has deployed USNORTHCOM Brigadier General Rich Kim to Puerto Rico to manage the Title 10 (military) response efforts in Puerto Rico and U.S. Virgin Islands. USSOUTHCOM continues to support relief activities elsewhere in the Caribbean. Airports and sea ports: Puerto Rico: 3 sea ports open; 5 sea ports open with restrictions, daylight hours only. 9 airports are open. Only San Juan Airport open to commercial air traffic, approximately 15-20 commercial flights. All other flights reserved for priority military and relief activities. U.S Virgin Islands: 4 sea ports open with restrictions, daylight hours only. U.S. VI airports closed except military and relief flights. Electricity: Puerto Rico: 1.57 million customers out of service. An estimate of 4% has been restored. Restoring power to airports, hospitals, sea ports and water treatment plants are still critical priorities. 80% of transmission lines damaged, power generation plants appear intact. U.S. Virgin Islands: 55,000 customers out of service, most of the islands. St. Thomas has five feeders partially energised. St. Croix has three feeders partially energized. Restoring power to airports, hospitals, sea ports and water treatment plants are still critical priorities. Telecommunications: Pictures posted on twitter of joint restoration meeting between telecommunications providers, FEMA and Puerto Rico Telecommunications Regulatory Board. From the logos & colors on shirts: Claro, T-Mobile, Sprint, and many other company logos I couldn't make out (estimate 20 people in the room). Reports of generators and fuel stolen from cell sites and remote telecommunications locations. This is not unusual during disasters. The Puerto Rico Telecommunications Industry Alliance, which appears to be a lobbying group of communication companies in Puerto Rico, has sent a letter about the need for FEMA to coordinate logistics and prioritize access to fuel and security. PRTIA (or APT in Spanish) has existed for a few years, but I can't judge if its letter represents telecommunication companies in Puerto Rico. Puerto Rico: 2,432 of 2,671 cell sites (91%) out of service. No update/change to cable and wireline systems, about 55% of central offices with voice, data and long-distance. The rest with only local voice, no inter-office connections. No clear description about status of local loops or subscribers with service. Pictures of Liberty Cable PR repair crews posted on twitter. I still haven't found a public statement about LibertyPR's status. Approximately 450-500 out of 1200 Internet networks and 35-38 out of 48 ASNs are present in the global Internet routing table, with occasional up/down changes due to restoration activity. U.S. Virigin Islands: 70 of 106 cell sites (66%) out of service. No update/change to cable and wireline systems. U.S. Virgin Islands Internet routes have nearly returned to normal, with occasional up/down blips due to restoration activity. I'm not ignoring the status competitive and smaller USVI and PR communication providers, its just difficult to find official statements from them. If you have status about them, let me know.

Sean, thank you for all the excellent updates you have been providing. Status.pr is disturbing since there is no context to the stats offered on this page. 49% of supermarkets may be open, but with nothing on their shelves. And 11k refugees? Who are they trying to kid with a number like that. -Barb +1.808.385.1677 mauigrrl@earthlink.net Written on the move, apologies for any errors.

On Sep 29, 2017, at 8:15 AM, Sean Donelan <sean@donelan.com> wrote:

Career federal employees are taught to write situation reports in very boring language with just the facts known. Nevertheless, after reading lots of situation reports, you start to notice when the bubureaucratic language changes. Perhaps the most famous was the commander of Apollo 13's report "Houston, We have a problem."

Puerto Rico has announced a new web site with current status:

https://urldefense.proofpoint.com/v2/url?u=http-3A__status.pr_&d=DwIBAg&c=FmY1u3PJp6wrcrwll3mSVzgfkbPSS6sJms7xcl4I5cM&r=eFHwbDul3gCazAMQCZYPBUi5FR29U9pfCEZA3KSPp1U&m=8QGAW2zyikBvyqdqem1ufMWHN1wmpYs5CHOkKkgxHuY&s=zr44KzVhB4CMsDiVsjPo0RNdkIMb14m0WxW3UV60JYY&e=

However, in the last 24 hours I've noticed some agency situation reports used different statistics to report "happy, happy, joy, joy" stuff. In the bureaucratic world, this is very concerning, such as when the Veterans Administration was misreporting appointment waiting times to look better.

You can't fix problems, if the real situation isn't being reported accurately to senior leadership even if its bad news.

On 2017-09-29 23:07, Sean Donelan wrote:

I don't know what FCC and PRTRB are counting:

286 working cell sites out of 2671 (according to FCC report) 96 working cell sites out of 1600 (according to PR Telecommunications Regulatory Board report)

I had noticed the different numbers too. My speculation: The 1600 may refer to antenna sites, whereas the 2671 may be the sum of the number of sites reported by each carrier (think a mast supporting antennas from multiple carriers). Assuming my logic is correct, the 96/1600 statistic may be of more use in a "can I dial 911" point of view. Having multiple carriers "up" at the same tower doesn't increase geographic footprint where some coverage exists.

From a disaster management point of view, in a town where each carrier has its own tower, deciding which one to light up first could be interesting. (aka carriers getting together to compare state of antennas in town and somehow elevating that info to whoever controls the generators (army corps of engineers who are "foreigners" with no local knowledge).

The whole thing is a disgrace. ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Phil Rosenthal <pr@isprime.com> Sent: Saturday, September 30, 2017 3:47 PM To: Jean-Francois Mezei Cc: nanog@nanog.org Subject: Re: Hurricane Maria: Summary of communication status - and lack of Has anyone heard anything about Liberty Cablevision / AS14638? Our Netflow stats show a traffic drop to zero at the moment of landfall of Maria, late on 9/19, and a continued flat line at zero until now. Almost 11 days without a single packet exchanged. This is (as far as I am aware), the #2 largest ISP in Puerto Rico. By comparison, Claro’s traffic certainly has dropped by a large degree, but it always stayed at least slightly above zero, and is roughly at 10% of normal traffic levels today. -Phil

On Sat, 30 Sep 2017, Sean Donelan wrote:

The first public statement I've seen from LibertyPR was yesterday. Their network was completely down. They've restored some of their main infrastructure, i.e. cable headends and main fiber connections. 100% of subscribers are out of service.

I've seen pictures on twitter of LibertyPR crews fixing cables and poles on the island.

Liberty cable Puerto Rico has put out a press release today. LibertyPR is opening one public WiFi hot spot in Bahia Urbana in San Juan from 3pm to 7pm Saturday, and 8am to 7pm daily starting Sunday. Additional hot spots will be announced by LibertyPR via press release in the future. I guess this is a sign LibertyPR's public relations office is back in operation.

The more I read about this, the more disturbed I get. On the one hand, we keep hearing that the trucks aren't moving because roads are impassable. Then I read that government officials are driving from their remote areas to San Juan to ask why no aid is coming, disputing the claims about the roads. We hear that there isn't fuel for the trucks, then a reporter from CNBC disputes that claim as well. The only thing that seems to be a common thread is that there are massive amounts of supplies sitting in San Juan and that they can't get truck drivers to deliver them. Do FEMA and the National Guard have the authority to commandeer the trucks and deliver the containers themselves? The telcom companies aren't going to be able to do much by way of repairs without supplies. On Sun, Oct 1, 2017 at 9:28 PM, Javier J <javier@advancedmachines.us> wrote:

At this point, I wouldn't trust status.pr and any media reports without verifying information. As far as LibertyPR is concerned my cousin who lives in Carolina, PR told me thieves were stealing fiber optic cable after the storm. I trust the Seon Donelan, FCC, US Military, FEMA reports in that order. There was a report that 33% of cell phone service was reported. That is BS. We know from FCC reports it is still at ~90% out as far as number of operational cell sites.

The media here in the states is no better. I have multiple confirmations and am looking for hard proof but the Teamsters Puerto Rico trucking union is refusing to move containers out of the port. Only 20% of truckers showed up for work. Perhaps someone who works at Crowley can give us more concrete info but if you can't even move supplies out of the port, how the heck are you supposed to replace wires/fiber/fuel etc?

Here is a CNBC report: https://www.youtube.com/watch?v=f4Z01o4tBlI

- Javier

On Sat, Sep 30, 2017 at 4:39 PM, Sean Donelan <sean@donelan.com> wrote:

...

On Sat, 30 Sep 2017, Sean Donelan wrote:

...

The first public statement I've seen from LibertyPR was yesterday. Their network was completely down. They've restored some of their main infrastructure, i.e. cable headends and main fiber connections. 100% of subscribers are out of service.

I've seen pictures on twitter of LibertyPR crews fixing cables and poles on the island.

Liberty cable Puerto Rico has put out a press release today.

LibertyPR is opening one public WiFi hot spot in Bahia Urbana in San Juan from 3pm to 7pm Saturday, and 8am to 7pm daily starting Sunday.

Additional hot spots will be announced by LibertyPR via press release in the future.

I guess this is a sign LibertyPR's public relations office is back in operation.

Do FEMA and the National Guard have the authority to commandeer the trucks and deliver the containers themselves?

I hope they do. There doesn't seem to be a shortage of FEMA, Army, etc personnel on the ground or a shortage of truck drivers in the US willing to help. If 80% of Truck drivers that pick up containers from the ports can't make it, then this needs to be supplemented any way possible to get things moving. On Sun, Oct 1, 2017 at 11:09 PM, Jason Baugher <jason@thebaughers.com> wrote:

The more I read about this, the more disturbed I get. On the one hand, we keep hearing that the trucks aren't moving because roads are impassable. Then I read that government officials are driving from their remote areas to San Juan to ask why no aid is coming, disputing the claims about the roads. We hear that there isn't fuel for the trucks, then a reporter from CNBC disputes that claim as well. The only thing that seems to be a common thread is that there are massive amounts of supplies sitting in San Juan and that they can't get truck drivers to deliver them.

Do FEMA and the National Guard have the authority to commandeer the trucks and deliver the containers themselves? The telcom companies aren't going to be able to do much by way of repairs without supplies.

On Sun, Oct 1, 2017 at 9:28 PM, Javier J <javier@advancedmachines.us> wrote:

...

At this point, I wouldn't trust status.pr and any media reports without verifying information. As far as LibertyPR is concerned my cousin who lives in Carolina, PR told me thieves were stealing fiber optic cable after the storm. I trust the Seon Donelan, FCC, US Military, FEMA reports in that order. There was a report that 33% of cell phone service was reported. That is BS. We know from FCC reports it is still at ~90% out as far as number of operational cell sites.

The media here in the states is no better. I have multiple confirmations and am looking for hard proof but the Teamsters Puerto Rico trucking union is refusing to move containers out of the port. Only 20% of truckers showed up for work. Perhaps someone who works at Crowley can give us more concrete info but if you can't even move supplies out of the port, how the heck are you supposed to replace wires/fiber/fuel etc?

Here is a CNBC report: https://www.youtube.com/watch?v=f4Z01o4tBlI

- Javier

On Sat, Sep 30, 2017 at 4:39 PM, Sean Donelan <sean@donelan.com> wrote:

...

On Sat, 30 Sep 2017, Sean Donelan wrote:

...

The first public statement I've seen from LibertyPR was yesterday. Their network was completely down. They've restored some of their main infrastructure, i.e. cable headends and main fiber connections. 100% of subscribers are out of service.

I've seen pictures on twitter of LibertyPR crews fixing cables and poles on the island.

Liberty cable Puerto Rico has put out a press release today.

LibertyPR is opening one public WiFi hot spot in Bahia Urbana in San Juan from 3pm to 7pm Saturday, and 8am to 7pm daily starting Sunday.

Additional hot spots will be announced by LibertyPR via press release in the future.

I guess this is a sign LibertyPR's public relations office is back in operation.

Well, that's why recovery efforts in broad scale events like this have to go from a central point to pushing a perimiter farther and farther out. Create a habital, functional zone where workers can return to both to organize and recouperate and then go back out and push farther afield. First restoring main arteries (whether that is in the form of roads, electrical dstribution, communications, water, or sewer) and then branch out from there. All of that takes time. It does no good, afterall, to repair the services in a neighborhood if the feeds into that neighborhood aren't going to be functional for weeks. And always remember that the first duty is to life and limb. The rest is of far less importance until that situation has been stabilized. On Mon, Oct 02, 2017 at 12:56:56AM -0400, Jean-Francois Mezei wrote:

On 2017-10-02 00:32, Javier J wrote:

...

I hope they do. There doesn't seem to be a shortage of FEMA, Army, etc personnel on the ground or a shortage of truck drivers in the US willing to help. If 80% of Truck drivers that pick up containers from the ports can't make it, then this needs to be supplemented any way possible to get things moving.

When disaster is in focused area (Like Houston), truck drivers can easily return to functional cities after delivering goods to the diaster zone (so not a strain on food/lodging in diaster zone).

If you bring truck drivers (and telecom, electrical etc) workiers into Puerto Rico, they can't go home every night, so become a strain on shelter/food resources.

And you can't "steal" your local workers if they are busy pickup up their belongings from collapsed homes, waiting in long queues for food and caring for their families.

In 1998 Ice Storm, Bombardier in Montr??al had full power and got a lot of bad publicity when it threatened to fire employees who didn't show up for work. Seesm like mamnagement lived in areas that had power and didn't realise how life changes when you have no power, queue up for wood provided by city etc. (and that is nothing compared to what people on Puerto Rico are dealing with).

--- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/

On Sun, 01 Oct 2017 22:09:55 -0500, Jason Baugher said:

The more I read about this, the more disturbed I get. On the one hand, we keep hearing that the trucks aren't moving because roads are impassable. Then I read that government officials are driving from their remote areas to San Juan to ask why no aid is coming, disputing the claims about the roads.

It's quite possible for both to be true. The fact that some government officials are able to make it to San Juan isn't proof that nobody on the island is stuck behind an impassable road.

On Sun, 01 Oct 2017 22:28:31 -0400, Javier J said:

The media here in the states is no better. I have multiple confirmations and am looking for hard proof but the Teamsters Puerto Rico trucking union is refusing to move containers out of the port. Only 20% of truckers showed up for work.

I haven't seen any reports of a Teamster union refusal. I *have* seen reports that only 10-30% of truck drivers are operational, because of one or more of: 1) Their rigs are stuck behind a highway outage due to washout or downed trees. 2) Their rigs are stuck due to lack of diesel. 3) Rigs are fine, but drivers are stuck due to 1) or 2), or they are too busy trying to save their families/etc to show up to work. I've seen too many reports of "I have been waiting in line for 5 hours for water / gasoline / ice for <relative>'s insulin" to get too irate at people who fail to show up for work.

On Mon, 2 Oct 2017, Sean Donelan wrote:

On Sun, 1 Oct 2017, valdis.kletnieks@vt.edu wrote:

...

I haven't seen any reports of a Teamster union refusal. I *have* seen reports that only 10-30% of truck drivers are operational, because of one or more of:

You're lucky. The bots have been pushing this very hard for several days. I don't know the local context or groups involved. There are at least two different groups

Snopes has published an article debunking this. But bots are still pushing it. Did Puerto Rico's Teamsters Union Go on Strike During Hurricane Maria Relief Efforts? Reports that truck drivers are on strike in Puerto Rico are false -- Teamsters have asked mainland truckers to distribute supplies in the U.S. territory. http://www.snopes.com/puerto-rico-teamsters/

After two weeks it appears the situation is stabilizing (not getting worse) on Puerto Rico and U.S. Virgin Islands. But recovery and logistics still seems very slow in both territories. A reminder, I am focusing on U.S. Territories, but other Caribbean islands are still recovering from Hurricane Irma and Maria. Fatalities Puerto Rico: 16 storm related (last report Sept. 27) Media estimate at least 60 storm related deaths CDC mortatility rate for PR: average 80 deaths per day all causes U.S. Virgin Islands: 5 storm related (last report Oct. 3) 30 deaths from all causes (natural causes, accidents, homicides) Telecommunications Satellite phones Iridium reports over 2,000 non-military satellite phones active, normally less than 10 non-military satellite phones active in Puerto Rico and U.S. Virgin Islands area. Iridium does not release military usage. Landline phones 813,546 landlines (CIA World Factbook) ILEC (Claro) reports all Central Offices have voice, data and long distance working. Estimate 40% of landline subscriber local loops are in service, mostly in metro areas. CLECs - no reports Cable - all cable subscribers currently out of service Wireless mobile phones 3,227,281 (CIA World Factbook) AT&T, Claro and T-Mobile announced extension of "open roaming" agreement between networks. I expect Sprint and Open Mobile will also extend their participation. Because open roaming makes accurate billing impossible, all carriers also announced they are waiving charges in Puerto Rico. Puerto Rico Telecommunications Regulatory Board announced carriers have continued their joint restoration agreement. PRTRB expects 60% restoration of service by end of October, and 100% restoration of service before Christmas. I can not evaluate this forecast, but it seems aggressive; or the level of service will be the bare minimum and capacity will be very congested. Claro reports it has at least one cell site active in 28 of 78 municipalities, covering about 310,000 subscribers. It may not be high-quality service, but its some service. AT&T, Open Mobile, Sprint and T-Mobile have not disclosed how much of their networks are operating. AT&T stated it is carrying 8 million calls and 4 million texts per day. FCC reports 2359 out of 2671 (88%) cell sites on Puerto Rico are out of service. PRTRB reports 1254 out of 1619 (77%) cell sites on Puerto Rico are out of service. I still do not understand the different statistics being reported by FCC and PRTRB or how they calculate their statistics. ROK Mobile and M2Catalyst, mobile metrics platforms, have published estimates of cell tower damage in Puerto Rico Claro: 14% cell sites operating Open Mobile: 8% cell sites operating Extended Network: 7% cell sites operating T-Mobile: 31% cell sites operating CLARO|TELCEL: 6% cell sites operating AT&T: 18% cell sites operating Percentages are affected by the denominators, i.e. share, which wasn't released. But it does show all carriers experienced a lot of damage. http://www.prnewswire.com/news-releases/presently-over-86-of-cell-sites-in-p... American Tower, Crown Castle and SBA, which collectively own many physical towers leased by mobile carriers on Puerto Rico, report little material structural damage to the towers themselves. But there was substantial damage to carrier customer equipment on the towers and lack of electric power at almost all towers. Note: there are twitter photos of at least two collapsed towers, but I don't know those tower owners. U.S. Virgin Islands has almost no change (or getting slightly worse) in number of working cell sites during the last week. Internet Services 647 IP networks out of 1205 are routed from Puerto Rico (RIPE) 66 IP networks out of 70 are routed from U.S. Virgin Islands (RIPE)

got curious about the FCC's definition of "cell site" in the Maria outages reports in Puerto Rico. In the Oct 4 report: Arecibo is reported as having 68 cell sites served, 65 being out. (95.2% outage) The FCC has an "ASR" (Antenna Structure Registration) search for cell sites, and this points to actual masts (which I assume need some permit above certain height). For ARECIBO, there are 31 entries, 1 dismantled, 4 granted 2 cancelled That leaves 24 "constructed". These registrations do not mention which carrier(s) uses the mast. And include some owners such as Caribbean Broadcasting Corporation which isn't likiely being used for cellular. For all of Puerto Rico, it reports 930 ASR registrations. (haven't done the parsing to see how many are "Constructed" vs Cancelled, granted, dismantled). Lets assume 900 for sake of discussion. So the ~1600 quoted by another organisation would have to include more than just registered antenna masts. Except for water towers, what other structures would be amenable to having multiple carrier's antennas? What is also not clear from such statistics is the fact you could have a town with an high antenna broadcasting 850 to the whole area, and then lots of DAS antennas at telephone pole height in the town at 1900 or 1700. Having the 850 up and running at the top of the hill might cover the whole town, even if it would represent only 1 of say 50 cell sites in the area. Similarly, covering a windy road in a canyon might be done with lots of DAS anetnnas on telephone poles along the way. They may all be down, but would normally serve 0 population, so is this number of "down" antennas relevant? During the 1998 ce storm in Québec, Hydro Québec was overwhelmed and asked cities to identify priority sites inside their territories. It's fancy "point to where the break is based on where everyone reports an outage" software was useless because many breaks continued to happen after power had been lost. So it had to start from where there was power and work its way, fixing breaks along the line towards those priority sites. (and once done, fan out from there to power the non priority areas). In many rural areas, this involved planting new poles for long distances, rebuilding from scratch. (And only once the poles are up can the telco restring its wiring). What the media doesn't show after a disaster is what is still standing, what is still working. It could be that a large portion of telephone poles are still standing and intact and only require minor individual fixes. Or it could be that large swaths ave seen the poles toppled and new ones needed with new power and telco wiring done from scratch. Statistics may look bad showing 100,000 without power. But if it is a single break by a branch it is easy to fix compared to having 1000 breaks by 1000 branches. So again, statistics don't give the full story on the real extent of damage.

On Thu, 5 Oct 2017, Jean-Francois Mezei wrote:

Statistics may look bad showing 100,000 without power. But if it is a single break by a branch it is easy to fix compared to having 1000 breaks by 1000 branches. So again, statistics don't give the full story on the real extent of damage.

The FCC is a passive entity collecting only the outage details service providers choose to voluntarily share. Data about cell sites is from the Wireless Resiliency Cooperative Framework and any other wireless carriers which choose to provide data. https://www.fcc.gov/wireless-resiliency-cooperative-framework https://ecfsapi.fcc.gov/file/60001707365.pdf This is the data collected by DIRS: Wireline Carriers CLLI code of switches/STPs that are out Names of PSAPs that are out Estimated users out of service Major facilities out of service (> 192 DS3) Very short status description of the effects of event (e.g., “No major equipment out”) Wireless Carriers CLLI code of MSCs/STPs that are out Total cell sites out in disaster area Very short status description

I have not ound the official announcements, but the press is reporting that the FCC has granted Google rights to fly 30 of its "Loon" high altitude ballons to provide cellular cervice in Puerto Rico for up to 6 months. (From my readings, there are glorified relays of ground based signals (which I assume some antennas have to be oriented to face up towards the balloons). The Loon will use spectrum allocated to the carriers they relay (and got their OK) Altitude 20km. (so not sure they need 30 balloons, 1 probably suffices to cover all of PR). I suspect more concrete info will be coming.

From a WISP in USVI

A quick perspective from the US Virgin Islands of how the carriers have fared / performed: AT&T = had a couple towers with some cell coverage after Irma and Maria. A testament to good engineering at the tower, and redundancy in their network design. Primarily microwave backhaul, but leasing some fiber from the ILEC named Viya. AT&T has a major undersea cable station and POP on STT in downtown Charlotte Amalie. They have been making progress fixing their network, STX is over 50% fixed 2 weeks after Maria. 75% market share Sprint = 100% down for the 3+ weeks after Irma. They have a single point of failure, relying on 10ft dishes to shoot 20-50 miles, from STT to Puerto Rico. These cheap bastards wouldn’t buy a backup connection from Viya or Broadband VI. I have called them out to the PSC. Still weeks away from anything working. Most of their customers can roam on Viya’s cell network. 15% market share and rapidly declining. Viya = Celluar = 30-50% up, Celluar = 10% market share. Rolling out LTE upgrade. Cable TV/Phone/Internet = 10% up, 75% market share, have a long road to recovery. Have to wait for power company poles to be replaced / fixed before they can repair their badly damaged plant. Broadband VI = WISP = 50% AP's up, 15% of customers. Got up quickly after Irma, STX stayed up, STT had backhaul to every major tower repaired in 5 days. After Maria 100% down. Had to re-aim / repair every major tower on STX, and most of STT. Moving focus from backhaul to repairing AP’s next week. Tower by tower, with installers / subs going to customers in that area (who have power, almost all via generator). In the middle of a Mikrotik 2 Cambium 450 forklift upgrade. Impressive survival rate for Cambium AP’s, and Ceragon IP-20. viNGN = Government fiber middle-mile, lost 90% of their drops because there were aerial. I am off to guide the FEMA re-fuelers to a remote tower which ran out of fuel last night. There have been some lessons learned. I will compile a report in the next few weeks. Mike Meluskey CTO and Founder Broadband VI ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: Javier J <javier@advancedmachines.us> To: Jean-Francois Mezei <jfmezei_nanog@vaxination.ca> Cc: nanog@nanog.org Sent: Sat, 07 Oct 2017 03:02:46 -0500 (CDT) Subject: Re: Hurricane Maria: Summary of communication status - and lack of @ Jean Interesting stuff. Please keep this thread updated with info on that initiative. On Fri, Oct 6, 2017 at 9:55 PM, Jean-Francois Mezei < jfmezei_nanog@vaxination.ca> wrote:

I have not ound the official announcements, but the press is reporting that the FCC has granted Google rights to fly 30 of its "Loon" high altitude ballons to provide cellular cervice in Puerto Rico for up to 6 months.

(From my readings, there are glorified relays of ground based signals (which I assume some antennas have to be oriented to face up towards the balloons).

The Loon will use spectrum allocated to the carriers they relay (and got their OK)

Altitude 20km. (so not sure they need 30 balloons, 1 probably suffices to cover all of PR).

I suspect more concrete info will be coming.

From what I have read, a great deal of the operating infrastructure is operating on backup generator. These generators are not meant for this duty cycle. (Recall that most units are sized such that they will be

Please note that there is another looming problem with restoration of services generally (not just telecommunications). The key here is the power grid. providing ~70% output if not higher and thus will run hard.) It will not be long before some of them begin to fail. Even if they can keep running for the longer term, they need to be shut down every so many hours for service (oil change, etc.) Depending on the unit, that may be measured in the hundreds of hours. One week is 168 hours. One month is 720 hours. Fail to do this and the unit evntually becomes a big pile of scrap metal. Any facility, beit a pumping station, hospital, airport, cell tower, central office, or sewage plant that must rely on generators for the foreseeable future must consider this. On Tue, Oct 10, 2017 at 12:47:21AM -0400, Sean Donelan wrote:

The Puerto Rico government has posted threee maps of cellular coverage and GPS coordinates of Cells on Wheels (COWs) in service.

http://www.status.pr/Maps/

It still looks grim in Puerto Ricofrom a telecommunications perspective. Its will be an interesting after-action study. Other than "it was a hurricane," I haven't gotten a good idea why so much of the telecommunications network failed and backups still aren't working more than 2 weeks later.

Claro, the ILEC but second in terms of mobile phone marketshare behind AT&T, has started to more fully explain what "restored" means, and that it doesn't mean everything as before the hurricane. It is minimum telecommunications. Claro has been more willing to talk about the situation in Puerto Rico, which is why I've referencing Claro a lot more than other carriers.

This is a google translate of an interview from spanish.

"It is important to clarify that the radio bases put into service to date, offer the same voice and data services as before the impact of the Hurricane. In other words, if the base radio is 4GLTE, that is the service it will offer. The other two components that influence the customer experience are the voice and data plan and the equipment of each user."

"The network is also open to third-party customers as part of our commitment to connect everyone in the country. In fact, over a quarter of a million customers from other providers have connected daily to the Claro network. When these customers connect to our network they only have voice service as stipulated in the roaming agreement with the other providers. As for the fixed network, this morning the service was restored in the central offices (OC) of Fajardo and Humacao, whose optical fibers had been affected by the destruction of Hurricane Maria. In this way already have fixed voice, internet and long distance services in these municipalities: Ceiba, Fajardo, Luquillo, Humacao, Naguabo and Yabucoa. Already a total of 57 municipalities have all 3 services. It is possible that some customers of Claro served by these OCs do not have internet. This is possible as there could be cables and posts broken and / or VRADs without AEE service."

https://www.metro.pr/pr/noticias/2017/10/06/senal-claro-esta-ya-accesible-34...

--- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/