On Wed, 20 Sep 2017, Daniel Brisson wrote:

“Strongest storm of the century” just hit San Juan.

The number of reachable networks in Puerto Rico is down by 50%. Puerto Rico still has connectivity to the island, but outside facilities and electrical grid is being damaged by Hurricane Maria (Cat 4).

the entire island is now without power: http://www.bbc.co.uk/news/world-latin-america-41340392 <http://www.google.com/url?q=http%3A%2F%2Fwww.bbc.co.uk%2Fnews%2Fworld-latin-america-41340392&sa=D&sntz=1&usg=AFQjCNEYKsHT4Y3MS40bMMoPBLC0X9-DMg> no bueno. t On Wed, Sep 20, 2017 at 1:36 PM, Mehmet Akcin <mehmet@akcin.net> wrote:

There is a major outage going on in Puerto Rico and you can see it here -

https://stat.ripe.net/PR#tabId=routing

I am putting together some analysis as time passes - i will publish them in a blog and share.

On Wed, Sep 20, 2017 at 5:45 AM, Sean Donelan <sean@donelan.com> wrote:

...

On Wed, 20 Sep 2017, Daniel Brisson wrote:

...

“Strongest storm of the century” just hit San Juan.

The number of reachable networks in Puerto Rico is down by 50%.

Puerto Rico still has connectivity to the island, but outside facilities and electrical grid is being damaged by Hurricane Maria (Cat 4).

On Wed, 20 Sep 2017, Javier J wrote:

How long usually till generators at cell sites run out of juice?

Rough, every provider is different, backup power hierarchy: Neighborhood pole boxes: 1-4 hours, batteries only. May be re-charged with portable generators when safe to access area. There is likely severe physical damage to neighborhood lines. Cell towers: 8-12 hours battery. Some, not all, towers have a natural gas generator or 24 hours diesel generator Central offices and cable headends: 8-12 hours battery, 1-3 days diesel generators. Core, tandem, and hub sites usually have more backup. Major colocation data centers: <1 hour battery, 3-14 days diesel generators Submarine cable landing points and satellite control stations: 24 hours battery, 30 days diesel generators

Things are getting only worst so far - most of the island is offline - see the screenshot or the link here live https://stat.ripe.net/PR#tabId=routing On Wed, Sep 20, 2017 at 8:29 PM, Javier J <javier@advancedmachines.us> wrote:

Thank you for this info!

I think most of us kind of know there are backup power strategies in place but this is very detailed and appreciated. The little communication I have had with family on the island they tell me no internet, no cable tv, etc so this timing is good to know for when the few cell towers that survived start to go dark.

- J

On Wed, Sep 20, 2017 at 4:36 PM, Sean Donelan <sean@donelan.com> wrote:

...

On Wed, 20 Sep 2017, Javier J wrote:

...

How long usually till generators at cell sites run out of juice?

Rough, every provider is different, backup power hierarchy:

Neighborhood pole boxes: 1-4 hours, batteries only. May be re-charged with portable generators when safe to access area. There is likely severe physical damage to neighborhood lines.

Cell towers: 8-12 hours battery. Some, not all, towers have a natural gas generator or 24 hours diesel generator

Central offices and cable headends: 8-12 hours battery, 1-3 days diesel generators. Core, tandem, and hub sites usually have more backup.

Major colocation data centers: <1 hour battery, 3-14 days diesel generators

Submarine cable landing points and satellite control stations: 24 hours battery, 30 days diesel generators

On Thu, 21 Sep 2017, Colin Johnston wrote:

one ripe atlas probe is still green though

It looks like the last Internet service provider to Puerto Rico just went down. Zero routes. Hopefully its just power, and they will be able to re-fuel and be back on line quickly.

During the 1998 ice storm, Hydro Québec stated its infrastructure had not been built to widthstand this once in 100 year event. Reporters did some research and the next day asked him if there was a trend in increased freezing rain events. "I'll have to look into it". The next day, the HQ CEO came back at the daily press conference to confirm a gradual increase in last 20 years in freezing rain events, and after looking at situation, HQ would change standards for its infrastructure to widstand more frequent freezing rain events. In Ontario, the govt passed new stronger standards for utility poles which while granfathering existing ones, required the new standards apply before you can add one more wire to a pole. This seemed innocusous until telcos (Bell and smaller ones) started to want to add fibre to poles, where, in many cases, poles had to be replaced at $30k a shot, and original owner retained onwership of new pole paid by the telco. During the same event, Bell Canada, whose disaster plans were overwhelmed by the extent of power outages didn't have enough mobile generators to keep every outdoor plant's batteries charged all the time. As a result many areas suffered rolling POTS and cellular blackouts until a truck could there there with a generator. Because of the extent of the event, Bell couldn't bring spare generators from the next town over because that town was also in short supply of generators. When the nature of disruptive weather events changes (or become more frequent), utilities needs to adapt by adding more resiliency to physical infrastructure and being prepared with more spare hardware to cope with the aftermath. Hurricanes have the advantage of giving a few days warning and predictions are becoming more accurate. In the case of Irma, utilises have the time to pre-position trucks/equipment so they can kick in as soon as winds/flooding go down. In the case of Hydro Québec, their own statistics showed significant long term increase in freezing rain events, so easy to justify spending money to upgrade infrastrtucture. In the case of recent hurricanes, it is still debatable whether those were unusual events (since many towns had not experienced such striong weather for over 50 years) or whetgher frequencty of such events was going to increase. This would affect how telcos plan how resilient their infrastructure needs to be.