This is what I tell outage complainers during natural disasters, such as the fires in California that recently took out a lot of power and communications: “Stop whining about how long it is taking to repair your Internet, your cell phone service, or your cable TV. You didn’t pay anything extra to recover from natural disasters, and none of us in the field are getting paid anything extra to restore your services. No, we don’t know how long it will take. It takes what it takes. That you don’t get instant gratification doesn’t make us incompetent. It makes you ungrateful. It’s a natural disaster. These are not scheduled. Your outage is nobody’s fault. We don’t have a duty to mitigate all conceivable failures. It takes time to repair. We’re not cheating you, or loafing around. We don’t owe you any special attention because of your status or reputation. So quit whining and be thankful you’re alive, and hopefully you haven’t lost too much. Maybe pitch in and help those who have.“ I also send this to ignorant journalists and grandstanding politicians. -mel via cell On May 11, 2019, at 4:29 AM, Mike Bolitho <mikebolitho@gmail.com<mailto:mikebolitho@gmail.com>> wrote: Trying not to get political, here goes... Something important to keep in mind: The current administration has been getting slammed for their lack of response in the aftermath of Michael since the hurricane hit. A lot of that criticism revolves around communications infrastructure and FEMA's lack of assistance. The current administration has, time and time again, used federal agencies (specifically their presidential appointees) to defend the administration's actions or inactions. I have read the full report and it is more or less a thinly veiled hit piece. I'm not going to link them here (they are easy enough to find via Google) but there are several very good articles written by reputable tech journalists that go into greater detail responding to the report. Worth checking out. I say all of that because most of us like to hate on telecom companies (many times rightly so) but I don't think they are entirely to blame here. There's nothing Verizon or AT&T can do if their backhaul is cut by a tree or some third party clean up crew. The report is a gross oversimplification of how telecommunication infrastructure works. I think anyone here that has ever worked a storm like this can attest to the complexity and difficulty you run into during recovery. Hanlon's Razor and all but this is the FCC and I would hope they would know better. Speaking specifically to point 51, it's impossible to coordinate between the thousands of crews working to clean things up and repair physical infrastructure after a massive storm like this. Many of the people doing physical cleanup are volunteers that are fully independent of any governing body or company. It is not a telco's responsibility to know when and where those crews are working. Further, even if those crews we're calling in and letting each telco know exactly where they were, what does that provide other than an impossibly large and fluid dataset to parse for any meaningful information. - Mike Bolitho On Thu, May 9, 2019, 4:43 PM Sean Donelan wrote: The FCC has released its report and analysis of Hurricane Michael impact on communications: preparation, effect and recovery. https://www.fcc.gov/document/fcc-releases-report-communication-impacts-hurri... Conclusions and Recommendations 51. Backhaul outages loomed large as an impediment to communications recovery. Uncoordinated post-storm recovery efforts between and among communications, utility, and debris removal teams created unnecessary delays to a speedy return to service. Customers who had communications service restored – only to lose it again almost immediately because of a fiber cut – provide a clear example of how better cross-sector coordination could have improved the restoration process.

In Florida, especially the panhandle, it's not possible to bury it. The water table is way too high. On Mon, May 13, 2019, 9:47 PM <frnkblk@iname.com> wrote:

This webinar may be of some interest to those in this group:

https://www.fcc.gov/small-rural-communications-provider-network-resiliency-w...

Here’s some additional color commentary on the FCC’s concerns:

https://urgentcomm.com/2019/05/10/backhaul-problems-disjointed-recovery-effo...

"“Uniti Fiber (Uniti) provides backhaul services to Verizon Wireless in Bay and Gulf Counties. Uniti indicates it experienced at least 33 separate fiber cuts during the recovery effort. These fiber cuts included damage to sections that already had been repaired. Commenters attributed fiber cuts to debris-removal crews, power-company restorations, and returning homeowners clearing their property.”

One of my takeaways from that article was that burying fiber underground could likely have avoided many/most of these fiber cuts, though I’m not familiar enough with the terrain to know how feasible that is.

Frank

*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Mel Beckman *Sent:* Saturday, May 11, 2019 9:52 AM *To:* Mike Bolitho <mikebolitho@gmail.com> *Cc:* nanog@nanog.org *Subject:* Re: FCC Hurricane Michael after-action report

This is what I tell outage complainers during natural disasters, such as the fires in California that recently took out a lot of power and communications:

“Stop whining about how long it is taking to repair your Internet, your cell phone service, or your cable TV. You didn’t pay anything extra to recover from natural disasters, and none of us in the field are getting paid anything extra to restore your services.

No, we don’t know how long it will take. It takes what it takes. That you don’t get instant gratification doesn’t make us incompetent. It makes you ungrateful.

It’s a natural disaster. These are not scheduled. Your outage is nobody’s fault. We don’t have a duty to mitigate all conceivable failures.

It takes time to repair. We’re not cheating you, or loafing around. We don’t owe you any special attention because of your status or reputation.

So quit whining and be thankful you’re alive, and hopefully you haven’t lost too much. Maybe pitch in and help those who have.“

I also send this to ignorant journalists and grandstanding politicians.

-mel via cell

On May 11, 2019, at 4:29 AM, Mike Bolitho <mikebolitho@gmail.com> wrote:

Trying not to get political, here goes...

Something important to keep in mind: The current administration has been getting slammed for their lack of response in the aftermath of Michael since the hurricane hit. A lot of that criticism revolves around communications infrastructure and FEMA's lack of assistance. The current administration has, time and time again, used federal agencies (specifically their presidential appointees) to defend the administration's actions or inactions. I have read the full report and it is more or less a thinly veiled hit piece. I'm not going to link them here (they are easy enough to find via Google) but there are several very good articles written by reputable tech journalists that go into greater detail responding to the report. Worth checking out.

I say all of that because most of us like to hate on telecom companies (many times rightly so) but I don't think they are entirely to blame here. There's nothing Verizon or AT&T can do if their backhaul is cut by a tree or some third party clean up crew. The report is a gross oversimplification of how telecommunication infrastructure works. I think anyone here that has ever worked a storm like this can attest to the complexity and difficulty you run into during recovery. Hanlon's Razor and all but this is the FCC and I would hope they would know better.

Speaking specifically to point 51, it's impossible to coordinate between the thousands of crews working to clean things up and repair physical infrastructure after a massive storm like this. Many of the people doing physical cleanup are volunteers that are fully independent of any governing body or company. It is not a telco's responsibility to know when and where those crews are working. Further, even if those crews we're calling in and letting each telco know exactly where they were, what does that provide other than an impossibly large and fluid dataset to parse for any meaningful information.

- Mike Bolitho

On Thu, May 9, 2019, 4:43 PM Sean Donelan wrote:

The FCC has released its report and analysis of Hurricane Michael impact on communications: preparation, effect and recovery.

https://www.fcc.gov/document/fcc-releases-report-communication-impacts-hurri...

Conclusions and Recommendations

51. Backhaul outages loomed large as an impediment to communications recovery. Uncoordinated post-storm recovery efforts between and among communications, utility, and debris removal teams created unnecessary delays to a speedy return to service. Customers who had communications service restored – only to lose it again almost immediately because of a fiber cut – provide a clear example of how better cross-sector coordination could have improved the restoration process.

There's more to it than this too. I was down there (I have sites I'm responsible for in Panama City Beach) in February and I was talking to a bunch of folks in the area as a result. This storm was fairly unusual for the area for a number of reasons. One, it normally doesn't hit the panhandle at anywhere near a category 5, and two, it was still a high category 3 by the time it hit Georgia. The amount of damage done was immense, is still not cleaned up (I drove past multiple buildings that were still piles of rubble, 4 months after the storm), and I was seeing forests full of damaged and destroyed trees all the way to I-10. All in all, the vast majority of Panama City looked much more like 4 months after a tornado rather than a hurricane, and all that damage continued all the way into Georgia. Thinking this was just like any other hurricane to hit the area is the absolute wrong tack to take - from what I heard there was some discussion of whether it was worth it to reopen Tyndall AFB, because the only thing left standing was some WWII era bomb-proof concrete hangars. On the flip side, improvements in response are a good thing - as long as people aren't beating up on the people who did the responding in the first place without cause. On Tue, May 14, 2019 at 9:52 AM Rich Kulawiec <rsk@gsp.org> wrote:

On Mon, May 13, 2019 at 11:48:02PM -0500, frnkblk@iname.com wrote:

...

One of my takeaways from that article was that burying fiber underground could likely have avoided many/most of these fiber cuts, though I???m not familiar enough with the terrain to know how feasible that is.

I suspect that may not be possible in (parts of) Florida.

However, even in places where it's possible, fiber installation is sometimes miserably executed. Like my neighborhood. A couple of years ago, Verizon decided to finally bring FIOS in. They put in the appropriate calls to utility services, who dutifully marked all the existing power/cable/gas/etc. lines and then their contractors (or sub-sub-contractors) showed up.

The principle outcome of their efforts quickly became clear, as one Comcast cable line after another was severed. Not a handful, not even dozens: well over a hundred. They managed to cut mine in three places, which was truly impressive. (Thanks for the extended outage, Verizon.) After this had gone on for a month, Comcast caught on and took the expedient route of just rolling a truck every morning. They'd park at the end of the road and just wait for the service calls that they knew were coming. Of course Comcast's lines were not the only victims of this incompetence and negligence. Amusingly, sometimes Verizon had to send its own repair crews for their copper lines.

There's a lot more but let me skip to the end result. After inflicting months of outages on everyone, after tearing up lots of lawns, after all of this, many of the fiber conduits that are allegedly underground: aren't.

---rsk

The majority of people doing locates are terrible at their job. (Un)fortunately, people doing the conduit installations are often terrible at their job as well. It's about a 50/50 split if the line was located correctly and the installation crew was careless or the line wasn't located correctly in the first places. Sometimes lines can be off by 10 feet. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Rich Kulawiec" <rsk@gsp.org> To: nanog@nanog.org Sent: Tuesday, May 14, 2019 8:51:13 AM Subject: Re: FCC Hurricane Michael after-action report On Mon, May 13, 2019 at 11:48:02PM -0500, frnkblk@iname.com wrote:

One of my takeaways from that article was that burying fiber underground could likely have avoided many/most of these fiber cuts, though I???m not familiar enough with the terrain to know how feasible that is.

I suspect that may not be possible in (parts of) Florida. However, even in places where it's possible, fiber installation is sometimes miserably executed. Like my neighborhood. A couple of years ago, Verizon decided to finally bring FIOS in. They put in the appropriate calls to utility services, who dutifully marked all the existing power/cable/gas/etc. lines and then their contractors (or sub-sub-contractors) showed up. The principle outcome of their efforts quickly became clear, as one Comcast cable line after another was severed. Not a handful, not even dozens: well over a hundred. They managed to cut mine in three places, which was truly impressive. (Thanks for the extended outage, Verizon.) After this had gone on for a month, Comcast caught on and took the expedient route of just rolling a truck every morning. They'd park at the end of the road and just wait for the service calls that they knew were coming. Of course Comcast's lines were not the only victims of this incompetence and negligence. Amusingly, sometimes Verizon had to send its own repair crews for their copper lines. There's a lot more but let me skip to the end result. After inflicting months of outages on everyone, after tearing up lots of lawns, after all of this, many of the fiber conduits that are allegedly underground: aren't. ---rsk

On 5/15/19 7:10 PM, Brandon Martin wrote:

I dunno how the big guys get away with it.  If I hit something, you can darn well bet someone's going to be on my neck immediately to shut the job down and pull my bond if possible.

It helps when the people in the field are like 3 subcontractors removed.