Northern California fires and telecomm outages
Hello fellow operators, The recent northern California fires wiped us out along with many other service providers and services including cellular, landline, e-911, cable and the like and I feel it's of operational concern the fragility of the telecommunications networks that link us together and the experiences had during this most recent event. I hope this topic is welcome and I apologize in advance for length. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=- I am the founder and owner of an Internet Service Provider business in Mendocino County, California and I also operate a CLEC (Competitive Local Exchange Carrier, a type of telecommunications company akin to ATT or Frontier), also in Mendocino County, California. The recent Northern California fires were again a direct and upfront demonstration of the potential for unanticipated widespread public disaster and again demonstrated how vulnerable we all are. Especially, the rising dependence on broadband service, and an expectation by end users that these services which are critical to them will continue functioning during emergencies as they may have no other access. There was a complete and total breakdown in telecommunication services due to fire which wiped out a primary telecommunications cable, and with it, e-911, land lines, cellular, cable and other telecommunication services. I hope that there will be a more complete public investigation and report on this latest incident, but in the mean time I would like to just offer what I know and to suggest where we could go from here. In the early morning of October 9, the Northern California fires were in effect and in Redwood Valley California, a fire was burning that destroyed a AT&T telecommunications cable that runs between Ukiah and Willits. This single cable is the primary carrier of nearly all telecommunications services north of Ukiah. Although I cannot comment on the totality of the outage due to this specific fire and line cut, it's widely reported elsewhere that services were affected all the way up to Arcata at least. For Mendocino County, I can report that this telecommunication outage resulted in loss of e-911, cellular, landline, cable and most all other broadband services, and caused a complete outage of our service too. This resulted in a virtual dark age, where reliable and up to date information concerning the emergency situation was unable to be communicated, people could not find their families or know whether they got out in time because nothing was working and no way to talk. We were reduced to using paper and posting notes at the Library or other emergency shelter locations or other places where people gathered in order to communicate. People in these areas also had to use cash to buy supplies. Nothing worked. We attempted to engage emergency restoration processes on our own for our own customers and services. We determined that we could rapidly deploy high capacity microwave connecting a remote location where we also already operate microwave, to our main point of presence at an ATT building in Ukiah, and provide a sufficient level of service to all of our customers that would be a huge benefit during this crisis. As a CLEC with an existing presence and power arrangement and so forth within this facility, it seemed natural that we should be able to go ahead and get a simple OK to go do the work. But, we encountered lethargy and insensitivity to the widespread outage and emergency conditions we are trying to address. We notified ATT of our need and only days later, without followup response, it was suggested that we somehow don't have the legal authority to install microwave (despite the fact that our 544 page contract with ATT that allows our presence within their facilities in the first place DOES SAY 'microwave' is allowed), and further they suggest we'd have to give up all of our existing contractual rights and that if it could happen, it would be only with months of delay and endless expense and bureaucratic nonsense that is only designed to slow down and de-incentivize any such initiative. The equipment we were proposing, is small, low power, lightweight, license free gear that could easily fit complete in a car and be deployed within hours without prior notice or coordination. The electrical and structural requirements are next to nothing, and present absolutely no risk to anyone or the facility. We could have had service restored and further, could have plugged a critical gap by creating an active backup telecommunications path which could be used in future emergencies. I am frustrated with the lack of cooperation to restore services during this latest incident and I think it speaks volumes about the mentality of the incumbent providers. They can't get it together, but here I am standing by with the solution at least for my own customers and why, they just don't want to allow me to do it either. Mendocino County (and northern counties also connected on this same telecommunication cable) have suffered similar telecommunications outages before. Prior to this event, a cable just north of Hopland was cut by two individuals thinking they were stealing copper, resulting in widespread outages lasting upwards of 14 hours. And prior to that, a car crash on the coast hit the one pole where the cable system comes in from the east, wiping out all services along the coast for an extended period of time. And before that, a fire in redwood valley just north of coyote valley casino along the roadway burned out the cable again as in this instance. And before that, another fire north of Willits burned the cable again. And the time before that, a backhoe south of Hopland dug up the cable and cut it, again resulting in far reaching outages affecting hundreds of thousands of people. These repeated outages are not being taken seriously and nothing in substance is being done to fortify the telecommunications network against the next disaster, despite the fact that disaster can and will occur again. We have repeatedly drawn attention to these vulnerabilities but the public perception and awareness of the risks wanes quickly once the crisis is over. I think the incumbent providers have had ample opportunity to address the underlaying causes of their repeated widespread outages, and the empty promises and assurances they have made after past prior disasters should be proof that no actual steps will be taken, that they either do not truly understand or care about the effect that loss of telecommunications services has and the risk it presents to the public. The basic fundamental problem facing the telecommunications network in Mendocino County, is the fact that it is structured without any sort of backup communication links. A primary communications cable extends north along highway 101 at Hopland and then goes into Ukiah. This cable then splits out at Ukiah and goes north up 101 thru Willits, while another segment goes west thru Boonville to Mendocino and then up and down along the coast. A cut at any of several points results in the complete isolation of communities past the cut as what occurred during the recent fire. A basic solution to such a situation then, is to have additional cables to increase the connections between areas, so that if a cable is cut and disrupts communication flowing one way, that the communications can flow over another path from another direction. But the incumbent providers have failed so far to do anything and are likely to continue to ignore this lack of redundancy. I believe that there are some realistic solutions available, and based on my business experience in Mendocino County, I have developed the beginnings of a plan of action. The short of it is, there needs to be a redundant network of geographically diverse, very high capacity telecommunication links that connect the core of the County, in order to overcome the existing problem of there being only 1 connection today. We would need new fiber to connect the major cities from alternate directions, and we further would need new connectivity to other sources of out of region telecommunication services besides AT&T (Such as Level3, who has their own independent cable route into the county). It would further require service providers and emergency responders to all lease connectivity on this redundant system for full effect, the net result however, could be a telecommunications system that stands up and continues functioning and with emergency first responders as well as the general public able to communicate during emergencies. Although I am busy with business and ensuring my customers recover from this event, I certainly am happy to add additional background or technical information if anyone is interested. Going forward, I also could also use help in identifying funding sources and other contacts who could help with some of the heavy lifting that is going to be needed here and so any pointers would certainly be appreciated. Thank you. Mike Ireton President Your Town Online, Inc DBA WillitsOnline Rural Broadband Now, LLC
Out of curiosity, why weren't there any existing emergency/last-resort microwave links already in place? It seems to be known that this cable is a single point of failure ahead of time. Were there previous plans to set up any microwave links that got put on the back burner or canceled? Is there too much focus on fiber being the only option? IMO it's harder for a fire to destroy a microwave path vs. a fiber path, and some capacity is better than no capacity. ~Seth
On 10/17/2017 03:19 PM, Seth Mattinen wrote:
Out of curiosity, why weren't there any existing emergency/last-resort microwave links already in place? It seems to be known that this cable is a single point of failure ahead of time. Were there previous plans to set up any microwave links that got put on the back burner or canceled?
Is there too much focus on fiber being the only option? IMO it's harder for a fire to destroy a microwave path vs. a fiber path, and some capacity is better than no capacity.
~Seth
The ATT building where we have collocation already however, would be quick and easy to add microwave, but they formally resist that idea and are of the position that we don't have the right to do so, despite it being in our interconnection agreement. They are digging in their heels and will have be addressed by the regulatory authority, which is not a quick process. The cost of pulling a new cable to another building that has sufficient line of sight, is prohibitive. Not impossible, but could add six figures to the cost, and therefore is not realistic. Mike-
Hi Mike, Thanks for sharing the story in what must be quite a difficult time. You mention that Level3 also has presence in the county, including their own independent route. Did they also suffer an outage during this latest incident? If not, then why did their connection not allow at least some customers to remain online? Also, as another poster pointed out, would also be interesting to know why the microwave link was not already established prior to this incident if the single point of failure was so well known. (Especially if your contract with AT&T already mentions that you are supposedly allowed to have microwave at their CO.) Cheers, Constantine. On 17/10/2017, Mike <mike-nanog@tiedyenetworks.com> wrote:
Hello fellow operators,
The recent northern California fires wiped us out along with many other service providers and services including cellular, landline, e-911, cable and the like and I feel it's of operational concern the fragility of the telecommunications networks that link us together and the experiences had during this most recent event. I hope this topic is welcome and I apologize in advance for length.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
I am the founder and owner of an Internet Service Provider business in Mendocino County, California and I also operate a CLEC (Competitive Local Exchange Carrier, a type of telecommunications company akin to ATT or Frontier), also in Mendocino County, California. The recent Northern California fires were again a direct and upfront demonstration of the potential for unanticipated widespread public disaster and again demonstrated how vulnerable we all are. Especially, the rising dependence on broadband service, and an expectation by end users that these services which are critical to them will continue functioning during emergencies as they may have no other access. There was a complete and total breakdown in telecommunication services due to fire which wiped out a primary telecommunications cable, and with it, e-911, land lines, cellular, cable and other telecommunication services. I hope that there will be a more complete public investigation and report on this latest incident, but in the mean time I would like to just offer what I know and to suggest where we could go from here.
In the early morning of October 9, the Northern California fires were in effect and in Redwood Valley California, a fire was burning that destroyed a AT&T telecommunications cable that runs between Ukiah and Willits. This single cable is the primary carrier of nearly all telecommunications services north of Ukiah. Although I cannot comment on the totality of the outage due to this specific fire and line cut, it's widely reported elsewhere that services were affected all the way up to Arcata at least. For Mendocino County, I can report that this telecommunication outage resulted in loss of e-911, cellular, landline, cable and most all other broadband services, and caused a complete outage of our service too. This resulted in a virtual dark age, where reliable and up to date information concerning the emergency situation was unable to be communicated, people could not find their families or know whether they got out in time because nothing was working and no way to talk. We were reduced to using paper and posting notes at the Library or other emergency shelter locations or other places where people gathered in order to communicate. People in these areas also had to use cash to buy supplies. Nothing worked.
We attempted to engage emergency restoration processes on our own for our own customers and services. We determined that we could rapidly deploy high capacity microwave connecting a remote location where we also already operate microwave, to our main point of presence at an ATT building in Ukiah, and provide a sufficient level of service to all of our customers that would be a huge benefit during this crisis. As a CLEC with an existing presence and power arrangement and so forth within this facility, it seemed natural that we should be able to go ahead and get a simple OK to go do the work. But, we encountered lethargy and insensitivity to the widespread outage and emergency conditions we are trying to address. We notified ATT of our need and only days later, without followup response, it was suggested that we somehow don't have the legal authority to install microwave (despite the fact that our 544 page contract with ATT that allows our presence within their facilities in the first place DOES SAY 'microwave' is allowed), and further they suggest we'd have to give up all of our existing contractual rights and that if it could happen, it would be only with months of delay and endless expense and bureaucratic nonsense that is only designed to slow down and de-incentivize any such initiative. The equipment we were proposing, is small, low power, lightweight, license free gear that could easily fit complete in a car and be deployed within hours without prior notice or coordination. The electrical and structural requirements are next to nothing, and present absolutely no risk to anyone or the facility. We could have had service restored and further, could have plugged a critical gap by creating an active backup telecommunications path which could be used in future emergencies. I am frustrated with the lack of cooperation to restore services during this latest incident and I think it speaks volumes about the mentality of the incumbent providers. They can't get it together, but here I am standing by with the solution at least for my own customers and why, they just don't want to allow me to do it either.
Mendocino County (and northern counties also connected on this same telecommunication cable) have suffered similar telecommunications outages before. Prior to this event, a cable just north of Hopland was cut by two individuals thinking they were stealing copper, resulting in widespread outages lasting upwards of 14 hours. And prior to that, a car crash on the coast hit the one pole where the cable system comes in from the east, wiping out all services along the coast for an extended period of time. And before that, a fire in redwood valley just north of coyote valley casino along the roadway burned out the cable again as in this instance. And before that, another fire north of Willits burned the cable again. And the time before that, a backhoe south of Hopland dug up the cable and cut it, again resulting in far reaching outages affecting hundreds of thousands of people.
These repeated outages are not being taken seriously and nothing in substance is being done to fortify the telecommunications network against the next disaster, despite the fact that disaster can and will occur again. We have repeatedly drawn attention to these vulnerabilities but the public perception and awareness of the risks wanes quickly once the crisis is over. I think the incumbent providers have had ample opportunity to address the underlaying causes of their repeated widespread outages, and the empty promises and assurances they have made after past prior disasters should be proof that no actual steps will be taken, that they either do not truly understand or care about the effect that loss of telecommunications services has and the risk it presents to the public.
The basic fundamental problem facing the telecommunications network in Mendocino County, is the fact that it is structured without any sort of backup communication links. A primary communications cable extends north along highway 101 at Hopland and then goes into Ukiah. This cable then splits out at Ukiah and goes north up 101 thru Willits, while another segment goes west thru Boonville to Mendocino and then up and down along the coast. A cut at any of several points results in the complete isolation of communities past the cut as what occurred during the recent fire. A basic solution to such a situation then, is to have additional cables to increase the connections between areas, so that if a cable is cut and disrupts communication flowing one way, that the communications can flow over another path from another direction. But the incumbent providers have failed so far to do anything and are likely to continue to ignore this lack of redundancy.
I believe that there are some realistic solutions available, and based on my business experience in Mendocino County, I have developed the beginnings of a plan of action. The short of it is, there needs to be a redundant network of geographically diverse, very high capacity telecommunication links that connect the core of the County, in order to overcome the existing problem of there being only 1 connection today. We would need new fiber to connect the major cities from alternate directions, and we further would need new connectivity to other sources of out of region telecommunication services besides AT&T (Such as Level3, who has their own independent cable route into the county). It would further require service providers and emergency responders to all lease connectivity on this redundant system for full effect, the net result however, could be a telecommunications system that stands up and continues functioning and with emergency first responders as well as the general public able to communicate during emergencies.
Although I am busy with business and ensuring my customers recover from this event, I certainly am happy to add additional background or technical information if anyone is interested. Going forward, I also could also use help in identifying funding sources and other contacts who could help with some of the heavy lifting that is going to be needed here and so any pointers would certainly be appreciated.
Thank you.
Mike Ireton President Your Town Online, Inc DBA WillitsOnline Rural Broadband Now, LLC
On 10/17/2017 10:05 PM, Constantine A. Murenin wrote:
Hi Mike,
Thanks for sharing the story in what must be quite a difficult time.
You mention that Level3 also has presence in the county, including their own independent route. Did they also suffer an outage during this latest incident? If not, then why did their connection not allow at least some customers to remain online?
Also, as another poster pointed out, would also be interesting to know why the microwave link was not already established prior to this incident if the single point of failure was so well known. (Especially if your contract with AT&T already mentions that you are supposedly allowed to have microwave at their CO.)
Cheers, Constantine.
We have attempted to engage exactly that for years - installing a microwave backup - but the only place it can be done, the top of the ATT central office where we are located - is resisted steadfastly by ATT despite it being a right enshrined in our contract, and there are no other realistic options that are also within our financial means to accomplish at this time. Mike-
On 2017-10-17 17:43, Mike wrote:
trying to address. We notified ATT of our need and only days later, without followup response, it was suggested that we somehow don't have the legal authority to install microwave
My reaction would have been to call local mayor, mayor in twon where you try to setup the microwave base, and then if that fails FEMA and tell them that you can restore Internet but AT&T is blocking you. They would have the means to give a call to ATT and get them to stop blocking you. This is an emergency and AT&T would look very very bad if this were to become public.
I'd like to think that most independent operators avoided the ILEC at all costs due to them being difficult to work with, but every now and then you get one that loves to be with the ILEC. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Jean-Francois Mezei" <jfmezei_nanog@vaxination.ca> To: nanog@nanog.org Sent: Wednesday, October 18, 2017 1:51:41 AM Subject: Re: Northern California fires and telecomm outages On 2017-10-17 17:43, Mike wrote:
trying to address. We notified ATT of our need and only days later, without followup response, it was suggested that we somehow don't have the legal authority to install microwave
My reaction would have been to call local mayor, mayor in twon where you try to setup the microwave base, and then if that fails FEMA and tell them that you can restore Internet but AT&T is blocking you. They would have the means to give a call to ATT and get them to stop blocking you. This is an emergency and AT&T would look very very bad if this were to become public.
participants (5)
-
Constantine A. Murenin -
Jean-Francois Mezei -
Mike -
Mike Hammett -
Seth Mattinen