I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul outages are associated with where this ATM terminates and that's the cause, rather than the service in/out of Santa Cruz County, but I have no way of telling which from here) My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise there), and I'm not seeing significant routing problems in/out of there with transit or peering. (Not even any down peers, so no inter-Equinix-site outage apparently). Matthew Kaufman matthew@eeph.com
Service to South Santa Clara county is completely down: Internet, landline, and cellphones. Both Verizon and AT&T are affected. 911 is also down. My cellphones show one or no bars. Normally they are all four bars. The idea that all of that is lumped in one fiber bundle is mind boggling. On Thu, Apr 9, 2009 at 11:05 AM, Matthew Kaufman <matthew@eeph.com> wrote:
I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul outages are associated with where this ATM terminates and that's the cause, rather than the service in/out of Santa Cruz County, but I have no way of telling which from here)
My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise there), and I'm not seeing significant routing problems in/out of there with transit or peering. (Not even any down peers, so no inter-Equinix-site outage apparently).
Matthew Kaufman matthew@eeph.com
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling. That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue. That there is no over-under wide-area back-up coverage for the cellular canopy ... We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality: The emperor has no clothes. Roy wrote:
Service to South Santa Clara county is completely down: Internet, landline, and cellphones. Both Verizon and AT&T are affected. 911 is also down.
My cellphones show one or no bars. Normally they are all four bars.
The idea that all of that is lumped in one fiber bundle is mind boggling.
On Thu, Apr 9, 2009 at 11:05 AM, Matthew Kaufman <matthew@eeph.com> wrote:
I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul outages are associated with where this ATM terminates and that's the cause, rather than the service in/out of Santa Cruz County, but I have no way of telling which from here)
My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise there), and I'm not seeing significant routing problems in/out of there with transit or peering. (Not even any down peers, so no inter-Equinix-site outage apparently).
Matthew Kaufman matthew@eeph.com
Robert M. Enger wrote:
We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality: The emperor has no clothes.
You wouldn't have clothes either if you could double your profit by not wearing any. Matthew Kaufman
A sobering touché. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Robert M. Enger [mailto:enger@enger.us] Sent: Thursday, April 09, 2009 1:59 PM To: Roy Cc: nanog@nanog.org Subject: Re: Fiber cut in SF area That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling. That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue. That there is no over-under wide-area back-up coverage for the cellular canopy ... We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality: The emperor has no clothes. Roy wrote:
Service to South Santa Clara county is completely down: Internet, landline, and cellphones. Both Verizon and AT&T are affected. 911 is also down.
My cellphones show one or no bars. Normally they are all four bars.
The idea that all of that is lumped in one fiber bundle is mind boggling.
On Thu, Apr 9, 2009 at 11:05 AM, Matthew Kaufman <matthew@eeph.com> wrote:
I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul outages are associated with where this ATM terminates and that's the cause, rather than the service in/out of Santa Cruz County, but I have no way of telling which from here)
My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise there), and I'm not seeing significant routing problems in/out of there with transit or peering. (Not even any down peers, so no inter-Equinix-site outage apparently).
Matthew Kaufman matthew@eeph.com
______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
On Apr 9, 2009, at 3:58 PM, Robert M. Enger wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS so get priority on wired/PSTN. They can also get radio priority w/ WPS.
That there is no over-under wide-area back-up coverage for the cellular canopy ...
The problem is how do you back up such a large area. WPS can get you priority.
We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality: The emperor has no clothes.
I think the problem is there are clothes, some people/areas have none, some have an abundance. If people don't plan for going out in public, there is a chance you'll walk outside naked ;)
Roy wrote:
Service to South Santa Clara county is completely down: Internet, landline, and cellphones. Both Verizon and AT&T are affected. 911 is also down.
My cellphones show one or no bars. Normally they are all four bars.
If you're an ISP, you may be able to obtain GETS or WPS for your engineers. This would allow you a better chance of getting a channel to respond to issues. This is a good test to see how your backup plans might work for communication in the case of a larger distaster (earthquake, or other). Make sure you test the tools you have. The people I know with GETS cards are encouraged to test them regularly and verify they work. If someone has one, I'd be interested to know if it proved to be of value today. - Jared
Jared Mauch wrote:
On Apr 9, 2009, at 3:58 PM, Robert M. Enger wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS so get priority on wired/PSTN. They can also get radio priority w/ WPS.
I didn't know about WPS. http://policechiefmagazine.org/magazine/index.cfm?fuseaction=display_arch&article_id=839&issue_id=32006 Interesting stuff.
No RF, no WPS. If all the base stations are knocked out in a region, and there is no "over" coverage from towers out of the affected region then there are no channels to which priority access can be allotted. A potential remedy (at least for conventional cell phones) would be to scatter back-up cell sites on high-vantage-point locations. Each would need to be equipped with multiple narrow sectors using high gain antennas. These lofty sites would form a secondary canopy over the region (hence the "over/under" naming). Assuming the secondary sites are hardened, provided with back-up power and trunked with physical diversity (perhaps one links using 70/80Ghz), they should provide some additional protection in situations like this. This would provide some service when primary towers in an entire sub-area are all knocked out. Who knows, in day to day routine usage they might even fill-in a few coverage holes that have been lingering in some systems. From the reports of "zero bars" on cell phones, we can presume no "over" canopy is in operation in that region. There are other radio systems, but their scope is limited. Cellular provides wider availability. Granny can use her Jitterbug to call for help. Similarly, many business burglar/fire alarm systems use cellular to transmit alarms to the central station. With terrestrial and radio alarm reporting knocked out, many businesses will be sitting ducks. But why waste the money on system improvements. Best to conserve the funds to pay bonuses to the corporate executives. No matter how egregious the error or omission, they always walk away with big checks, and the rest of us waddle away looking for Preparation-H. Charles Wyble wrote:
Jared Mauch wrote:
On Apr 9, 2009, at 3:58 PM, Robert M. Enger wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS so get priority on wired/PSTN. They can also get radio priority w/ WPS.
I didn't know about WPS.
Interesting stuff.
On Thu, Apr 9, 2009 at 5:29 PM, Jared Mauch <jared@puck.nether.net> wrote:
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS ...
Which is fine if you're a law-enforcement folk. Kinda sucks if you're an ordinary private citizen who tries to dial 911 and gets a reorder tone. Which I presume is what is happening, since everybody is saying 911 is down. What's the point of having all the emergency service personnel communicating with each other if they can't get 911 calls in the first place? (Rhetorical question, I know there are other ways they can find out about emergencies, but 911 is the big one.) Maybe nostalgia just ain't what it used to be, but I thought the PSTN used to be more reliable than this. #ifdef CONSPIRACY_THEORIST What if this isn't simple vandalism? #endif -- Ben
On Thu, Apr 9, 2009 at 5:52 PM, Ben Scott <mailvortex@gmail.com> wrote:
#ifdef CONSPIRACY_THEORIST
What if this isn't simple vandalism?
#endif
If my read is correct, this is multiple cuts in multiple locations. To answer the what-if ("What if this isn't simple vandalism?") : It's not. -jamie
On Thu, 9 Apr 2009, Jared Mauch wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
Only helps with N-1 breaks. Unfortunately, sometimes there are N+1 breaks. Check the NANOG archives, I believe there were 5 breaks in one day in the 1990's; and even in the last year there have been 2-4 breaks on some transoceanic cables at the same time. On the other hand, I've never heard a carrier complain about digging more fiber as long as someone is willing to pay for it. How much more is someone willing to pay to get more diversity? Not willing to pay for it? I guess that's an answer too.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
This costs $$$ and usually isn't a problem as there are other ways to communicate. The law-enforcement folks qualify for GETS so get priority on wired/PSTN. They can also get radio priority w/ WPS.
If you don't know the acronyms, see www.ncs.gov. GETS and WPS are good as long as the system is still connected. TSP and SHARES helps when the system becomes disconnected. Some carriers also have mutual aid pacts, and work with members of the mutual aid pact with spare facilities. Its better to sign up ahead time, rather than waiting until after the problem happens. Even though those tools are useful, also work on how to maintain your own self-sufficiency until help arrives. There will always be some prioritization of repair efforts. Although it had a big impact on some of the largest carriers in the region, especially for local services; its always interesting to see other stuff kept working. Not everything broke.
If you're an ISP, you may be able to obtain GETS or WPS for your engineers. This would allow you a better chance of getting a channel to respond to issues. This is a good test to see how your backup plans might work for communication in the case of a larger distaster (earthquake, or other).
Make sure you test the tools you have. The people I know with GETS cards are encouraged to test them regularly and verify they work. If someone has one, I'd be interested to know if it proved to be of value today.
It sucked, but its also an opportunity for ISPs to figure out better ways to do things. personal opinions only
Yep.... it leads to: Activity Type Code Desc: PROGRESS COMMENTS Activity Type Code: PROG OTDR readings were taken by AT&T West and a cut was located 1600 ft from the San Jose, CA central office. AT&T West technicians are onsite working to isolate the exact location of the cut. There are 4 cables impacted. AT&T Mobility has 61 GSM and 45 co-located UMTS sites out of service off of Santa Clara Base Station Controllers 15 & 23, and Santa Clara Radio Network Controller 4. E911 has 52 Location Measuring Units down. The AT&T West Santa Cruz 11 central office (41,803 ATNs) is experiencing an SS7 isolation and the San Martin central office (11,904 ATNs) lost it's umbilical and is isolated at this time. The Bailey remote site (4,973 ATNs) is also isolated. Scott's Valley has 3 out of 4 SS7 links down. The Santa Cruz 01, Aptos, Scott's Valley, Felton, Boulder Creek, Ben Lomand, San Jose 11, San Jose 13, San Jose 21 central offices have trunks impacted such that all lines are busy and incoming calls are receiving trouble messages. The Santa Cruz County SO (178,040 ATNs), Scott's Valley PD (12,007 ATNs) and the UC Santa Cruz PD (14,909 ATNs) are all without ALI at this time. The Gilroy PD PSAP and the Morgan Hill PD and CDF have been rerouted with ALI/ANI. The Felton CDF has not been rerouted. There are 17 DSLAMS and 4 ATMS out of service impacting DSL service. There are 3 SMDI Links down impacting voicemail service. Verizon's Morgan Hill and Gilroy central offices are currently isolated. There have been 224,865 blocked calls. Robert M. Enger wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
That there is no over-under wide-area back-up coverage for the cellular canopy ...
We posture and orate about being prepared for terrorist attacks and natural disasters, and then events like these reveal the reality: The emperor has no clothes.
Roy wrote:
Service to South Santa Clara county is completely down: Internet, landline, and cellphones. Both Verizon and AT&T are affected. 911 is also down.
My cellphones show one or no bars. Normally they are all four bars.
The idea that all of that is lumped in one fiber bundle is mind boggling.
On Thu, Apr 9, 2009 at 11:05 AM, Matthew Kaufman <matthew@eeph.com> wrote:
I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul outages are associated with where this ATM terminates and that's the cause, rather than the service in/out of Santa Cruz County, but I have no way of telling which from here)
My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise there), and I'm not seeing significant routing problems in/out of there with transit or peering. (Not even any down peers, so no inter-Equinix-site outage apparently).
Matthew Kaufman matthew@eeph.com
Robert M. Enger wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral is mind boggling.
That our crack (or on crack) govt contracting/emergency-preparedness staff didn't demand protected facilities for 911 is another mind boggling issue.
911 centers can work just fine without phones. They use radio for intra-agency communications with police, fire, etc. There is also a large ham radio community who jumps in to help with communications when needed (testing, simulations, and real disasters). The primary problem a 911 center has if the phones go out is that people can't *reach* the 911 center if *their* phone lines don't work. jc
participants (11)
-
Ben Scott -
Charles Wyble -
jamie rishaw -
Jared Mauch -
JC Dill -
Jorge Amodio -
Matthew Kaufman -
Murphy, Jay, DOH -
Robert M. Enger -
Roy -
Sean Donelan