FCC Outage Reports ..(.was Verizon outage in Southern California?)
I wasn't thinking in terms of automatic monitoring, that would open up a can of worms security wise. Just looking at some way of getting the manual reporting (that is still taking place to the FCC) back in the (semi?)public domain. Due to terrorism concerns, that information is no longer available online. I'm pretty sure the LEC's and IXC's like it that way also, as they no longer have to air their dirty laundry. I was able to get some information under the Freedom of Information act for an outage that affected me directly , but that takes days or weeks. As close to real-time information as possible is what's needed to assess and respond to a major outage, i.e. routing voice/data via different carriers, being able to explain to end users why their email or phone calls didn't go through , etc. and eliminating the need to open tons of trouble tickets during a major event. One master ticket - such as fiber cut affect xxx OC48's would suffice. Not sure how this can be balanced against DHS perceived needs though...any suggestions? -----Original Message----- From: Vicky Rode [mailto:vickyr@socal.rr.com] Sent: Wednesday, October 19, 2005 5:45 PM To: Wallace Keith Subject: Re: Verizon outage in Southern California? -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I wonder how would Telcos, ISPs and GOV agencies feel about a third party polling their devices, not to mention security. I think netcarft comes close to what you're suggesting. regards, /virendra Wallace Keith wrote:
All this speculation!! Remember the good old days when you could see faxes of FCC outage reports online? Was sure nice to know what was going on, before the FCC took these offline (due to DHS?) It would really by nice to have some sort of an online clearing house, and gain some visibility again into overall network status. This will become even more important as things continue to converge. DACS and DC Power failures tend to affect multiple services and in the case of power, multiple carriers that are colo'd in the CO. -Keith
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Vicky Rode Sent: Wednesday, October 19, 2005 1:29 PM To: wb8foz@nrk.com Cc: nanog list Subject: Re: Verizon outage in Southern California?
I wonder what ever happened to redundancy? I guess 5 9s (dunno what the going number is) got blown out of the water for them.
regards, /virendra
David Lesher wrote:
Speaking on Deep Background, the Press Secretary whispered:
I'm not completely familiar with the telco jargon. Does Tandem mean the same as a local central office, where POTS lines terminate at the switch? Long Beach has a population of 470,000. The C/Os I know of are:
A "Central Office" switch talks to subscribers aka end-users. On its backside, it talks to other CO's and tandems. Time was, that was also VF copper pairs, but it's long since all DS1 and up.....
A tandem is a switch that talks not to subs, but only to CO's. In days
of old, when a {dialup} call went to the other side of town, chances are it went you-yourCO-downtown tandem-joesCO-joe. {copper all the way...}.
A tandem was always housed in large CO building, but might have been ATT's vice the operationg company, etc...
But ESS's and ""classless switching"" and massive expansion of the plant really muddled the picture. An ESS could be both a CO switch [for multiple prefixes and even multiple NPA's..] AND act like a tandem.. And oh, the actual "line cards" can be remoted 100 miles away
in a horz. phonebooth box alongside the road in Smallville.... with DS1's/OC coming back.
My guess is a DACS, a cross-connect point that is an software-driven patch panel, lost its marbles. [engineering term of art.....] A DACS could have dozen->MANY dozen DS1/DS3/OC-n going hither and yon. Some will be leased circuits. Others will be the CO trunks going from one switch to another. It may/may not have muxes internal, so that what arrives on a DS1 leaves in a OC96..
I note it went down at 2:20 AM. That SCREAMS software upgrade/cutover.
What's to bet GEE, no...VZEEE, was doing just that and there was a major ohshit.
Sean noted a long while back that somehow, DACS crashes always seem to
take hours to recover. Maybe the backups are on Kansas City standard tapes, I donno.. but this sounds like that..
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDVr5HpbZvCIJx1bcRApndAKCyFEbNgzqSeqxfCj8I6A/Eq4x/QQCgoFRj CuS3SSCRkdL7vtHE+Bsto3E= =MQzx -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thinking out loud. I guess some sort of trust model would help similar to what nsp-sec has in place (not sure its current state). It could be nice if there was some sort of a consensus among this consortium to distribute executive health metrics with the help of some secure trusted monitoring mechanism or maybe push model to a central database of some sort. Like to hear more thoughts as well. regards, /virendra Wallace Keith wrote:
I wasn't thinking in terms of automatic monitoring, that would open up a can of worms security wise. Just looking at some way of getting the manual reporting (that is still taking place to the FCC) back in the (semi?)public domain. Due to terrorism concerns, that information is no longer available online. I'm pretty sure the LEC's and IXC's like it that way also, as they no longer have to air their dirty laundry. I was able to get some information under the Freedom of Information act for an outage that affected me directly , but that takes days or weeks. As close to real-time information as possible is what's needed to assess and respond to a major outage, i.e. routing voice/data via different carriers, being able to explain to end users why their email or phone calls didn't go through , etc. and eliminating the need to open tons of trouble tickets during a major event. One master ticket - such as fiber cut affect xxx OC48's would suffice. Not sure how this can be balanced against DHS perceived needs though...any suggestions?
-----Original Message----- From: Vicky Rode [mailto:vickyr@socal.rr.com] Sent: Wednesday, October 19, 2005 5:45 PM To: Wallace Keith Subject: Re: Verizon outage in Southern California?
I wonder how would Telcos, ISPs and GOV agencies feel about a third party polling their devices, not to mention security.
I think netcarft comes close to what you're suggesting.
regards, /virendra
Wallace Keith wrote:
All this speculation!! Remember the good old days when you could see faxes of FCC outage reports online? Was sure nice to know what was going on, before the FCC took these offline (due to DHS?) It would really by nice to have some sort of an online clearing house, and gain some visibility again into overall network status. This will become even more important as things continue to converge. DACS and DC Power failures tend to affect multiple services and in the case of power, multiple carriers that are colo'd in the CO. -Keith
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Vicky Rode Sent: Wednesday, October 19, 2005 1:29 PM To: wb8foz@nrk.com Cc: nanog list Subject: Re: Verizon outage in Southern California?
I wonder what ever happened to redundancy? I guess 5 9s (dunno what the going number is) got blown out of the water for them.
regards, /virendra
David Lesher wrote:
Speaking on Deep Background, the Press Secretary whispered:
I'm not completely familiar with the telco jargon. Does Tandem mean the same as a local central office, where POTS lines terminate at the switch? Long Beach has a population of 470,000. The C/Os I know of are:
A "Central Office" switch talks to subscribers aka end-users. On its backside, it talks to other CO's and tandems. Time was, that was also VF copper pairs, but it's long since all DS1 and up.....
A tandem is a switch that talks not to subs, but only to CO's. In days
of old, when a {dialup} call went to the other side of town, chances are it went you-yourCO-downtown tandem-joesCO-joe. {copper all the way...}.
A tandem was always housed in large CO building, but might have been ATT's vice the operationg company, etc...
But ESS's and ""classless switching"" and massive expansion of the plant really muddled the picture. An ESS could be both a CO switch [for multiple prefixes and even multiple NPA's..] AND act like a tandem.. And oh, the actual "line cards" can be remoted 100 miles away
in a horz. phonebooth box alongside the road in Smallville.... with DS1's/OC coming back.
My guess is a DACS, a cross-connect point that is an software-driven patch panel, lost its marbles. [engineering term of art.....] A DACS could have dozen->MANY dozen DS1/DS3/OC-n going hither and yon. Some will be leased circuits. Others will be the CO trunks going from one switch to another. It may/may not have muxes internal, so that what arrives on a DS1 leaves in a OC96..
I note it went down at 2:20 AM. That SCREAMS software
upgrade/cutover.
What's to bet GEE, no...VZEEE, was doing just that and there was a major ohshit.
Sean noted a long while back that somehow, DACS crashes always seem to
take hours to recover. Maybe the backups are on Kansas City standard tapes, I donno.. but this sounds like that..
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFDWHPxpbZvCIJx1bcRAv/DAJwPg4J5CNSZc6z0kTv3UwFKHBKq7QCePH2M As5MkIaZrrH0N+XnxT7oVAI= =G/S3 -----END PGP SIGNATURE-----
Vicky Rode wrote:
Thinking out loud.
I guess some sort of trust model would help similar to what nsp-sec has in place (not sure its current state).
It could be nice if there was some sort of a consensus among this consortium to distribute executive health metrics with the help of some secure trusted monitoring mechanism or maybe push model to a central database of some sort.
Like to hear more thoughts as well.
Here we see again that the secrecy ("to prevent terrorism") of this information costs more than having it in the open as the FCC did in the past. The whole terrorism sham was just a convenient excuse to prevent outsiders from assessing the quality of the carriers network. Even if, which it does not, secrecy of this information would prevent any kind of external force terrorism we now have to suffer the terrorism from dishonest carriers and intransparent phone and bandwidth markets. One can only guess the cost shouldered by carriers customers because of unknown or deliberately wrong information. Guess how many procurements would have been made differently if true reliability and physical route information were available. Do I feel better that neither me nor the terrorist know that my "redundant" fiber routes are in the same dig? Or in the same cable even? We all know how reliable the carriers bonus driven sales droid promises are... -- Andre BTW: Often overlooked fact: Living is deadly.
On Fri, 21 Oct 2005, Andre Oppermann wrote:
Here we see again that the secrecy ("to prevent terrorism") of this information costs more than having it in the open as the FCC did in the past. The whole terrorism sham was just a convenient excuse to prevent outsiders from assessing the quality of the carriers network.
In the field of security engineering, this is something called security through obscurity. Terrorists are well funded, and they, no doubt, can get hold on those 'secret' fiber maps if they have interest in them.
Do I feel better that neither me nor the terrorist know that my "redundant" fiber routes are in the same dig? Or in the same cable even? We all know how reliable the carriers bonus driven sales droid promises are...
Only ones suffering are us... -- juuso lehtinen
participants (4)
-
Andre Oppermann -
Juuso Lehtinen -
Vicky Rode -
Wallace Keith