RE: Verizon outage in Southern California?
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Matthew Black Sent: Tuesday, October 18, 2005 3:13 PM
Telephone service is beginning to be restored in the Long Beach area but is still sporadic.
Our ATM WAN link through Sprint came back up around 1345 Central time, and the two DS1s for the school's Internet service were revived about fifteen minutes ago (1507 CDT). They've been rock-solid so something must be going right out there. When I called Sprint about any information they might have for the outage the tech said that the area was down due to a Verizon DACS failure. That must have been a spectacular failure, because I'm reading that it wiped out most everything ( http://www2.presstelegram.com/news/ci_3128087 indicates four tandems hit?! ) in the area. The articles are primarily focusing on the impact to E911 services, followed with the hit to POTS lines. I have yet to see any mention of impact to data in any of 'em. Here's what intrigues me about this outage: if it wiped out E911, most of the POTS and also impacted data services (as Jay Hannigan and I can attest), how did the cell towers that are also served by the network live through it? Jason "Feren" Olsen Senior Network Engineer DeVry, Inc Em: jolsen@devry.com Ph: 630-645-1607 One Tower Lane INOC-DBA: 19258*526 Fx: 630-389-2929 Oakbrook Terrace, IL 60181
On Tue, 18 Oct 2005 15:38:06 -0500 "Olsen, Jason" <jolsen@devry.com> wrote:
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Matthew Black Sent: Tuesday, October 18, 2005 3:13 PM
Telephone service is beginning to be restored in the Long Beach area but is still sporadic.
Our ATM WAN link through Sprint came back up around 1345 Central time, and the two DS1s for the school's Internet service were revived about fifteen minutes ago (1507 CDT). They've been rock-solid so something must be going right out there.
When I called Sprint about any information they might have for the outage the tech said that the area was down due to a Verizon DACS failure. That must have been a spectacular failure, because I'm reading that it wiped out most everything ( http://www2.presstelegram.com/news/ci_3128087 indicates four tandems hit?! ) in the area. The articles are primarily focusing on the impact to E911 services, followed with the hit to POTS lines. I have yet to see any mention of impact to data in any of 'em. Here's what intrigues me about this outage: if it wiped out E911, most of the POTS and also impacted data services (as Jay Hannigan and I can attest), how did the cell towers that are also served by the network live through it?
Jason "Feren" Olsen Senior Network Engineer DeVry, Inc
Thanks for the link to the story update. Our OC3C (155 Mbps link) goes through a place called WestEd (not WestCom) in Seal Beach which is the headquarters for CENIC, the CA higher education network. We never saw any data outage for CSULB. I'm not completely familiar with the telco jargon. Does Tandem mean the same as a local central office, where POTS lines terminate at the switch? Long Beach has a population of 470,000. The C/Os I know of are: Alamitos at 7th Street and Termino, ZIP 90814 Clark near Clark Ave and Pacific Coast Highway, ZIP 90804 LongBeach at 6th Street and Elm Ave, ZIP 90802 Lakewood at Clark Ave and Connant St, ZIP 90808 LNBHCAXG at 3440 California Ave, ZIP 90807 (for my home) I have no idea whether cell service was truly affected. The announcements we sent to our campus suggested people use their cell phones for 911 service which would be serviced by the CA Highway Patrol (Erik Estrada, etc.) or a campus telephone which is serviced by our local campus police (sworn state police). I was completely unaware of the outage until someone else mentioned it in my office. matthew black california state university, long beach
Speaking on Deep Background, the Press Secretary whispered:
I'm not completely familiar with the telco jargon. Does Tandem mean the same as a local central office, where POTS lines terminate at the switch? Long Beach has a population of 470,000. The C/Os I know of are:
A "Central Office" switch talks to subscribers aka end-users. On its backside, it talks to other CO's and tandems. Time was, that was also VF copper pairs, but it's long since all DS1 and up..... A tandem is a switch that talks not to subs, but only to CO's. In days of old, when a {dialup} call went to the other side of town, chances are it went you-yourCO-downtown tandem-joesCO-joe. {copper all the way...}. A tandem was always housed in large CO building, but might have been ATT's vice the operationg company, etc... But ESS's and ""classless switching"" and massive expansion of the plant really muddled the picture. An ESS could be both a CO switch [for multiple prefixes and even multiple NPA's..] AND act like a tandem.. And oh, the actual "line cards" can be remoted 100 miles away in a horz. phonebooth box alongside the road in Smallville.... with DS1's/OC coming back. My guess is a DACS, a cross-connect point that is an software-driven patch panel, lost its marbles. [engineering term of art.....] A DACS could have dozen->MANY dozen DS1/DS3/OC-n going hither and yon. Some will be leased circuits. Others will be the CO trunks going from one switch to another. It may/may not have muxes internal, so that what arrives on a DS1 leaves in a OC96.. I note it went down at 2:20 AM. That SCREAMS software upgrade/cutover. What's to bet GEE, no...VZEEE, was doing just that and there was a major ohshit. Sean noted a long while back that somehow, DACS crashes always seem to take hours to recover. Maybe the backups are on Kansas City standard tapes, I donno.. but this sounds like that.. -- A host is a host from coast to coast.................wb8foz@nrk.com & no one will talk to a host that's close........[v].(301) 56-LINUX Unless the host (that isn't close).........................pob 1433 is busy, hung or dead....................................20915-1433
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I wonder what ever happened to redundancy? I guess 5 9s (dunno what the going number is) got blown out of the water for them. regards, /virendra David Lesher wrote:
Speaking on Deep Background, the Press Secretary whispered:
I'm not completely familiar with the telco jargon. Does Tandem mean the same as a local central office, where POTS lines terminate at the switch? Long Beach has a population of 470,000. The C/Os I know of are:
A "Central Office" switch talks to subscribers aka end-users. On its backside, it talks to other CO's and tandems. Time was, that was also VF copper pairs, but it's long since all DS1 and up.....
A tandem is a switch that talks not to subs, but only to CO's. In days of old, when a {dialup} call went to the other side of town, chances are it went you-yourCO-downtown tandem-joesCO-joe. {copper all the way...}.
A tandem was always housed in large CO building, but might have been ATT's vice the operationg company, etc...
But ESS's and ""classless switching"" and massive expansion of the plant really muddled the picture. An ESS could be both a CO switch [for multiple prefixes and even multiple NPA's..] AND act like a tandem.. And oh, the actual "line cards" can be remoted 100 miles away in a horz. phonebooth box alongside the road in Smallville.... with DS1's/OC coming back.
My guess is a DACS, a cross-connect point that is an software-driven patch panel, lost its marbles. [engineering term of art.....] A DACS could have dozen->MANY dozen DS1/DS3/OC-n going hither and yon. Some will be leased circuits. Others will be the CO trunks going from one switch to another. It may/may not have muxes internal, so that what arrives on a DS1 leaves in a OC96..
I note it went down at 2:20 AM. That SCREAMS software upgrade/cutover. What's to bet GEE, no...VZEEE, was doing just that and there was a major ohshit.
Sean noted a long while back that somehow, DACS crashes always seem to take hours to recover. Maybe the backups are on Kansas City standard tapes, I donno.. but this sounds like that..
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDVoJXpbZvCIJx1bcRAstJAJ0dnrQL1P2QJyxNU3r0T/X8g9fukQCgnm/N yW5EvW7gI3gfjY7XSozyMds= =ocNd -----END PGP SIGNATURE-----
5 9s can be measured all sorts of ways... Network wide, it probably isn't even a blip. Even in terms of all of California service its probably not much more than a blip. Vicky Rode wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I wonder what ever happened to redundancy? I guess 5 9s (dunno what the going number is) got blown out of the water for them.
regards, /virendra
David Lesher wrote:
Speaking on Deep Background, the Press Secretary whispered:
I'm not completely familiar with the telco jargon. Does Tandem mean the same as a local central office, where POTS lines terminate at the switch? Long Beach has a population of 470,000. The C/Os I know of are:
A "Central Office" switch talks to subscribers aka end-users. On its backside, it talks to other CO's and tandems. Time was, that was also VF copper pairs, but it's long since all DS1 and up.....
A tandem is a switch that talks not to subs, but only to CO's. In days of old, when a {dialup} call went to the other side of town, chances are it went you-yourCO-downtown tandem-joesCO-joe. {copper all the way...}.
A tandem was always housed in large CO building, but might have been ATT's vice the operationg company, etc...
But ESS's and ""classless switching"" and massive expansion of the plant really muddled the picture. An ESS could be both a CO switch [for multiple prefixes and even multiple NPA's..] AND act like a tandem.. And oh, the actual "line cards" can be remoted 100 miles away in a horz. phonebooth box alongside the road in Smallville.... with DS1's/OC coming back.
My guess is a DACS, a cross-connect point that is an software-driven patch panel, lost its marbles. [engineering term of art.....] A DACS could have dozen->MANY dozen DS1/DS3/OC-n going hither and yon. Some will be leased circuits. Others will be the CO trunks going from one switch to another. It may/may not have muxes internal, so that what arrives on a DS1 leaves in a OC96..
I note it went down at 2:20 AM. That SCREAMS software upgrade/cutover. What's to bet GEE, no...VZEEE, was doing just that and there was a major ohshit.
Sean noted a long while back that somehow, DACS crashes always seem to take hours to recover. Maybe the backups are on Kansas City standard tapes, I donno.. but this sounds like that..
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFDVoJXpbZvCIJx1bcRAstJAJ0dnrQL1P2QJyxNU3r0T/X8g9fukQCgnm/N yW5EvW7gI3gfjY7XSozyMds= =ocNd -----END PGP SIGNATURE-----
participants (5)
-
David Lesher -
Deepak Jain -
Matthew Black -
Olsen, Jason -
Vicky Rode