NYT: Attacks Expose Telephone's Soft Underbelly
The New York Times is reporting about some of the issues Verizon is facing with its New York recovery. http://www.nytimes.com/2001/10/15/technology/15PHON.html Although most people are focusing on Verizon, things are much more interesting when you look at the interaction of all the different carriers in lower Manhattan. And what I think is more important, what happened outside of Manhattan. Carriers restored service be re-routing circuits through Cleveland and London (Yes, England). One issue that did come up was government planning was limited to Verizon. So alternate capacity which could have been used during the first few days went unused, and some services were disrupted because employees of non-Verizon carriers weren't allowed into the area.
I'm confused as to this comment: "Assuming they are willing to spend the money, business customers can achieve redundancy, or surplus and backup capacity, by running cables to several different central offices or, in some cases, by using several different communications carriers. Several of Verizon's competitors, in fact, have benefited from the disruptions by signing up new customers in Lower Manhattan. " Are they referring to voice CLECs (or data CLECs for that matter)? I don't see how this situation could have helped them, but only hurt them. I mean, if you have a physical facilities issue (severed copper/fiber optic, damaged CO), then you are gonna have problems with telephone or data service no matter who you use. Plus, some of us know how long opening and resolution of a trouble ticket with an ILEC can take when coming from a CLEC, but now you could only imagine when Verizon has a serious issue, like this. Even if it was a CLEC who ran/leased their own fiber/copper outside of Verizon's network, chances are they are in the same tunnels as Verizon. Are they maybe talking about telephone/data over satellites/microwave or something else wireless? - james -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Sean Donelan Sent: Monday, October 15, 2001 1:52 AM To: nanog@merit.edu Subject: NYT: Attacks Expose Telephone's Soft Underbelly The New York Times is reporting about some of the issues Verizon is facing with its New York recovery. http://www.nytimes.com/2001/10/15/technology/15PHON.html Although most people are focusing on Verizon, things are much more interesting when you look at the interaction of all the different carriers in lower Manhattan. And what I think is more important, what happened outside of Manhattan. Carriers restored service be re-routing circuits through Cleveland and London (Yes, England). One issue that did come up was government planning was limited to Verizon. So alternate capacity which could have been used during the first few days went unused, and some services were disrupted because employees of non-Verizon carriers weren't allowed into the area.
On Mon, Oct 15, 2001 at 02:15:38AM -0400, Kath wrote:
I don't see how this situation could have helped them, but only hurt them. I mean, if you have a physical facilities issue (severed copper/fiber optic, damaged CO), then you are gonna have problems with telephone or data service no matter who you use. Plus, some of us know how long opening and resolution of a trouble ticket with an ILEC can take when coming from a CLEC, but now you could only imagine when Verizon has a serious issue, like this.
My employer is one example of an alternative, and I know of several other companies that could fit the bill as well. MFN (www.mmfn.com) can sell a private fiber optic ring to a customer. In many cases (and Manhattan is a good one) that can be a truly physically diverse ring. We have a number of customers who insist on truly diverse rings, and things like two building entrances at least 100 yards apart. For the most part (because nothing is 100% sure) it is possible to protect against any single cable cut. The largest problem with our service right now is that the barrier to entry for the customer is higher than it should be -- if you need a single T1 we can't compete. For what it's worth, I believe it to be the case (*) that we had fiber rings with a node in one of the trade center towers that were fully operational (minus that node) after the disaster. In the end, it's often a matter of what you're willing to pay for, or if you leave it up to your carrier what risk they are willing to assume. We've all fought to drive the price of T1's into the ground, and at $200-$300 for a local circuit they aren't going to build redundant fiber rings. You're going to get a single run collapsed ring every time, simple economics. Copper services which are still used in so many places are virtually impossible to make path redundant due to the technologies and distance limitations involved. Verizon offers products that connect you to two CO's. I've ordered them before. Companies like MFN can bypass CO's completely with redundant fiber rings. Companies like Yipes and Cogent often build their IP services in redundant metro configurations. Companies like Level 3 and MFS have upscale redundant access products. There are alternatives that could have protected the vast majority of the service that went down. The problem with all of them is that they cost more $$$'s than the non-redundant service. I think this event is making a lot of companies look hard at their cost control strategies and reconsider their disaster recovery plans. Can you be 100% sure you'll always be up? No. Can you add several 9's to the services currently available, absolutely. * I don't work for the fiber side of the business and only know a little bit about the impact of this specific disaster to our network. -- Leo Bicknell - bicknell@ufp.org Systems Engineer - Internetworking Engineer - CCIE 3440 Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
The majority of the CLECs pick up your copper in the 'remote', each Central Office is serviced by a large number of remotes that concentrate the copper and usually implement a protocol known as GR.303 to oversubscribe the remote, normally at about a 4:1 ratio. The resulting lines are then routed back to the CO via high cap circuits, usually on a SONET ring. The remotes may be just fine, but if the CO is damaged/destroyed your calls have nowhere to terminate. CLECs and DLECs pick up the copper at the remotes just like the incumbent telco, but they jump off of the SONET ring in different locations, not at the CO. If you had service via the incumbent AND via one of the xLECs it would generally require widespread devastation in multiple locations (or at the remote itself) to take you completely out of service for any extended time period. FYI: We don't service the NorthEast US and I don't have explicit topology information for NYC, but this description covers every telephone office I have ever seen. Tim McKee -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Kath Sent: Monday, October 15, 2001 02:16 To: Sean Donelan; nanog@merit.edu Subject: RE: Attacks Expose Telephone's Soft Underbelly I'm confused as to this comment: "Assuming they are willing to spend the money, business customers can achieve redundancy, or surplus and backup capacity, by running cables to several different central offices or, in some cases, by using several different communications carriers. Several of Verizon's competitors, in fact, have benefited from the disruptions by signing up new customers in Lower Manhattan. " Are they referring to voice CLECs (or data CLECs for that matter)? I don't see how this situation could have helped them, but only hurt them. I mean, if you have a physical facilities issue (severed copper/fiber optic, damaged CO), then you are gonna have problems with telephone or data service no matter who you use. Plus, some of us know how long opening and resolution of a trouble ticket with an ILEC can take when coming from a CLEC, but now you could only imagine when Verizon has a serious issue, like this. Even if it was a CLEC who ran/leased their own fiber/copper outside of Verizon's network, chances are they are in the same tunnels as Verizon. Are they maybe talking about telephone/data over satellites/microwave or something else wireless? - james -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Sean Donelan Sent: Monday, October 15, 2001 1:52 AM To: nanog@merit.edu Subject: NYT: Attacks Expose Telephone's Soft Underbelly The New York Times is reporting about some of the issues Verizon is facing with its New York recovery. http://www.nytimes.com/2001/10/15/technology/15PHON.html Although most people are focusing on Verizon, things are much more interesting when you look at the interaction of all the different carriers in lower Manhattan. And what I think is more important, what happened outside of Manhattan. Carriers restored service be re-routing circuits through Cleveland and London (Yes, England). One issue that did come up was government planning was limited to Verizon. So alternate capacity which could have been used during the first few days went unused, and some services were disrupted because employees of non-Verizon carriers weren't allowed into the area.
participants (4)
-
Kath -
Leo Bicknell -
Sean Donelan -
Timothy R. McKee