Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted
What's really interesing is the fragility of the existing telecom infrastructure. These six cables were apparently very close to each other in the water. In other words, despite all the preaching about physical diversity, it was ignored in practice. Indeed, undersea cables very often use the same conduits for terrestrial backhaul since it is the most cost effective solution. However, that means that diversifying across undersea cables does not buy the sort of physical diversity that is anticipated. Roderick S. Beck EMEA and North American Sales Hibernia Atlantic rod.beck@hiberniaatlantic.com http://www.hiberniaatlantic.com This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Atlantic has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment
That's news? The same still happens with much land-based sonet, where "diverse paths" still share the same entrance to a given facility. Unless each end can negotiate cost sharing for diverse paths, or unless the owner of the fiber can cost justify the same, chances are you're not going to see the ideal. Money will always speak louder than idealism. Undersea paths complicate this even further. On Sun, 21 Jan 2007, Rod Beck wrote: :What's really interesing is the fragility of the existing telecom infrastructure. These six cables were apparently very close to each other in the water. In other words, despite all the preaching about physical diversity, it was ignored in practice. Indeed, undersea cables very often use the same conduits for terrestrial backhaul since it is the most cost effective solution. However, that means that diversifying across undersea cables does not buy the sort of physical diversity that is anticipated. : :Roderick S. Beck :EMEA and North American Sales :Hibernia Atlantic
Hi Brian, Unfortunately it is news to the decision makers, the buyers of network capacity at many of the major IP backbones. Indeed, the Atlantic route has problems quite similar to the Pacific. :Roderick S. Beck :EMEA and North American Sales :Hibernia Atlantic This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Atlantic has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment
On Sun, 21 Jan 2007, Rod Beck wrote:
Unfortunately it is news to the decision makers, the buyers of network capacity at many of the major IP backbones. Indeed, the Atlantic route has problems quite similar to the Pacific.
If this is news to them, perhaps its time to get new decision makers and buyers of network capacity at major IP backbones :-) Unfotunately people have to learn the same lessons over and over again. http://www.atis.org/ndai/ "End-to-end multi-carrier circuit diversity assurance currently cannot be conducted in a scalable manner. The cost and level of manual effort required demonstrated that an ongoing program for end-to-end multi-carrier circuit diversity assurance cannot currently be widely offered." http://www.atis.org/PRESS/pressreleases2006/031506.htm "The NDAI report confirmed our suspicions that diversity assurance is not for the meek," Malphrus added. "It is expensive and requires commitment by the customer to work closely with carriers in performing due diligence. Until the problem is solved, circuit route diversity should not be promoted as a general customer best practice."
Well, gentlemen, you have to ask for the fiber maps and have them placed in the contract as an exhibit. Most of the large commercial banks are doing it. It's doable, but it does require effort. And again, sorry for the dislaimer. It should be gone tomorrow. Regards, - Roderick. -----Original Message----- From: owner-nanog@merit.edu on behalf of Sean Donelan Sent: Sun 1/21/2007 8:13 PM To: nanog@merit.edu Subject: RE: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted On Sun, 21 Jan 2007, Rod Beck wrote:
Unfortunately it is news to the decision makers, the buyers of network capacity at many of the major IP backbones. Indeed, the Atlantic route has problems quite similar to the Pacific.
If this is news to them, perhaps its time to get new decision makers and buyers of network capacity at major IP backbones :-) Unfotunately people have to learn the same lessons over and over again. http://www.atis.org/ndai/ "End-to-end multi-carrier circuit diversity assurance currently cannot be conducted in a scalable manner. The cost and level of manual effort required demonstrated that an ongoing program for end-to-end multi-carrier circuit diversity assurance cannot currently be widely offered." http://www.atis.org/PRESS/pressreleases2006/031506.htm "The NDAI report confirmed our suspicions that diversity assurance is not for the meek," Malphrus added. "It is expensive and requires commitment by the customer to work closely with carriers in performing due diligence. Until the problem is solved, circuit route diversity should not be promoted as a general customer best practice." This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Atlantic has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment
On Sun, 21 Jan 2007, Rod Beck wrote:
Well, gentlemen, you have to ask for the fiber maps and have them placed in the contract as an exhibit.
Most of the large commercial banks are doing it. It's doable, but it does require effort.
Uhm, did you bother to read the NDAI report? The Federal Reserve learned several lessons. Fiber maps are not sufficient. If you are relying just on fiber maps, you are going to learn the same lesson again and again. The FAA, Federal Reserve, SFTI and SMART are probably at the top as far as trying to engineer their networks and maintain diversity assurances. But even the Federal Reserve found the cost more than it could afford. What commercial banks are doing is impressive, but only in a "commercially reasonable" way. Some residual risk and outages are always going to exist. No matter what the salesman tells you, Murphy still lives.
Hi Sean, I don't really understand your argument. I have no clue what this 'assurance' means in the context of managing telecommunications networks. No one is claiming that risk can be eliminated - but can be greatly reduced by proper physical diversity. And for the Federal Reserve, I don't necessarily believe they are experts in building telecommunication networks. They may be, but you have do more than just assert it. For all I know, the groups you cited are simply not that good at managing network risk. Maybe there is a compelling argument, but you have elaborate it. - R. -----Original Message----- From: owner-nanog@merit.edu on behalf of Sean Donelan Sent: Sun 1/21/2007 11:39 PM To: nanog@merit.edu Subject: RE: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted On Sun, 21 Jan 2007, Rod Beck wrote:
Well, gentlemen, you have to ask for the fiber maps and have them placed in the contract as an exhibit.
Most of the large commercial banks are doing it. It's doable, but it does require effort.
Uhm, did you bother to read the NDAI report? The Federal Reserve learned several lessons. Fiber maps are not sufficient. If you are relying just on fiber maps, you are going to learn the same lesson again and again. The FAA, Federal Reserve, SFTI and SMART are probably at the top as far as trying to engineer their networks and maintain diversity assurances. But even the Federal Reserve found the cost more than it could afford. What commercial banks are doing is impressive, but only in a "commercially reasonable" way. Some residual risk and outages are always going to exist. No matter what the salesman tells you, Murphy still lives. This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Atlantic has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment
On Jan 21, 2007, at 12:05 AM, Brian Wallingford wrote:
That's news?
The same still happens with much land-based sonet, where "diverse paths" still share the same entrance to a given facility. Unless each end can
Entrances, ha. Anyone remember that railroad tunnel in Baltimore ? And I am pretty sure that Fairfax County isn't much better. Regards Marshall
negotiate cost sharing for diverse paths, or unless the owner of the fiber can cost justify the same, chances are you're not going to see the ideal.
Money will always speak louder than idealism.
Undersea paths complicate this even further.
On Sun, 21 Jan 2007, Rod Beck wrote:
:What's really interesing is the fragility of the existing telecom infrastructure. These six cables were apparently very close to each other in the water. In other words, despite all the preaching about physical diversity, it was ignored in practice. Indeed, undersea cables very often use the same conduits for terrestrial backhaul since it is the most cost effective solution. However, that means that diversifying across undersea cables does not buy the sort of physical diversity that is anticipated. : :Roderick S. Beck :EMEA and North American Sales :Hibernia Atlantic
Or how about the ship that sank off the coast of Pakistan and cut both the SWM3 spur into Pakistan and the Flag link? Buying capacity on both systems into Pakistan would have done zero for you in that case ... By the way, I will try to remove the disclaimer tomorrow. Regards, Roderick S. Beck. Hibernia Atlantic http://www.hiberniaatlantic.com rod.beck@hiberniaatlantic.com This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Atlantic has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment
On Sun, 21 Jan 2007, Rod Beck wrote:
Or how about the ship that sank off the coast of Pakistan and cut both the SWM3 spur into Pakistan and the Flag link?
This is a function of the cable-head being close to a port and close to it's neighbor cable-head, right? If the cable heads were on opposite sides of the harbor or in adjacent towns that probably wouldn't have occured, right? In many places (based on a quick scan of the telegeography map from 200 posts ago...) it seems like cable landings are all very much centrally located in any one geographic area. There are like 5 on the east coast hear NYC, with many of the cables coming into the same landing place. Diversity on cable system and landing is probably one of your metrics to watch, not just cable system :( This probably also means: 1) you bought direct from the coalition running the cable system 2) you knew enough to ask: "is this in the same landing area or within 10km of same?' 3) your pointy haired person didn't say: "buy from the same provider, so we get one bill! ya know, 'on-net!'" and all that :( Social issues and budget issues probably kill real diversity 90% of the time :( (until you get bit)... which someone else already said I think?
By the way, I will try to remove the disclaimer tomorrow.
that'll keep the randy-complain-o-gram level down :)
In many places (based on a quick scan of the telegeography map from 200 posts ago...) it seems like cable landings are all very much centrally located in any one geographic area. There are like 5 on the east coast near NYC, with many of the cables coming into the same landing place.
That's true, but they're far enough apart that a single accident is unlikely to knock out the cables at more than one landing. The two in NJ cross Long Beach Island, then shallow Barnegat Bay, to the landing sites. Once crosses in Harvey Cedars and lands in Manahawkin, the other crosses in Beach Haven and lands in Tuckerton. My family has a beach house in Harvey Cedars a block from the cable crossing and it's clear they picked the sites because there is nothing there likely to mess them up. Both are summer communities with no industry, the commercial boat harbors, which are not very big, are all safely away from the crossings. The main way you know where they are is a pair of largish signs at each end of the street saying DON'T ANCHOR HERE and signs on the phone poles saying, roughly, don't dig unless there is an AT&T employee standing next to you. I haven't been to the landing site in Rhode Island, but I gather it is similarly undeveloped. Running a major cable in through a busy harbor is just a bad idea. so I'm not surprised that they don't do it here. R's, John
Hi John, There I disagree. Not with your statement, which is correct, but the implication. Most transatlantic cables are in the same backhaul conduit systems. For example, the three systems that land in New Jersey use the same conduit to backhaul their traffic to New York. The other three that land on Long Island use the same conduit system to reach NYC. By the way, the situation is even worse on the UK side where most of these cables are in one conduit system. And very few of those systems can avoid New York, which is a diversity requirement of many banks and one which the IP backbones should probably also adopt. You can't claim to have sufficient physical diversity when of the 7 major TransAtlantic cables, five of them terminate at the same end points. Only Apollo and Hibernia have diversity in that respect. Apollo's Southern cable lands in France and Hibernia lands in Canada and Northern England. And yes, I will remove the gargantuan disclaimer tomorrow. Regards, Roderick. -----Original Message----- From: owner-nanog@merit.edu on behalf of John Levine Sent: Sun 1/21/2007 9:05 PM To: nanog@nanog.org Cc: christopher.morrow@verizonbusiness.com Subject: Re: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted
In many places (based on a quick scan of the telegeography map from 200 posts ago...) it seems like cable landings are all very much centrally located in any one geographic area. There are like 5 on the east coast near NYC, with many of the cables coming into the same landing place.
That's true, but they're far enough apart that a single accident is unlikely to knock out the cables at more than one landing. The two in NJ cross Long Beach Island, then shallow Barnegat Bay, to the landing sites. Once crosses in Harvey Cedars and lands in Manahawkin, the other crosses in Beach Haven and lands in Tuckerton. My family has a beach house in Harvey Cedars a block from the cable crossing and it's clear they picked the sites because there is nothing there likely to mess them up. Both are summer communities with no industry, the commercial boat harbors, which are not very big, are all safely away from the crossings. The main way you know where they are is a pair of largish signs at each end of the street saying DON'T ANCHOR HERE and signs on the phone poles saying, roughly, don't dig unless there is an AT&T employee standing next to you. I haven't been to the landing site in Rhode Island, but I gather it is similarly undeveloped. Running a major cable in through a busy harbor is just a bad idea. so I'm not surprised that they don't do it here. R's, John This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Atlantic has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Marshall Eubanks Sent: Sunday, January 21, 2007 8:01 AM To: Brian Wallingford Cc: Rod Beck; nanog@merit.edu Subject: Re: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted
On Jan 21, 2007, at 12:05 AM, Brian Wallingford wrote:
That's news?
The same still happens with much land-based sonet, where "diverse paths" still share the same entrance to a given facility. Unless
each end
can
Entrances, ha. Anyone remember that railroad tunnel in Baltimore ? And I am pretty sure that Fairfax County isn't much better.
We have a railroad tunnel in Fairfax? On the less snarky side, I suspect that one wrong move by a backhoe along the Dulles Toll Road would screw about half the east coast. Jamie Bowden -- "It was half way to Rivendell when the drugs began to take hold" Hunter S Tolkien "Fear and Loathing in Barad Dur" Iain Bowen <alaric@alaric.org.uk>
"Jamie Bowden" <jamie@photon.com> writes:
Entrances, ha. Anyone remember that railroad tunnel in Baltimore ? And I am pretty sure that Fairfax County isn't much better.
We have a railroad tunnel in Fairfax?
single points of failure, like f'rinstance collapsed backbone segments on boone blvd. not railroad tunnels.
On the less snarky side, I suspect that one wrong move by a backhoe along the Dulles Toll Road would screw about half the east coast.
the only fiber that i'm aware of that is actually along the toll road itself belongs to the toll road folks. it would screw up the smart tag transponders for sure, but the ensuing traffic backups are unlikely to affect half the east coast (even though traffic in nova sometimes feels that way). sunrise valley dr. and sunset hills rd. are an entirely different matter though. update your maps before you go to us rentals eh? :-) ---rob
Jamie Bowden -- "It was half way to Rivendell when the drugs began to take hold" Hunter S Tolkien "Fear and Loathing in Barad Dur" Iain Bowen <alaric@alaric.org.uk>
That massive bundle of visible conduit running under the toll road where Centreville Road crosses always grabs my attention. I'm sure there's nothing critical inside of it. Marc -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Robert E. Seastrom Sent: Monday, January 22, 2007 11:04 AM To: Jamie Bowden Cc: Marshall Eubanks; nanog@merit.edu Subject: Re: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted
the only fiber that i'm aware of that is actually along the toll road itself belongs to the toll road folks. it would screw up the smart tag transponders for sure, but the ensuing traffic backups are unlikely to affect half the east coast (even though traffic in nova sometimes feels that way). sunrise valley dr. and sunset hills rd. are an entirely different matter though. update your maps before you go to us rentals eh? :-) ---rob
On 1/20/07, Brian Wallingford <brian@meganet.net> wrote:
That's news?
The same still happens with much land-based sonet, where "diverse paths" still share the same entrance to a given facility. Unless each end can negotiate cost sharing for diverse paths, or unless the owner of the fiber can cost justify the same, chances are you're not going to see the ideal.
Money will always speak louder than idealism.
Undersea paths complicate this even further.
Just the other night I was trolling marketing materials for various lit services from a number of providers and I ran across what I found to be an interesting PDF from the ol' SBC (can't find it at the moment). It was a government services product briefing and in it it detailed six levels of path diversity. These six levels ranged from "additional fiber on the same cable" to "redundant, diverse paths to redundant facility entrances into redundant wire centers". What struck me as interesting is that the government gets clearly definied levels of diversity for their services, but I've never run across anything similar in the commercial/enterprise/wholesale market. Are the Sprints/Verizons/ATTs/FLAGs/etc of the world clearly defining levels of diversity for their services to people?
On Sun, 21 Jan 2007, Aaron Glenn wrote:
Just the other night I was trolling marketing materials for various lit services from a number of providers and I ran across what I found to be an interesting PDF from the ol' SBC (can't find it at the moment). It was a government services product briefing and in it it detailed six levels of path diversity. These six levels ranged from "additional fiber on the same cable" to "redundant, diverse paths to redundant facility entrances into redundant wire centers". What struck me as interesting is that the government gets clearly definied levels of diversity for their services, but I've never run across anything similar in the commercial/enterprise/wholesale market.
I believe such levels of diversity and detail were specifically mandated for the Fedwire.
Are the Sprints/Verizons/ATTs/FLAGs/etc of the world clearly defining levels of diversity for their services to people?
From past discussions with them when I was in the ISP world, I'd have to say for the most part the answer is no, and the bits of info that deviated from that stance were normally divulged under NDA.
jms
Well, I work for an undersea cable system and we are quite to willing to share the information under NDA that is required to make an intelligent decision. That means the street-level fiber maps and details of the undersea routes. However, there is a general reluctance because so many carriers are using the same conduits. A lot of fiber trunks can put in a conduit system so it was the norm for carriers to joint builds. For example, in the NYC metropolitan area virtually all carriers use the same conduit to move their traffic through the streets of New York. And again, I will remove the disclaimer. Regards, Roderick. -----Original Message----- From: owner-nanog@merit.edu on behalf of Aaron Glenn Sent: Sun 1/21/2007 8:40 PM To: nanog@merit.edu Subject: Re: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted On 1/20/07, Brian Wallingford <brian@meganet.net> wrote:
That's news?
The same still happens with much land-based sonet, where "diverse paths" still share the same entrance to a given facility. Unless each end can negotiate cost sharing for diverse paths, or unless the owner of the fiber can cost justify the same, chances are you're not going to see the ideal.
Money will always speak louder than idealism.
Undersea paths complicate this even further.
Just the other night I was trolling marketing materials for various lit services from a number of providers and I ran across what I found to be an interesting PDF from the ol' SBC (can't find it at the moment). It was a government services product briefing and in it it detailed six levels of path diversity. These six levels ranged from "additional fiber on the same cable" to "redundant, diverse paths to redundant facility entrances into redundant wire centers". What struck me as interesting is that the government gets clearly definied levels of diversity for their services, but I've never run across anything similar in the commercial/enterprise/wholesale market. Are the Sprints/Verizons/ATTs/FLAGs/etc of the world clearly defining levels of diversity for their services to people? This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Atlantic has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment
participants (11)
-
Aaron Glenn -
Brian Wallingford -
Chris L. Morrow -
Jamie Bowden -
John Levine -
Justin M. Streiner -
Marcus H. Sachs -
Marshall Eubanks -
Robert E. Seastrom -
Rod Beck -
Sean Donelan