Vulnerbilities of Interconnection
Thanks for all the answers on the previous question about Equinix. The reason for the question is that we are working on a study here at George Mason on assesing the vulnerbilities of the Internet and telecommunications infrastructure to physical attack. One part that we are looking at are the vulnerbilites of interconnection facilites. Doomsday scenarios like 60 Hudson street being a target instead of the WTC or the loss of MAE east, etc. The earlier question about Equinix and the Big 7 was really a vulnerbility question. Are interonnection locations aggreating or dispersing. How real of a threat is the physical loss of a major interconnection point. Incidents like the Balitmore train tunnel wreck point in one direction. We are working on putting several models together to try and simulate some of these scenarios with varying degrees of success, but I would very much like to avoid doing the research in a vaccuum. I was hoping a discussion on NANOG wold be a good first step. The project is quite hot with the politicos and I very much want to make sure to best recommendations are made. Formal industrsy cooperation is one side of this, but I think a lot of information can be gained from an informal approach as well. Any and all feedback is greatly appreciated Thanks, sean
On Thu, 5 Sep 2002 sgorman1@gmu.edu wrote:
very much like to avoid doing the research in a vaccuum. I was hoping a discussion on NANOG wold be a good first step. The project is quite hot with the politicos and I very much want to make sure to best recommendations are made. Formal industrsy cooperation is one side of this, but I think a lot of information can be gained from an informal approach as well. Any and all feedback is greatly appreciated
http://www.infosecuritymag.com/2002/sep/2002survey/voices/verylarge.shtml On security reporting... "Since Sept. 11, state, local and federal authorities have tried to get their arms around the potential threats to the nation's infrastructure--including the telecommunications infrastructure. They have asked us questions like, 'What are your 100 most vulnerable places in the network?'" "As much as we would like to help the government in its attempt to help us, we believe it would be counterproductive to share such information widely because if it were released, it would provide a terrorist with a roadmap to our key locations. Unless the government agrees that it can protect our information, we will continue to respectfully decline such blanket requests." Bill Smith CTO and President of Interconnection Services, BellSouth
The crux of the issue are FOIA requests. The government won't make these types of vulnerability reports immmune to FOIA requests - thus a foreign terrorist or home-grown "farmbelt fuhrer" could simply order up a list of the most vulnerable sites, and select some to attack. Due to the distributed nature of the internet, and the routing protocols that regulate it's traffic flow, there is no single point of failure. However, we have seen how concerted attacks can be made at multiple locations, almost simultaneously. If the government could agree to allow this information to remain confidential, it would greatly expedite the process of hardening appropriate facilities, and identifying weaknesses. - Daniel Golding
Sean Donelan Said...
On Thu, 5 Sep 2002 sgorman1@gmu.edu wrote:
very much like to avoid doing the research in a vaccuum. I was hoping a discussion on NANOG wold be a good first step. The project is quite hot with the politicos and I very much want to make sure to best recommendations are made. Formal industrsy cooperation is one side of this, but I think a lot of information can be gained from an informal approach as well. Any and all feedback is greatly appreciated
http://www.infosecuritymag.com/2002/sep/2002survey/voices/verylarge.shtml
On security reporting... "Since Sept. 11, state, local and federal authorities have tried to get their arms around the potential threats to the nation's infrastructure--including the telecommunications infrastructure. They have asked us questions like, 'What are your 100 most vulnerable places in the network?'"
"As much as we would like to help the government in its attempt to help us, we believe it would be counterproductive to share such information widely because if it were released, it would provide a terrorist with a roadmap to our key locations. Unless the government agrees that it can protect our information, we will continue to respectfully decline such blanket requests."
Bill Smith CTO and President of Interconnection Services, BellSouth
Unnamed Administration sources reported that Daniel Golding said:
The crux of the issue are FOIA requests. The government won't make these types of vulnerability reports immmune to FOIA requests - thus a foreign terrorist or home-grown "farmbelt fuhrer" could simply order up a list of the most vulnerable sites, and select some to attack.
Suffice to say, there's another side to the story as well. There is already a FOIA exemption, but the current Administration is making a daily policy of denying virtually all FOIA requests. Judges are not always that submissive; hence the push for new legislation. You might look at epic.org and aclu.org for other views than than those of Clark & the Ministry of Fatherlan^H^H^H Homeland Security. -- A host is a host from coast to coast.................wb8foz@nrk.com & no one will talk to a host that's close........[v].(301) 56-LINUX Unless the host (that isn't close).........................pob 1433 is busy, hung or dead....................................20915-1433
At 12:44 PM 9/5/2002 -0400, sgorman1@gmu.edu wrote:
One part that we are looking at are the vulnerbilites of interconnection facilites.
A quick point...Several folks have postulated that the internal (non-physical) threat dwarfs that of the physical threat, due to the lack of visibility, the difficulty of tracking and coordinating a response, and the millions of vulnerable systems world-wide capable of launching an internal attack. A physical attack (a hole in a wall for example) can typically be detected and corrected in a matter of hours or days, while an effective internal attack could be varied in time and scope causing at least as much damage invisibly for a much longer period of time. That said, a few years back I wrote the "Interconnection Strategies for ISPs" white paper, which speaks to the economics of peering using exchange points vs. using pt-to-pt circuits. It documents a clear break even point where large capacity circuits (or dark fiber loops) into an IX with fiber cross connects within a building are a better fit (financially) than pt-to-pt circuits. A couple physical security considerations came out of that research: 1) Consider that man holes are not always secured, providing access to metro fiber runs, while there is generally greater security within colocation environments 2) It is faster to repair physical disruptions at fewer points, leveraging cutovers to alternative providers present in the collocation IX model, as opposed to the Direct Circuit model where provisioning additional capacities to many end points may take days or months. Finally, I have seen a balancing act between how much it costs to protect against a disruption versus the cost of the disruption. In today's economy (unlike say a few years ago) more folks seem to be focused on doing this mathematically calculation rather than just picking full mesh interconnect topologies. Bill --------------------------------------------------------------------------------------------------------------- William B. Norton <wbn@equinix.com> 650.315.8635 Co-Founder and Chief Technical Liaison Equinix, Inc. Yahoo Instant Messenger ID: WilliamBNorton
That said, a few years back I wrote the "Interconnection Strategies for ISPs" white paper, which speaks to the economics of peering using exchange points vs. using pt-to-pt circuits. It documents a clear break even point where large capacity circuits (or dark fiber loops) into an IX with fiber cross connects within a building are a better fit (financially) than pt-to-pt circuits.
This obviously would be a thesis of Equinix and other collo space providers, since this is exactly the service that they provide. It won't, hower, be a thesis of any major network that either already has a lot of infrastructure in place or has to be a network that is supposed to survive a physical attack.
A couple physical security considerations came out of that research: 1) Consider that man holes are not always secured, providing access to metro fiber runs, while there is generally greater security within colocation environments
This is all great, except that the same metro fiber runs are used to get carriers into the super-secure facility, and, since neither those who originate information, nor those who ultimately consume the information are located completely within facility, you still have the same problem. If we add to it that the diverse fibers tend to aggregate in the basement of the building that houses the facility, multiple carriers use the same manholes for their diverse fiber and so on.
2) It is faster to repair physical disruptions at fewer points, leveraging cutovers to alternative providers present in the collocation IX model, as opposed to the Direct Circuit model where provisioning additional capacities to many end points may take days or months.
This again is great in theory, unless you are talking about someone who is planning on taking out the IX not accidently, but deliberately. To illustrate this, one just needs to recall the infamous fiber cut in McLean in 1999 when a backhoe not just cut Worldcom and Level(3) circuits, but somehow let a cement truck to pour cement into Verizon's manhole that was used by Level(3) and Worldcom. Alex
At 02:45 PM 9/5/2002 -0400, alex@yuriev.com wrote:
This obviously would be a thesis of Equinix and other collo space providers, since this is exactly the service that they provide. It won't, hower, be a thesis of any major network that either already has a lot of infrastructure in place or has to be a network that is supposed to survive a physical attack.
Actually, the underlying assumption of this paper is that major networks already have a large global backbone that need to interconnect in n-regions. The choice between Direct Circuits and Colo-based cross connects is discussed and documented with costs and tradeoffs. Surviving a major attack was not the focus of the paper...but... When I did this research I asked ISPs how many Exchange Points they felt were needed in a region. Many said one was sufficient, that they were resilient across multiple exchange points and transit relationships, and preferred to engineer their own diversity separate from regional exchanges. A bunch said that two was the right number, each with different operating procedures, geographic locations, providers of fiber, etc. , as different as possible. Folks seemed unanimous about there not being more than two IXes in a region, that to do so would splinter the peering population. Bill Woodcock was the exception to this last claim, positing (paraphrasing) that peering is an local routing optimization and that many inexpensive (relatively insecured) IXes are acceptable. The loss of any one simply removes the local routing optimization and that transit is always an alternative for that traffic.
A couple physical security considerations came out of that research: 1) Consider that man holes are not always secured, providing access to metro fiber runs, while there is generally greater security within colocation environments
This is all great, except that the same metro fiber runs are used to get carriers into the super-secure facility, and, since neither those who originate information, nor those who ultimately consume the information are located completely within facility, you still have the same problem. If we add to it that the diverse fibers tend to aggregate in the basement of the building that houses the facility, multiple carriers use the same manholes for their diverse fiber and so on.
Fine - we both agree that no transport provider is entirely protected from physical tampering if its fiber travels through insecure passageways. Note that some transport capacity into an IX doesn't necessarily travel along the same path as the metro providers, particularly those IXes located outside a metro region. There are also a multitude of paths, proportional to the # of providers still around in the metro area, that provide alternative paths into the IX. Within an IX therefore is a concentration of alternative providers, and these alternative providers can be used as needed in the event of a path cut.
2) It is faster to repair physical disruptions at fewer points, leveraging cutovers to alternative providers present in the collocation IX model, as opposed to the Direct Circuit model where provisioning additional capacities to many end points may take days or months.
This again is great in theory, unless you are talking about someone who is planning on taking out the IX not accidently, but deliberately. To illustrate this, one just needs to recall the infamous fiber cut in McLean in 1999 when a backhoe not just cut Worldcom and Level(3) circuits, but somehow let a cement truck to pour cement into Verizon's manhole that was used by Level(3) and Worldcom.
Terrorists in cement trucks? Again, it seems more likely and more technically effective to attack internally than physically. Focus again here on the cost/benefit analysis from both the provider and disrupter perspective and you will see what I mean.
Alex
Actually, the underlying assumption of this paper is that major networks already have a large global backbone that need to interconnect in n-regions. The choice between Direct Circuits and Colo-based cross connects is discussed and documented with costs and tradeoffs. Surviving a major attack was not the focus of the paper...but...
If the major networks in questions are long-distance companies and local phone companies, then they are already interconnected in N places. For one reason or another, at the present time they are simply not running IP at those points. It is equivalent to having networks in the common facilities that choose not to interconnect.
When I did this research I asked ISPs how many Exchange Points they felt were needed in a region. Many said one was sufficient, that they were resilient across multiple exchange points and transit relationships, and preferred to engineer their own diversity separate from regional exchanges.
Very few ISPs in reality have any physical divercity.
A bunch said that two was the right number, each with different operating procedures, geographic locations, providers of fiber, etc. , as different as possible. Folks seemed unanimous about there not being more than two IXes in a region, that to do so would splinter the peering population.
Security is always considered a waste of money. It is nothing new. The reason for that is that it is impossible to see the benefits when there is no problem.
Fine - we both agree that no transport provider is entirely protected from physical tampering if its fiber travels through insecure passageways. Note that some transport capacity into an IX doesn't necessarily travel along the same path as the metro providers, particularly those IXes located outside a metro region. There are also a multitude of paths, proportional to the # of providers still around in the metro area, that provide alternative paths into the IX. Within an IX therefore is a concentration of alternative providers, and these alternative providers can be used as needed in the event of a path cut.
They are using the same paths to get into the buildings. If they are not using the same paths exactly, their paths are close enough to each other within N meters from the building.
2) It is faster to repair physical disruptions at fewer points, leveraging cutovers to alternative providers present in the collocation IX model, as opposed to the Direct Circuit model where provisioning additional capacities to many end points may take days or months.
This again is great in theory, unless you are talking about someone who is planning on taking out the IX not accidently, but deliberately. To illustrate this, one just needs to recall the infamous fiber cut in McLean in 1999 when a backhoe not just cut Worldcom and Level(3) circuits, but somehow let a cement truck to pour cement into Verizon's manhole that was used by Level(3) and Worldcom.
Terrorists in cement trucks?
No, but since that caused a multi-day outages for certain customers due to a single point of failure, I am sure someone can appreciate the outage that can be caused by detonating a hundred killograms of high explosives inside a collo facility.
Again, it seems more likely and more technically effective to attack internally than physically. Focus again here on the cost/benefit analysis from both the provider and disrupter perspective and you will see what I mean.
Easily accessible brute-force *always* wins. Any chain is not stronger than its weakest link and concentrated infrastructure was, is and always will be, the weakest link if one can mount an attack using bruce-force. Neither the data centers, nor COs nor exchange points that are vital so far had been designed in a way that they could withstand a direct physical attack even by an individual with a handgun, not to mention anyone carrying explosives. When that problem gets solved, we can concentrate on attracks against IP infrastructure. Alex
On Thu, 05 Sep 2002 12:04:16 -0700 "William B. Norton" <wbn@equinix.com> wrote:
Terrorists in cement trucks?
Again, it seems more likely and more technically effective to attack internally than physically. Focus again here on the cost/benefit analysis from both the provider and disrupter perspective and you will see what I mean.
reflecting on my experiences in such facilities... usually all i've ever needed to do at the door is sign in after proving that i work for a company that has colo space. my boxes of equipment have never been inspected. therefore, to attack many colo facilities, it is sufficient to sign contracts that i never intend to honor and then carry boxes of "stuff" up that has nothing to do with colo. richard -- Richard Welty rwelty@averillpark.net Averill Park Networking 518-573-7592 Unix, Linux, IP Network Engineering, Security
On Thu, 5 Sep 2002, Richard Welty wrote:
usually all i've ever needed to do at the door is sign in after proving that i work for a company that has colo space. my boxes of equipment have never been inspected.
How many banks know what their customers have put in the safe deposit boxes stored in the bank's vaults? Do you want guard rummaging through your equipment? Even if they opened the boxes how would a guard know what's inside a 12000 router? Rent the movie Infinity (1996) or read Richard Feynman's books describing the security around The Manhattan Project at Los Alamos.
participants (7)
-
alex@yuriev.com
-
Daniel Golding
-
David Lesher
-
Richard Welty
-
Sean Donelan
-
sgorman1@gmu.edu
-
William B. Norton