But it does bring up another issue. As far as I know, Exodus is the only colocation designated a "national infrastructure asset. http://www.thestreet.com/tech/internet/1090327.html

I have no idea what that means in practical terms. But I did want to raise the question. As an industry, we aren't vertically integrated. Instead its an inter-linked set of dependencies. Its not like the old days when the government could just call up Ma Bell, and find out what is happening.

It is my understanding that the US Government has "national infrastructure" inside Exodus facilities. As for what that means precisely, it's anyone's guess. Security at the Exodus facility in Atlanta has always been pretty good, but I wouldn't call it stellar. I know that the USG at one point had significant infrastructure inside the Atlanta facility. I recall seeing some serious looking cages around routers and Sun gear, but I couldn't say for sure; it was a long time ago. Tim

On Sun, 23 Sep 2001, Bob Bownes wrote:

USG has assets inside a number of co-lo providers. One of the companies I work with happens to provide some under sub-contract to DOE. No details available as to where it is however. :-}

Various departments and agencies of the US, state and local government have assets in lots and lots of co-lo providers. I had several circuits designated for "continuity of government." However, no other provider has been designated by the White House as a "National Infrastructure Asset." Sprint and Worldcom provide FTS2001, but they haven't been designated "national infrastructure asset" by the White House. Other than an Exodus press release, I can't find any record of the phrase in the Code of Federal Regulations, the Federal Register, or any official White House public record in the govdocs database. I don't know if being designated a "national infrastructure asset" is the same as the 50th wedding anniversary greeting the White House sent my grandparents (which looks very impressive framed on the mantel), or if it actually has some practical effect. But my question wasn't really a debate over the phrase "national infrastructure asset," but whether there are any other assets the Internet community believes should be included in pre-planned responses.

On Mon, 24 Sep 2001 00:00:30 +0100 Alex Bligh <alex@alex.org.uk> wrote: AB> 111 8th Avenue. These buildings are significant PoF in NY, and AB> NY itself is pretty much an SPoF as far as transatlantic communication AB> is concerned. Well.. I would declare the SPoF rather in the whole area. Fibre trails are coming in from the seaside mostly near Mineola. AB> A preplanned response would be useful here. Not having AB> the PoF's would be more useful. The question has to be answered by the Fibre Owners. They can solve the issue. If they do is another question --jan -- Jan-Ahrent Czmok http://www.czmok.de/ mailto:jan.czmok@epost.de Tel. +49-(0)-6127-997448 Cell. +49-(0)-174-3074404 "Beware of routing loops.. You may fall into them"

On Sun, 23 Sep 2001 19:55:35 -0400 Jeffrey Meltzer <meltzer@villageworld.com> wrote: JM> Last I checked, Mineola was right in the middle of the Island :) JM> Most fiber comes via Shirley in the Town of Brookhaven. I only visited NY once yet (would one it more often - nice city) when visiting Lightning Internet Services. But i guess, You're right... --jan -- Jan-Ahrent Czmok http://www.czmok.de/ mailto:jan.czmok@epost.de Tel. +49-(0)-6127-997448 Cell. +49-(0)-174-3074404 "Beware of routing loops.. You may fall into them"

On Sun, 23 Sep 2001 16:59:34 -0700 "Wayne E. Bouchard" <web@typo.org> wrote: WEB> When refering to points of failure, its important to note that not WEB> just economic centers are vulerable. I mean, ever notice how much WEB> fiber goes through kansas city? Therefore i would declare all important exchange points, cities and anything which has more than 1 telco there as volunerable. --jan -- Jan-Ahrent Czmok http://www.czmok.de/ mailto:jan.czmok@epost.de Tel. +49-(0)-6127-997448 Cell. +49-(0)-174-3074404 "Beware of routing loops.. You may fall into them"

On Sunday, September 23, 2001, at 09:33 PM, Alex Rubenstein wrote:

Mostly? I donut think so; my recollection is a vast majority of trans-atlantic stuff coming in ocean and monmouth county, NJ (manasquan, etc).

IIRC, a good deal of transatlantic fibre lands at the National Guard beach in Sea Girt (no public access). -Bill

tat-14's landing sites are green hill, rhode-island and shirley, new york are they not? joelja On Mon, 24 Sep 2001, Anthony Townsend wrote:

dont forget too that a good part of transatlantic fiber is backhauled through the holland tunnel into lower manhattan, one of many reasons why all the co-los are on the lower west side (in addition to cheap buildings with heavy load capacity). also one of the (again many) reasons i presume the tunnel has remained closed.

according to Telegeography, Inc of Washington DC most of the BIG capacity cables like Mercus-1 and TAT14 come ashore in New Jersey around Manasquan. its mostly the older stuff that arrives on Long Island.

--- Alex Rubenstein <alex@nac.net> wrote:

...

On Mon, 24 Sep 2001, Jan-Ahrent Czmok wrote:

...

On Mon, 24 Sep 2001 00:00:30 +0100 Alex Bligh <alex@alex.org.uk> wrote:

AB> 111 8th Avenue. These buildings are

...

AB> NY itself is pretty much an SPoF as far as

significant PoF in NY, and transatlantic communication

...

AB> is concerned.

Well.. I would declare the SPoF rather in the whole area. Fibre trails are coming in from the seaside mostly near Mineola.

Mostly? I donut think so; my recollection is a vast majority of trans-atlantic stuff coming in ocean and monmouth county, NJ (manasquan, etc).

===== Anthony Townsend Taub Urban Research Center New York University email: anthony.townsend@nyu.edu tel: 212-998-7502 SMS: anthony@voicestream.net (140 chars max) Yahoo Messenger: townsnda

__________________________________________________ Do You Yahoo!? Get email alerts & NEW webcam video instant messaging with Yahoo! Messenger. http://im.yahoo.com

-- -------------------------------------------------------------------------- Joel Jaeggli joelja@darkwing.uoregon.edu Academic User Services consult@gladstone.uoregon.edu PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E -------------------------------------------------------------------------- It is clear that the arm of criticism cannot replace the criticism of arms. Karl Marx -- Introduction to the critique of Hegel's Philosophy of the right, 1843.

On Mon, 24 Sep 2001, Alex Bligh wrote:

I think the effect on the Internet & Telecoms infrastructure (as opposed more important things such as human life) would have been far greater had the 2 NY planes hit 60 Hudson and 111 8th Avenue. These buildings are significant PoF in NY, and NY itself is pretty much an SPoF as far as transatlantic communication is concerned. A preplanned response would be useful here. Not having the PoF's would be more useful.

There are, and always will be points of failure. The mistake people make is thinking they can build a bunker strong enough. You can't build a building, vault, bunker, missle silo which can withstand everything. The question is really how do you manage your network diversity. The loss of something like 60 Hudson shouldn't cause more than an annoying route flap in your network. Exchange points like MAE-East have completely failed in the past. There have been multiple fiber cuts in the same day. Well designed networks continued to work. Although I'l admit, it is hard work. You can't rely on a carrier to do it for you. Grooming happens. The biggest risk in most networks aren't the national exchange points, although they get the press. All national providers, and most regional providers are interconnected to multiple geographically diverse points. The point in the network with limited diversity is the LEC end-offices. And I use "LEC" deliberately, because even if you use a CLEC, most of the time you are using the LEC for the last mile. You may have great path diversity for 3,000 miles across the continent, but then you go through 140 West Street, or Rochelle Park or some other LEC office. Even if you thought you went through Broad Street, a lot of folks found out they were in fact routed through West Street. Ok, so I just said you can't build a bunker strong enough. Are carrier hotels, like 60 Hudson, history? I don't think so. They have better diversity, better backup systems, and better security than normal offices. The concentration of bandwidth and carriers allowed very fast restoration and re-routing between locations still standing. Carriers were using other carrier's circuits to restore facilities. We may see some movement away from downtown areas, where the danger is a near miss instead of being a direct target. If you can't afford to build your own colo, are you better off hanging off a spoke from a LEC central office. Or putting your equipment in a building with built in diversity. In general, it is best to put your equipment as close as possible to the point of diversity. You can do this by either moving the point of diversity close to you, or moving your equipment closer to the diversity. Being at the end of a 6,000 foot T1 circuit to a CO is the worst of both worlds. You will go down if an airplane hits either your office, or the CO, or any point along that 6,000 feet of T1 circuit.

On Mon, 24 Sep 2001, Alex Bligh wrote:

My point being that building a network which doesn't have more than an annoying route flap, if /both/ 60 Hudson and 111 8th avenue are lost, is extremely hard (*) (especially if it has a transatlantic component). And that's true even if you have your own fiber.

(*) hard means that it isn't compatible with existing topologies, and building new ones is expensive.

Which brings me back to my original question. Are there specific locations which are more important to the functioning of the Internet than others? You can't simply say everything is important. The FAA breaks airports down into several catagories, large airports, medium airports and small airports. A large airport has 1% or more of the passenger traffic. Are there specific locations which handle 1% or more of the Internet's traffic (assuming we had figures for the total amount of traffic).

"Grant A. Kirkwood" wrote:

Sean Donelan wrote:

...

On Mon, 24 Sep 2001, Alex Bligh wrote:

The national air traffic system makes a poor analogy to the Internet in this case, IMHO. If O'Hare got nuked tomorrow, we'd have some serious disruption in passenger traffic. If PAIX fell into the ocean, OTOH, traffic would simply route around it. Isn't that how we try to engineer the Internet?

So in other words, yes, everything is important, and yes, nothing is particularly important.

But there was a point in time when taking out a certain parking garage in Va could have caused us a very great deal of difficulty. But I'd say we are past that, for the most part. Bob

Sean Donelan wrote:

On Mon, 24 Sep 2001, Bob Bownes wrote:

...

But there was a point in time when taking out a certain parking garage in Va could have caused us a very great deal of difficulty. But I'd say we are past that, for the most part.

Are we?

When 25 Broadway failed, approximately 1% of the global Internet routing table also disappeared. Which I would guess qualifies it as a "major" hub.

But does that mean that X number of sites were unreachable, or that there were simply Y number fewer routes to X sites? (Excluding those *directly* affected, ie; those *in* 25 Broadway)

Verizon still has 100,000 lines out of service, and only now begun to restore service to "small" businesses.

Yes, but my understanding was that we were referring to IP traffic. POTS doesn't exactly have a built-in routing protocol.

A couple of years ago a fiber cut in Ohio disrupted about 20% of the Internet routing table.

But again, does this mean that 20% of the Internet was unreachable, or that there were 20% fewer routes to a given number of (hopefully multihomed) sites? No, this question is not rhetorical... I simply don't have any imperical evidence to look at that could adequately answer this question. Grant -- Grant A. Kirkwood - grant@virtical.net Chief Technology Officer - Virtical Solutions, Inc. http://www.virtical.net/

Sean Donelan wrote:

...

On Mon, 24 Sep 2001, Bob Bownes wrote:

...

But there was a point in time when taking out a certain parking garage in Va could have caused us a very great deal of difficulty. But I'd say we are past that, for the most part.

Are we?

When 25 Broadway failed, approximately 1% of the global Internet routing table also disappeared. Which I would guess qualifies it as a "major" hub.

But does that mean that X number of sites were unreachable, or that there were simply Y number fewer routes to X sites? (Excluding those *directly* affected, ie; those *in* 25 Broadway)

From what point did 1% of the routing table disappear? Was the same visable from multiple, diverse points? I expect that from some perspectives, 100% of the routing table disappeared and some places didn't even see a blip. --bill

On Tue, 25 Sep 2001, Sean Donelan wrote: :But percentage of routes is just one way to measure "importance." :It may not be the best way. Other methods include : : 1. Number of stock options owned by Very Important People : 2. CAIDA skitter traces of routers of confluence : 3. Number of OC-192 links in a building : 4. Number of "Tier 1" providers in a building : 5. Government fiat : 6. Wait for the building to fall down and see what happens Is there a geometric method of measuring the 'meshedness' of a given set? If you take all the as-paths from a sampling of peers across the Internet, and show the relative density of where the respective paths converge, you can get a good picture of who's transiting the most routes. Now this doesn't show physical connections, as an AS can represent an area that spans continents, but it shows who is responsible, which in any DRP/BCP is among the first things established. :Assuming there are locations more impotant than others, should :we do anything? Or should we just hope no one else figures out :where they are? Well, the gov, or an industry consortium can find these dense transit areas and require that organizations with (eg.)2 or more peers have some semblance of a DRP/BCP plan that can be audited. The plan doesn't nessecarily have to garuntee connectivity, but establish whether they can be trusted to route packets if the DRP/BCP has to be initialized. After all, we are talking about the Internet, and though many orgs control lots of different parts of the infrastructure, maybe a plan just for layer 3 might be worth persuing. So maybe a large percentage of traffic gets routed through 60 hudson, the providers that are located there would each have to have diverse enough infrastructure/routing policies to contend with the unavailability of their equipment at that facility, to qualify as an Infrastructure Carrier. In short, I think that a plan like this should start on ground we all know and have the power to negotiate on, which is layer 3. -- batz Reluctant Ninja Defective Technologies

note that richer meshes may increase forwarding reliability but they exacerbate routing convergence problems. see abha's nanog presentation. randy

Some cites have peering and co-locations diversity, some don't. InfoMart & Westin Building come to mind. Those should rank high by your list. At 13:19 -0400 25-09-2001, Sean Donelan wrote:

On Mon, 24 Sep 2001 bmanning@vacation.karoshi.com wrote:

...

...

...

When 25 Broadway failed, approximately 1% of the global Internet routing table also disappeared. Which I would guess qualifies it From what point did 1% of the routing table disappear? Was the same visable from multiple, diverse points?

I expect that from some perspectives, 100% of the routing table disappeared and some places didn't even see a blip.

The Internet as we know it is just a collective illusion.

You are correct from one side of the partion, 99% of the routes disappeared and on the other side 1% of the routes disappeared. I checked four different BGP feeds from a mix of providers, and they were fairly consistent.

But percentage of routes is just one way to measure "importance." It may not be the best way. Other methods include

1. Number of stock options owned by Very Important People 2. CAIDA skitter traces of routers of confluence 3. Number of OC-192 links in a building 4. Number of "Tier 1" providers in a building 5. Government fiat 6. Wait for the building to fall down and see what happens

Assuming there are locations more impotant than others, should we do anything? Or should we just hope no one else figures out where they are?

-- Joseph T. Klein +1 414 915 7489 Senior Network Engineer jtk@titania.net Adelphia Business Solutions joseph.klein@adelphiacom.com "... the true value of the Internet is its connectedness ..." -- John W. Stewart III

Which brings me back to my original question. Are there specific locations which are more important to the functioning of the Internet than others? You can't simply say everything is important. The FAA

Ona physical layer, there are WAY too many points where lots of fiber from multiple providers crosses the same bridge.... uses the same railroad right of way... too many to think about and stay sane,

Everyone: I know it's difficult to refrain from comment, but let's try to remember that the bad guys read this list too. While they may not have the knowledge of critical communication infrastructure points, they can certainly find and target them if we point them in the right direction. This pertains not only to our side of the ponds, but to overseas as well. We all know where the 'soft targets' of our infrastructures are located - let's keep it to ourselves or, at the very least, within small private discussion groups where everyone knows everyone and not on the public list. Tim

secrets are bad for everyone... joelja On Mon, 24 Sep 2001, Timothy R. McKee wrote:

Everyone: I know it's difficult to refrain from comment, but let's try to remember that the bad guys read this list too. While they may not have the knowledge of critical communication infrastructure points, they can certainly find and target them if we point them in the right direction. This pertains not only to our side of the ponds, but to overseas as well. We all know where the 'soft targets' of our infrastructures are located - let's keep it to ourselves or, at the very least, within small private discussion groups where everyone knows everyone and not on the public list. Tim

-- -------------------------------------------------------------------------- Joel Jaeggli joelja@darkwing.uoregon.edu Academic User Services consult@gladstone.uoregon.edu PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E -------------------------------------------------------------------------- It is clear that the arm of criticism cannot replace the criticism of arms. Karl Marx -- Introduction to the critique of Hegel's Philosophy of the right, 1843.

I know it's difficult to refrain from comment, but let's try to remember that the bad guys read this list too. While they may not have the knowledge of critical communication infrastructure points, they can certainly find and target them if we point them in the right direction.

I'd actually argue the opposite. It's difficult to face this, but we know we ARE vulnerable. The important long term solution is that we need to address our weaknesses. By acknowleding where the critical points are, AND PLANNING TO DEAL WITH THEIR LOSS, we make the system that much harder to defeat. It's burying of our heads in the sand that caused the problem in the first place (the WTC bombing, that is). People thought it would be easier to pretend terrorists would never figure something like this out, than asking if 'feel-good' measures with no real substance were worth bothering with.

This pertains not only to our side of the ponds, but to overseas as well. We all know where the 'soft targets' of our infrastructures are located - let's keep it to ourselves or, at the very least, within small private discussion groups where everyone knows everyone and not on the public list.

While I'd agree that there's no reason to constantly advertise the problems, it's in free and open discussions that the best solutions are often found. Groups of known members have a tendency to fall into patterns, missing the same holes and making the same mistakes. The more open the group, the more likely any solutions will solve the general case instead of focusing on the same details and missing others. (Granted, it's also more likely that discussions will fall into petty bickering, but hopefully we're all motivated enough to come back to looking at the real problem)

Being at the end of a 6,000 foot T1 circuit to a CO is the worst of both worlds.

Hum... but in this case I have direct control on the physical access to my hardware. For some, this outweighes the "proximity to diversity" of telecoms paths. If I want (non-authentic) copies of the data spread far and wide, thats a service I can buy from a number of suppliers. I don't have to put -MY- hardware close to the telecoms mesh. I want my keyserver where I know I can protect it from barbarian hordes. In such a case, a long, thin gauntlet might be the best choice. --bill