--On fredag, fredag 28 nov 2008 17.10.14 +0000 Simon Waters <simonw@zynet.net> wrote:

I'm thinking physical security is over done in some data centers. Sure it is a great idea to make sure no one steals the hardware, but much beyond that and allowing in expected personnel only, it soon gets to being counter productive.

I was once back-up for a facility so "secure" I never got to visit it?! I'm not saying I might not have been that useful if I was ever called on to provide support - guess we'll never know. Although for that one I did at least happen to know where it was despite it not being sign posted.

There are places whose location we do not talk about, where important stuff gets done, like peering. In Sweden, the Post and Telecommunications Authority has oversight over a number of first-rate data centres that are designed for those bits and pieces of infrastructure that need to work under all circumstances. Typically they rent space to telcos and ISP's for things like important central systems, backbone routers / transmission etc. The largest Internet exchange in Sweden, Netnod, has its five largest sites in these facilities. These data centres are designed to Swedish military command center specifications (not like Cheyenne Mountain but significantly better than, say, a Minuteman site) to withstand a number of adverse conditions, like near-misses from nuclear weapons, prolonged power outages, poison gas clouds, etc. Typically, they are buried in bedrock close to major cities. Exactly where is of course known in the business, but not so well that it is OK to post their locations on Nanog. Yes, we've got excellent bedrock in Sweden, and we use it ;-) -- Måns Nilsson M A C H I N A I left my WALLET in the BATHROOM!!

On 2008-11-28, at 16:04, Jean-François Mezei wrote:

If you look at Toronto, the main carrier hotel is quite famous at 151 Front Street, very near to the main train station, convention centre etc (aka: right at the core of the downtown). People who do not know about the internet infrastructure may not realise what this building is about, but anyone who knows how ISPs operate would know the strategic importance of that building.

People who do not know that there's a Front Street East as well as a Front Street West also like to fight their way through the mantrap to front desk security and demand to see the dentist. So if anybody ever finds an operational advantage to having equipment in a building regularly visited by people with bad teeth, bear that in mind. Joe

On Nov 28, 2008, at 4:04 PM, Jean-François Mezei wrote:

The thing about a carrier hotel is that it cannot be a secret location since you need to allow various carriers and ISPs to have physical access to the building so they can install/manage their servers/routers/switches.

The advantage of this swedish data centre is that even if its location is well known, it is pretty hard to harm the building. You can't run a truck full of explosives into it for instance.

Unfortunately, you also cannot run your own fiber there, colo equipment there, visit it for any reason, etc. I was going to say 'this probably hinders customers adoption at NetNod', but I know for a fact the "probably" is superfluous. -- TTFN, patrick

Fault free datacenters include neither people, nor computers, nor connectivity, nor HVAC, nor electricity. If you can eliminate those things you will have a 100% uptime datacenter.

Andrew

Is this the network equivalent of Yin and Yang, or Darkness and Light being the same? Perhaps it is like an old joke: "How many Microsoft programmers does it take to change a lightbulb?" "None, they just make darkness the new standard." I guess, if uptime is a measure of your promised availability, then if you promise total unavailability, your uptime is 100% if no-one can reach you during the measured period. Not terribly useful, however, and likely to get breached, when those with means want to find out what you're hiding.

On Nov 30, 2008, at 10:50 PM, Niels Bakker wrote:

* patrick@ianai.net (Patrick W. Gilmore) [Mon 01 Dec 2008, 02:34 CET]:

...

On Nov 28, 2008, at 4:04 PM, Jean-François Mezei wrote:

...

The advantage of this swedish data centre is that even if its location is well known, it is pretty hard to harm the building. You can't run a truck full of explosives into it for instance.

Unfortunately, you also cannot run your own fiber there, colo equipment there, visit it for any reason, etc.

I was going to say 'this probably hinders customers adoption at NetNod', but I know for a fact the "probably" is superfluous.

I don't really get your reasoning here, Patrick. What were you going to do? Put your servers in the same racks as Netnod's switches? Rate their patch fiber management skills?

I can buy the argument that there is one bit of infrastructure (a string of dark fiber) more between your router and the IX infrastructure than you will get in other locations but in this age of people connecting remotely to IXPs all the time this seems pretty minor, especially given the box full of advantages it gives the IXP operator regarding facility security and having a very clear demarcation point.

I didn't say it would stop everyone. Of course some people will not be deterred, but some absolutely have. And most people are uninterested in the "box full of advantages it gives the IXP operator". Further, I would submit the "box full of advantages" are ephemeral at best, and arguably imaginary. Name another major IXP anywhere on the planet that has ever had a single issue NetNod's model would have avoided. Now compare that to forcing every single participant to use unknown fiber paths into an unknown facility. When are these fibers groomed, and onto which unknown paths? Which fiber maintenance schedules might impact me without my knowledge? Which construction projects elsewhere in the city might take me down and there's no way for me to even predict that? Etc., etc. I would prefer to take my chances with the known quantity, thankyouverymuch. Feel free to do with your network as you please. -- TTFN, patrick P.S. The demarcation point thing is pure BS and you know it.

On Dec 1, 2008, at 4:58 AM, Måns Nilsson wrote:

--On söndag, söndag 30 nov 2008 23.05.01 -0500 "Patrick W. Gilmore" <patrick@ianai.net> wrote:

In Sweden, the reason to not choose NetNod (and to go with the smaller exchangepoints) is price and only price. No swedish ISP I know of has stated that the fact that the Stokab fibre is bought by the IXP and not the ISP is a problem per se. Some might have a better wholesale deal than NetNod has but that is still just about price.

I don't think any IXP can become a significant player on the Internet today by only attracting participants from the country in question. The Internet is not bound by political borders. (Usually. :)

...

Now compare that to forcing every single participant to use unknown fiber paths into an unknown facility. When are these fibers groomed, and onto which unknown paths? Which fiber maintenance schedules might impact me without my knowledge? Which construction projects elsewhere in the city might take me down and there's no way for me to even predict that? Etc., etc.

The fiber paths into these facilities are national security issues. Expect them to be guarded accordingly (as in running them in specially blasted tunnels 30-60 meters down in the ground for the last aggregated path to the facility). I have not experienced more unpredictability nor more outages because Netnod buys the cable than when the ISP does. Same cable. And Stokab does indeed know where the cables are.

I'm glad to hear the fibers seem to be stable. Past performance is no guarantee of future profits and all that, but it is good to know care has been taken in the past. As for the blasting of tunnels and national security angle, this is an IXP, not nuclear missile launch control. It should not be your only vector to get bits from point A to B. And if it is, then you have a larger problem than worrying about the facility withstanding physical attack. And no, attaching to multiple NetNod nodes is not a solution, since only Stockholm has a large number of participants. End of day, an IXP is not some magical thing. It is an ethernet switch allowing multiple networks to exchange traffic more easily than direct interconnection - and that is all it should be. It should not be mission critical. Treating it as such raises the cost, and therefore barrier to entry, which lowers its value. -- TTFN, patrick

On Dec 1, 2008, at 9:12 AM, Randy Bush wrote:

...

I don't think any IXP can become a significant player on the Internet today by only attracting participants from the country in question.

netnod is very successful. i guess they must operate from more than sweden, then, eh?

NetNod is successful. Very is a matter of opinion. As for "operate from more than sweden", that is trivial to confirm by looking at their member list. So now that we have agreed, did you have a point, or just want to run up your word count?

engineers judge by results, not word count.

Wow, Randy, we are in agreement again. To be clear, are you suggesting IXPs consider hiding their switches, forcing you to use a single fiber providers, not allowing anyone to know the path, etc.? I want to be sure I understand what you mean, since "engineers" like to make serious points, not flippant sound bites. -- TTFN, patrick

On Dec 1, 2008, at 9:30 AM, Randy Bush wrote:

some go to sweden for the weather. some go for netnode. netnode does not go to them. and yes, netnod is bunkered up the ying yang. qed.

By your logic, every IXP which has any participants is a good model and cannot be improved. An obvious logical fallacy. One could assume this means you have no clue what you are talking about, but I will give you the benefit of the doubt. IOW: You are only interested in your word count. QED. -- TTFN, patrick

--On måndag, måndag 1 dec 2008 09.08.09 -0500 "Patrick W. Gilmore" <patrick@ianai.net> wrote:

I don't think any IXP can become a significant player on the Internet today by only attracting participants from the country in question. The Internet is not bound by political borders. (Usually. :)

There is a significant amount of traffic being exchanged between swedish operators on Netnod. It might be the case that the broadband penetration in Sweden justifies the establishment of local IXPen. This is however irrelevant to the discussion at hand -- or did you think about some kind of issue with connectivity from Stockholm and abroad? At least 3-4 providers sell connectivity into Stockholm on own fiber paths. Is Netnod useless to you because you are not one of them?

As for the blasting of tunnels and national security angle, this is an IXP, not nuclear missile launch control. It should not be your only vector to get bits from point A to B. And if it is, then you have a larger problem than worrying about the facility withstanding physical attack.

It is an optimisation, a very well engineered one.

And no, attaching to multiple NetNod nodes is not a solution, since only Stockholm has a large number of participants.

Probably true for international clients. Less so for Swedish ISPen.

End of day, an IXP is not some magical thing. It is an ethernet switch allowing multiple networks to exchange traffic more easily than direct interconnection - and that is all it should be. It should not be mission critical. Treating it as such raises the cost, and therefore barrier to entry, which lowers its value.

You did not answer my question on usability of fiber based on amount of knowledge about where it is. -- Måns Nilsson M A C H I N A There's a little picture of ED MCMAHON doing BAD THINGS to JOAN RIVERS in a $200,000 MALIBU BEACH HOUSE!!

--On måndag, måndag 1 dec 2008 11.53.58 -0500 "Patrick W. Gilmore" <patrick@ianai.net> wrote:

On Dec 1, 2008, at 11:06 AM, Måns Nilsson wrote:

...

...

End of day, an IXP is not some magical thing. It is an ethernet switch allowing multiple networks to exchange traffic more easily than direct interconnection - and that is all it should be. It should not be mission critical. Treating it as such raises the cost, and therefore barrier to entry, which lowers its value.

Yes. I do not disagree. The alternates that popped up and made Netnod switch to Ethernet from SRP were most welcome. SRR mode on that ring was not funny, btw.

Of course knowing where the fiber is does not stop the backhoes. It was obvious you were being silly, so I ignored it.

Ok. Indeed.

By that logic, providers should not check any fiber path they purchase because it will not stop the backhoes. I suspect most providers will continue to buy from multiple providers, check the paths themselves, ensure grooming onto a single path is not a problem, and several other perfectly valid operational best practices which are impossible at NetNod.

Netnod with the help of Stokab can guarantee that the two paths to switches A and B are diverse. It is a normal requirement one can make (at a cost, but that is to be expected) when sourcing Stokab fibre. They know where their stuff is and understand the importance of getting it properly separated. Other providers in Sweden are similar. I have no reason not to trust them, having seen the inside of several large calls for tender on dispersed path plants, with fibre paths well documented. That the path of the last mile to the cave is hidden and secret is a very small problem.

OTOH: My paragraph above yours is a serious consideration, which you have blithely ignored.

Not so anymore, if I've understood correctly. Drop this dead horse? -- Måns Nilsson M A C H I N A Hello, GORRY-O!! I'm a GENIUS from HARVARD!!

On Dec 1, 2008, at 2:05 PM, Jean-François Mezei wrote:

Patrick W. Gilmore wrote:

...

End of day, an IXP is not some magical thing. It is an ethernet switch allowing multiple networks to exchange traffic more easily than direct interconnection - and that is all it should be. It should not be mission critical. Treating it as such raises the cost, and therefore barrier to entry, which lowers its value.

Exchange points are often located in the same building as a carrier hotel which houses infrastructure for many ISPs and many transit providers.

If you consider the internet is used only by teenage males to learn about female anatomy (pictures and movies), then your statement is acceptable. But with the Internet now used for serious applications, the focus point of a carrier hotel and exchange becomes much more mission critical.

Ane because it is a focus point, it becomes much harder to have redundancy in the buildings (to provide for disaster tolerance). So the natural avenue is to strenghten/re-inforce your one central building.

It is not. The Internet can be mission critical. (Well, not really, but it's trying.) And for something mission critical, a single point, no matter how well reinforced, is not good enough. The exchange point should _NOT_ be mission critical. As I explained multiple times in the thread, if that is your only vector, your design is broken. Period. Care to argue otherwise? And if the IXP is not your only vector, if your redundancy is greater than any single building however deeply it is buried, then that IXP / building / vector is not mission critical. Treating it at such raises its price, which raises its barrier of entry, which lowers its utility. Unless you think only NORAD-approved networks should peer? -- TTFN, patrick

On Mon, Dec 01, 2008 at 08:14:20PM +0100, Kurt Erik Lindqvist wrote: [snip]

On 1 dec 2008, at 15.08, Patrick W. Gilmore wrote: [snip]

...

I don't think any IXP can become a significant player on the Internet today by only attracting participants from the country in question. The Internet is not bound by political borders. (Usually. :)

Despite the huge amount of "content which transcends the language barrier" [tip of the hat wbn], it is worth noting that there is a non-trivial amount of language-/culture-specific traffic that doesn't need or want to traverse globally (viz massive IXes & large xTTH deplyoments in otherwise 'small' countries). Sometimes that maps near to the political boundaries. Joe [by all means, do not take this as a SPoF endorsement] -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2 dec 2008, at 00.47, Randy Bush wrote:

...

Despite the huge amount of "content which transcends the language barrier" [tip of the hat wbn], it is worth noting that there is a non-trivial amount of language-/culture-specific traffic that doesn't need or want to traverse globally (viz massive IXes & large xTTH deplyoments in otherwise 'small' countries). Sometimes that maps near to the political boundaries.

<http://archive.psg.com/970210.nanog.pdf> slide 6

of course, these data are a bit long in the tooth

I most say I agree with Randy, already in 2001 I had a presentation (that Randy and those of you at RIPE in Dubai saw a copy of in EIX-WG) based on data from the KPNQwest network - where we saw that data had shifted from 80% US based to 80% national or regional. This was a clear change in traffic patterns all across Europe, at least from the data that I saw then. And keep in mind that this was before p2p skewed the data of user behavior. I have been arguing for the theory that 1. Dense exchange of traffic in Europe early on came as a result of a) Dereguation in the telco market b) Unwillingess to pay the "big US telcos" for exchange of local/ european traffic 2. The dense exchange of traffic made local services more viable and attractive 3. (2) Helped local(-language) services develop 4. (3) Helped the development of broadband adoption I do realize that the above is a huge simplification (And the slide set is much longer, and the paper will be even longer), but there are still lessons to be learnt in how the local language services and dense peering developed in Europe. Best regards, - - kurtis - -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkk1dBcACgkQAFdZ6xrc/t66+wCdFttkZsBxN7UuHlIS8x3jWFE1 3E8An2mfO++tc2BjO918KDf7yq0XVMJo =D078 -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Patrick, On 1 dec 2008, at 02.33, Patrick W. Gilmore wrote:

On Nov 28, 2008, at 4:04 PM, Jean-François Mezei wrote:

...

The thing about a carrier hotel is that it cannot be a secret location since you need to allow various carriers and ISPs to have physical access to the building so they can install/manage their servers/routers/switches.

The advantage of this swedish data centre is that even if its location is well known, it is pretty hard to harm the building. You can't run a truck full of explosives into it for instance.

Unfortunately, you also cannot run your own fiber there, colo equipment there, visit it for any reason, etc.

for the non-Stockholm locations that is not true. As a matter of fact, you will have to get your own fibers to Netnod there. As for co-lo of equipment, not as easy as in a neutral co-location. As for visits, why would you need to? As for fibers, Stockholm has a fiber monopoloy run by the city of Stockholm. So you would have to buy fibers from that monopoloy in any case.

I was going to say 'this probably hinders customers adoption at NetNod', but I know for a fact the "probably" is superfluous.

That is your judgement. We have seen the largest growth for a long time in the last year. Best regards, - - - kurtis - - -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkk0MeMACgkQAFdZ6xrc/t7REACfThTzW+3+mvA0ttvViTTVmMfv qgUAmwQyiuAaB/+vTD9wMtqCq7PDhw0F =ycFe - -----END PGP SIGNATURE----- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkk0PIgACgkQAFdZ6xrc/t727wCgvi0zOw4ivBe7AG98hb+DqoGI qicAn0WKn/yUoqYLln2yP7GuxM16NHzT =7Njx -----END PGP SIGNATURE-----

Måns Nilsson wrote:

These data centres are designed to Swedish military command center specifications (not like Cheyenne Mountain but significantly better than, say, a Minuteman site)

At one point some time ago, on NANOG we discussed putting exchanges in old minuteman silos. (so long ago a quick Google didn't find it -- where are all the old NANOG archives?)

pardon me for resurrecting this topic... For sites that are built in caves, how do they deal with cabling ? In the pretty pictures of the swedish site, there didn't seem to be an obvious raised floor. And it appeared to be solid concrete floor between the wings containing the systems. And no massive cable risers or suspended cable paths. It is all nice if neat, but in real life, wouldn't they need to be stringing tons and tons of cable ? Or it is a case of the pretty pictures with the fog having been taken with empty racks, before the data centre was outfitted with real equipment and now, there would be cables everywhere ?

--On onsdag, onsdag 3 dec 2008 18.29.54 +0100 Måns Nilsson <mansaxel@besserwisser.org> wrote:

...

In the pretty pictures of the swedish site, there didn't seem to be an obvious raised floor.

There is a raised floor, iirc.

There is a raised floor. Have a look at <http://www.sweclockers.com/imagebank/200809/Bahnhof_10stor001.jpg> -- Måns Nilsson M A C H I N A I brought my BOWLING BALL -- and some DRUGS!!