- Test - lists.nanog.org

cooling door
by Paul Vixie 31 Mar '08

31 Mar '08

page 10 and 11 of <http://www.panduit.com/products/brochures/105309.pdf> says there's a way to move 20kW of heat away from a rack if your normal CRAC is moving 10kW (it depends on that basic air flow), permitting six blade servers in a rack. panduit licensed this tech from IBM a couple of years ago. i am intrigued by the possible drop in total energy cost per delivered kW, though in practice most datacenters can't get enough utility and backup power to run at this density. if "cooling doors" were to take off, we'd see data centers partitioned off and converted to cubicles. -- Paul Vixie

10 11

Re: cooling door
by Frank Coluccio 31 Mar '08

31 Mar '08

re: "Are you aware of any such cases?" Before I run too far afield of my main point, it's the LAN closets right now that are superficial and require the most attention and overhauling. Also on tap are local in-building data centers, remotes, colos, etc. And not necessarily in that order. >>every other piece of hardware that I have been responsible for in even a 35-person, simple LAN setup ... has had to have me lay hands on it at odd, unplanned times to keep working.<< You can't lay your hands on equipment that's no longer in existence because it's been eliminated, for starters. This is probably the most important point to grasp. There is no LAN equipment on site anymore once you've backhauled your desktops directly to a central site over optical channels. And to do this, fiber to the desktop (and a generous smattering of WLANs never hurts) must be considered first, no longer brought back to the local closet, but rather to one or two central locations that could either be in-building or to a nearby data center or colo. Or as far away as applications will permit. That last sentence should serve as evidence that I've learned my lesson ;) Some enterprises (including several large federal government agencies) have done parts of what I'm referring to, but none that I am aware of has done it all. By the same token, nothing that I've mentioned has NOT been tried somewhere by some enterprise before, which is to say that every aspect of fiber extension that I've mentioned thus far is currently working somewhere in an enterprise or shared data center today. In the large enterprise space we've already seen, time and again now, how leased optical facilities and dark fiber networks alike support server farms in remote data centers and public colos bringing data back to desktops in business offices. VPNs also come to mind. Think: FiOS. Why would supporting the desktops of a 1,000 or 10,000 employee company, which desktops would be scattered over a half-dozen or a hundred urban and exurban sites, be any different than supporting the same number of residential subscribers who are being supported by fiber to the home? Indeed Verizon has already deployed the equivalent of FiOS to some federal government agencies, while holding back in the enterprise. I suppose they are holding back in the enterprise for fear of cannibalizing its own lucrative leased line business, although there could be other reasons having to do with the awkward and time consuming process they undergo while formally "productizing" their services. Multiple cable system pperator Bright House Networks is using passive-optical networks (PONs) today akin to Verizon's FiOS, sans the video part, to do this very thing in support of businesses in central Florida as I type. http://www.allbusiness.com/services/business-services/4516302-1.html The method of supporting end users being used by Bright House Networks is the same as I outlined earlier in respect to the LAN admin operating from Mumbai. In general, the LAN part, or more to the point, fiber to the desktop is a relative slam dunk compared to the intricacies of tuning applications within data centers, colos and IXes, I grant you that. As they say, however, you've gotta start somewhere. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sat Mar 29 21:14 , Patrick Giagnocavo sent: > >Frank Coluccio wrote: >> > >I'd like to see a case where someone actually did what you proposed and >were successful at it, in a for-profit company. > >Are you aware of any such cases? > >Because outside of Solaris and Linux servers, every other piece of >hardware that I have been responsible for in even a 35-person, simple >LAN setup (the smallest and simplest config anyone on this list could >possibly have), has had to have me lay hands on it at odd, unplanned >times to keep working. > >Cordially > >Patrick Giagnocavo >patrick(a)zill.net

3 2

Re: rack power question
by MARLON BORBA 31 Mar '08

31 Mar '08

Do not forget physical security (including, but not limited to, access control & surveillance -- different logs, videos, and people to control), local/municipal/state laws and regulations (e.g. fire control standards), personnel to manage all that sites (even third-party)... IMHO too much administrative burden. :-( Abraços, Marlon Borba, CISSP, APC DataCenter Associate Técnico Judiciário - Segurança da Informação TRF 3 Região (11) 3012-1683 -- Practically no IT system is risk free. (NIST Special Publication 800-30) -- >>> "vijay gill" <vgill(a)vijaygill.com> 31/03/08 2:27 >>> [...] On Sun, Mar 23, 2008 at 2:15 PM, <michael.dillon(a)bt.com> wrote: > > Given that power and HVAC are such key issues in building > big datacenters, and that fiber to the office is now a reality > virtually everywhere, one wonders why someone doesn't start > building out distributed data centers. Essentially, you put > mini data centers in every office building, possibly by > outsourcing the enterprise data centers. Then, you have a > more tractable power and HVAC problem. You still need to > scale things but it since each data center is roughly comparable > in size it is a lot easier than trying to build out one > big data center. > Latency matters. Also, multiple small data centers will be more expensive than a few big ones, especially if you are planning on average load vs peak load heat rejection models. [...]

1 0

Re: cooling door
by paul＠vix.com 31 Mar '08

31 Mar '08

> I have a need for a 1U that will just act as a backup (higher MX) mailserver > and, occasionally, deliver some large .iso images at under 10Mbit/Sec .... > :) And I'm sure that there are other technically saavy users just like me > that could help you out with this "surplus" space! :) see http://www.vix.com/personalcolo/ for some places to host that backup MX. (note, i have no business affiliation with any of the entities listed there.)

1 0

Re: NXDOMAIN data needed for survey
by Scott Weeks 31 Mar '08

31 Mar '08

---------------------------------- [...]the flaming, I'm sure a bunch of netops wrote back to the guy offering to sell NXDOMAIN data. ---------------------------------- Why would you assume this? That wouldn't be my first assumption after reading the thread. I would assume folks would Do The Right Thing. scott

2 1

Re: cooling door
by Brandon Butterworth 30 Mar '08

30 Mar '08

> I can lease 10 racks, > put T1600s in two of them, and leave the other 8 empty; but > that hasn't helped either me the customer or the exchange > point provider; they've had to burn more real estate for empty > racks that can never be filled Seems fine to me, you used your power in two racks, to get more power will cost the provider more so you will have to pay more. This is just an extreme example of blade servers causing half a rack to be empty the facts haven't changed no matter how bad seeing empty racks feels Waste implies you could use it for no additional cost, old pricing models were vulnerable to gaming on combinations they'd not thought through. It might be easier for people to understand in these cases if the provider put yellow/black stripe tape over the unused space with a big sign saying "not yours" > I'm paying for floor space in my > cage that I'm probably going to end up using for storage rather > than just have it go to waste That's nice, I hate cages where you have no room for tools or to work and the kit hits the walls when you try and unrack it. I've no idea how they fit a normal engineer in some > and we still have the problem of > two very hot spots that need relatively 'point' cooling solutions. Accepted, big fan on the back of the rack? Plenty of empty space for such solutions. High density servers seem to be vendor driven, they can charge more and make you buy the switch and other ancillaries you'd likely choose cheaper from others. And when new models come out the whole lot gets replaced rather than just the odd few U of servers. The convenience may be worth the high price for some situations Density is just another DC design parameter to be optimised for profit brandon -- You know a nanog thread has gone on too long when I overcome inertia and post. More science please.

1 0

Re: latency (was: RE: cooling door)
by Frank Coluccio 30 Mar '08

30 Mar '08

Silly me. I didn't mean "turns" alone, but also intended to include the number of state "transitions" (e-o, o-e, e-e, etc.) in my preceding reply, as well. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sun Mar 30 16:47 , Frank Coluccio sent: >Mikael, I see your points more clearly now in respect to the number of turns >affecting latency. In analyzing this further, however, it becomes apparent that >the collapsed backbone regimen may, in many scenarios offer far fewer >opportunities for turns, and more occasions for others. > >To the former class of winning applications, because it eliminates local >access/distribution/aggregation switches and then an entire lineage of >hierarchical in-building routing elements. > >To the latter class of loser applications, no doubt, if a collapsed backbone >design were to be dropped-shipped in place on a Friday Evening, as is, the there >would surely be some losers that would require re-designing, or maybe simply some >re-tuning, or they may need to be treated as one-offs entirely. > >BTW, in case there is any confusion concerning my earlier allusion to "SMB", it >had nothing to do with the size of message blocks, protocols, or anything else >affecting a transaction profile's latency numbers. Instead, I was referring to >the "_s_mall-to-_m_edium-sized _b_usiness" class of customers that the cable >operator Bright House Networks was targeting with its passive optical network >business-grade offering, fwiw. >-- > >Mikael, All, I truly appreciate the comments and criticisms you've offered on >this subject up until now in connection with the upstream hypothesis that began >with a post by Michael Dillon. However, I shall not impose this topic on the >larger audience any further. I would, however, welcome a continuation _offlist _ >with anyone so inclined. If anything worthwhile results I'd be pleased to post it >here at a later date. TIA. > >Frank A. Coluccio >DTI Consulting Inc. >212-587-8150 Office >347-526-6788 Mobile > >On Sun Mar 30 3:17 , Mikael Abrahamsson sent: > >>On Sat, 29 Mar 2008, Frank Coluccio wrote: >> >>> Understandably, some applications fall into a class that requires very-short >>> distances for the reasons you cite, although I'm still not comfortable with the >>> setup you've outlined. Why, for example, are you showing two Ethernet switches >>> for the fiber option (which would naturally double the switch-induced latency), >>> but only a single switch for the UTP option? >> >>Yes, I am showing a case where you have switches in each rack so each rack >>is uplinked with a fiber to a central aggregation switch, as opposed to >>having a lot of UTP from the rack directly into the aggregation switch. >> >>> Now, I'm comfortable in ceding this point. I should have made allowances for this >>> type of exception in my introductory post, but didn't, as I also omitted mention >>> of other considerations for the sake of brevity. For what it's worth, propagation >>> over copper is faster propagation over fiber, as copper has a higher nominal >>> velocity of propagation (NVP) rating than does fiber, but not significantly >>> greater to cause the difference you've cited. >> >>The 2/3 speed of light in fiber as opposed to propagation speed in copper >>was not in my mind. >> >>> As an aside, the manner in which o-e-o and e-o-e conversions take place when >>> transitioning from electronic to optical states, and back, affects latency >>> differently across differing link assembly approaches used. In cases where 10Gbps >> >>My opinion is that the major factors of added end-to-end latency in my >>example is that the packet has to be serialisted three times as opposed to >>once and there are three lookups instead of one. Lookups take time, >>putting the packet on the wire take time. >> >>Back in the 10 megabit/s days, there were switches that did cut-through, >>ie if the output port was not being used the instant the packet came in, >>it could start to send out the packet on the outgoing port before it was >>completely taken in on the incoming port (when the header was received, >>the forwarding decision was taken and the equipment would start to send >>the packet out before it was completely received from the input port). >> >>> By chance, is the "deserialization" you cited earlier, perhaps related to this >>> inverse muxing process? If so, then that would explain the disconnect, and if it >>> is so, then one shouldn't despair, because there is a direct path to avoiding >this. >> >>No, it's the store-and-forward architecture used in all modern equipment >>(that I know of). A packet has to be completely taken in over the wire >>into a buffer, a lookup has to be done as to where this packet should be >>put out, it needs to be sent over a bus or fabric, and then it has to be >>clocked out on the outgoing port from another buffer. This adds latency in >>each switch hop on the way. >> >>As Adrian Chadd mentioned in the email sent after yours, this can of >>course be handled by modifying or creating new protocols that handle this >>fact. It's just that with what is available today, this is a problem. Each >>directory listing or file access takes a bit longer over NFS with added >>latency, and this reduces performance in current protocols. >> >>Programmers who do client/server applications are starting to notice this >>and I know of companies that put latency-inducing applications in the >>development servers so that the programmer is exposed to the same >>conditions in the development environment as in the real world. This means >>for some that they have to write more advanced SQL queries to get >>everything done in a single query instead of asking multiple and changing >>the queries depending on what the first query result was. >> >>Also, protocols such as SMB and NFS that use message blocks over TCP have >>to be abandonded and replaced with real streaming protocols and large >>window sizes. Xmodem wasn't a good idea back then, it's not a good idea >>now (even though the blocks now are larger than the 128 bytes of 20-30 >>years ago). >> >>-- >>Mikael Abrahamsson email: swmike(a)swm.pp.se > >

1 0

Re: latency (was: RE: cooling door)
by Frank Coluccio 30 Mar '08

30 Mar '08

Mikael, I see your points more clearly now in respect to the number of turns affecting latency. In analyzing this further, however, it becomes apparent that the collapsed backbone regimen may, in many scenarios offer far fewer opportunities for turns, and more occasions for others. To the former class of winning applications, because it eliminates local access/distribution/aggregation switches and then an entire lineage of hierarchical in-building routing elements. To the latter class of loser applications, no doubt, if a collapsed backbone design were to be dropped-shipped in place on a Friday Evening, as is, the there would surely be some losers that would require re-designing, or maybe simply some re-tuning, or they may need to be treated as one-offs entirely. BTW, in case there is any confusion concerning my earlier allusion to "SMB", it had nothing to do with the size of message blocks, protocols, or anything else affecting a transaction profile's latency numbers. Instead, I was referring to the "_s_mall-to-_m_edium-sized _b_usiness" class of customers that the cable operator Bright House Networks was targeting with its passive optical network business-grade offering, fwiw. -- Mikael, All, I truly appreciate the comments and criticisms you've offered on this subject up until now in connection with the upstream hypothesis that began with a post by Michael Dillon. However, I shall not impose this topic on the larger audience any further. I would, however, welcome a continuation _offlist _ with anyone so inclined. If anything worthwhile results I'd be pleased to post it here at a later date. TIA. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sun Mar 30 3:17 , Mikael Abrahamsson sent: >On Sat, 29 Mar 2008, Frank Coluccio wrote: > >> Understandably, some applications fall into a class that requires very-short >> distances for the reasons you cite, although I'm still not comfortable with the >> setup you've outlined. Why, for example, are you showing two Ethernet switches >> for the fiber option (which would naturally double the switch-induced latency), >> but only a single switch for the UTP option? > >Yes, I am showing a case where you have switches in each rack so each rack >is uplinked with a fiber to a central aggregation switch, as opposed to >having a lot of UTP from the rack directly into the aggregation switch. > >> Now, I'm comfortable in ceding this point. I should have made allowances for this >> type of exception in my introductory post, but didn't, as I also omitted mention >> of other considerations for the sake of brevity. For what it's worth, propagation >> over copper is faster propagation over fiber, as copper has a higher nominal >> velocity of propagation (NVP) rating than does fiber, but not significantly >> greater to cause the difference you've cited. > >The 2/3 speed of light in fiber as opposed to propagation speed in copper >was not in my mind. > >> As an aside, the manner in which o-e-o and e-o-e conversions take place when >> transitioning from electronic to optical states, and back, affects latency >> differently across differing link assembly approaches used. In cases where 10Gbps > >My opinion is that the major factors of added end-to-end latency in my >example is that the packet has to be serialisted three times as opposed to >once and there are three lookups instead of one. Lookups take time, >putting the packet on the wire take time. > >Back in the 10 megabit/s days, there were switches that did cut-through, >ie if the output port was not being used the instant the packet came in, >it could start to send out the packet on the outgoing port before it was >completely taken in on the incoming port (when the header was received, >the forwarding decision was taken and the equipment would start to send >the packet out before it was completely received from the input port). > >> By chance, is the "deserialization" you cited earlier, perhaps related to this >> inverse muxing process? If so, then that would explain the disconnect, and if it >> is so, then one shouldn't despair, because there is a direct path to avoiding this. > >No, it's the store-and-forward architecture used in all modern equipment >(that I know of). A packet has to be completely taken in over the wire >into a buffer, a lookup has to be done as to where this packet should be >put out, it needs to be sent over a bus or fabric, and then it has to be >clocked out on the outgoing port from another buffer. This adds latency in >each switch hop on the way. > >As Adrian Chadd mentioned in the email sent after yours, this can of >course be handled by modifying or creating new protocols that handle this >fact. It's just that with what is available today, this is a problem. Each >directory listing or file access takes a bit longer over NFS with added >latency, and this reduces performance in current protocols. > >Programmers who do client/server applications are starting to notice this >and I know of companies that put latency-inducing applications in the >development servers so that the programmer is exposed to the same >conditions in the development environment as in the real world. This means >for some that they have to write more advanced SQL queries to get >everything done in a single query instead of asking multiple and changing >the queries depending on what the first query result was. > >Also, protocols such as SMB and NFS that use message blocks over TCP have >to be abandonded and replaced with real streaming protocols and large >window sizes. Xmodem wasn't a good idea back then, it's not a good idea >now (even though the blocks now are larger than the 128 bytes of 20-30 >years ago). > >-- >Mikael Abrahamsson email: swmike(a)swm.pp.se

1 0

Re: latency (was: RE: cooling door)
by Frank Coluccio 30 Mar '08

30 Mar '08

Please clarify. To which network element are you referring in connection with extended lookup times? Is it the collapsed optical backbone switch, or the upstream L3 element, or perhaps both? Certainly, some applications will demand far less latency than others. Gamers and some financial (program) traders, for instance, will not tolerate delays caused by access provisions that are extended over vast WAN, or even large Metro, distances. But in a local/intramural setting, where optical paths amount to no more than a klick or so, the impact is almost negligible, even to the class of users mentioned above. Worst case, run the enterprise over the optical model and treat those latency-sensitive users as the one-offs that they actually are by tying them into colos that are closer to their targets. That's what a growing number of financial firms from around the country have done in NY and CHI colos, in any case. As for cost, while individual ports may be significantly more expensive in one scenario than another, the architectural decision is seldom based on a single element cost. It's the TCO of all architectural considerations that must be taken into account. Going back to my original multi-story building example-- better yet, let's use one of the forty-story structures now being erected at Ground Zero as a case in point: When all is said and done it will have created a minimum of two internal data centers (main/backup/load-sharing) and a minimum of eighty (80) LAN enclosures, with each room consisting of two L2 access switches (where each of the latter possesses multiple 10Gbps uplinks, anyway), UPS/HVAC/Raised flooring, firestopping, sprinklers, and a commitment to consume power for twenty years in order to keep all this junk purring. I think you see my point. So even where cost may appear to be the issue when viewing cost comparisons of discreet elements, in most cases that qualify for this type of design, i.e. where an organization reaches critical mass beyond so many users, I submit that it really is not an issue. In fact, a pervasively-lighted environment may actually cost far less. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sat Mar 29 19:20 , Mikael Abrahamsson sent: > >On Sat, 29 Mar 2008, Frank Coluccio wrote: > >> We often discuss the empowerment afforded by optical technology, but we've barely >> scratched the surface of its ability to effect meaningful architectural changes. > >If you talk to the server people, they have an issue with this: > >Latency. > >I've talked to people who have collapsed layers in their LAN because they >can see performance degradation for each additional switch packets have to >pass in their NFS-mount. Yes, higher speeds means lower serialisation >delay, but there is still a lookup time involved and 10GE is >substantionally more expensive than GE. > >-- >Mikael Abrahamsson email: swmike(a)swm.pp.se

6 6

Re: latency (was: RE: cooling door)
by Frank Coluccio 30 Mar '08

30 Mar '08

Understandably, some applications fall into a class that requires very-short distances for the reasons you cite, although I'm still not comfortable with the setup you've outlined. Why, for example, are you showing two Ethernet switches for the fiber option (which would naturally double the switch-induced latency), but only a single switch for the UTP option? Now, I'm comfortable in ceding this point. I should have made allowances for this type of exception in my introductory post, but didn't, as I also omitted mention of other considerations for the sake of brevity. For what it's worth, propagation over copper is faster propagation over fiber, as copper has a higher nominal velocity of propagation (NVP) rating than does fiber, but not significantly greater to cause the difference you've cited. As an aside, the manner in which o-e-o and e-o-e conversions take place when transitioning from electronic to optical states, and back, affects latency differently across differing link assembly approaches used. In cases where 10Gbps or greater is being sent across a "multi-mode" fiber link in a data center or other in-building venue, for instance, "parallel optics" are most ofen used, i.e., multiple optical channels (either fibers or wavelengths) that undergo multiplexing and de-multiplexing (collectively: inverse multiplexing or channel bonding) -- as opposed to a single fiber (or a single wavelength) operating at the link's rated wire speed. By chance, is the "deserialization" you cited earlier, perhaps related to this inverse muxing process? If so, then that would explain the disconnect, and if it is so, then one shouldn't despair, because there is a direct path to avoiding this. In parallel optics, e-o processing and o-e processing is intensive at both ends of the 10G link, respectively. These have the effect of adding more latency than a single-channel approach would. Yet, most of the TIA activity taking place today that is geared to increasing data rates over in-building fiber links continues to favor multi-mode and the use of parallel optics, as opposed to specifying single-mode supporting a single channel. But singlemode solutions are also available to those who dare to be different. I'll look more closely at these issues and your original exception during the coming week, since they represent an important aspect in assessing the overall model. Thanks. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Sat Mar 29 20:30 , Mikael Abrahamsson sent: > >On Sat, 29 Mar 2008, Frank Coluccio wrote: > >> Please clarify. To which network element are you referring in connection with >> extended lookup times? Is it the collapsed optical backbone switch, or the >> upstream L3 element, or perhaps both? > >I am talking about the matter that the following topology: > >server - 5 meter UTP - switch - 20 meter fiber - switch - 20 meter >fiber - switch - 5 meter UTP - server > >has worse NFS performance than: > >server - 25 meter UTP - switch - 25 meter UTP - server > >Imagine bringing this into metro with 1-2ms delay instead of 0.1-0.5ms. > >This is one of the issues that the server/storage people have to deal >with. > >-- >Mikael Abrahamsson email: swmike(a)swm.pp.se

4 5