2006.06.06 NANOG-NOTES IDC power and cooling panel
From high-end router space--it's been simple, the bandwidth demand has grown faster than the power efficiency can keep up with. In the past, had
(ok, one more set of notes and then off to sit in traffic for an hour on the way to work... --Matt) 2006.06.06 Power and Cooling panel Dan Golding, Tier1 research, moderator Hot Time in the Big IDC Cooling, Power, and the Data Center 3 IDC vendors, 4 hardware vendors Michael Laudon, force10 Jay Park, equinix Rob Snevely, Sun Josh Snowhorn, terremark David Tsiang, cisco Brad Turner, juniper Brian Young, S&D The power and cooling crisis internet datacenters are getting full most of the slack capacity has been used up devices are using more and more power low power density - routers, full sized servers medium power density - 1u servers, switches high power density - blade servers Many data centers are full at 70-80% floor space utilized North America IDC occupancy is around 50% most sought-after space is around 70% full when power and cooling capacity is used up, floor space is vacant but can't be used. There is a relationship between power and cooling devices are not 100% efficient I^2R losses means that power becomes heat (conservation of energy) heat must be dissipated The ability to dissipate heat with normal cooling technologies is hitting the wall need new techniques Some quick rules of thumb a rack or cabinet is a standard unit of space from 30-40sqft per rack power is measured in watts many facilities do around 80-100w/sqft; at 30sqft per rack, that's about 3kw/rack high how did we get here? what is current situation where are we going? [dang, he's flying through his slides!!] Hardware engineers T-series hardware engineer for Juniper CRS-1 hardware E-series datacenter design issues for Sun, there were other hardware vendors who were not interested in showing up, these people were brave for coming up here! Josh snowhorn, IDC planner Jay Park, electrial engineer for equinix Brian Young, S&D cage design specialist What do the IDC vendors feel the current situation is in terms of power/cooling, how did we get here? Josh--designed datacenters at 100w/sq/ft, more than enough for the carriers; the server guys hit 100w/sqft in a quarter rack. you could cannabalize some power and cooling, but still ran out of cooling. Now spend hundreds of millions to make 200wsqft datacenters, or higher. Now, to hardware vendors--why are their boxes using up so much electricity, putting out so much heat? What are economics behind increasing density and heat load? the ability to improve keep up, do power spins about every 2 years, half power; but now bandwidth is doubling every year, but takes two years to drop power in half. We've been loosing at this game for a while, and running out of room on the voltage scale; 90nm is down at 1v, can't go much lower, since diode drop is at 0.7v; at 65nm, it's still at 1v, there's no big hammer anymore for power efficiency. Need to pull some tricks out, but may need to do clock gating, may get some 20-30% efficiency gains, but not much more that can be pulled out of the bag now. Newton was right; you can do some tricks, but no magic. Chip multithreading is one area they're trying to squeeze more performance out of; don't replicate ancillary ASICs for each core. Also can more easily share memory, and nobody has a 100% efficient power supply, so you lose some power there too. More and more getting squeezed in each rack. Also a drive on cost; amortizing costs over space and capability. reducing costs per port is a big driver. And customers are pushing for more and more density, since the cost of real-estate is getting so high, since each square foot costs so high. In Ginza, $120/sq ft for space. If you go to places where realestate is cheap, easier/cheaper to just build really big rooms, and let power dissipate more naturally. IDC people agree, some cities are just crazy in real-estate costs. But for those in suburban areas, cost of real-estate isn't so expensive. 3kw per blade server, put a few in a rack, you hit nearly 10kw in a rack. Soon, will need direct chilled water in the rack to cool them. But chilled water mixed with other colocation and lower density cabinets is very challenging to build. But need to have enclosed space to handle local chilled water coolers in localized racks. 20 years ago at IBM, nobody wanted chilled water in their hardware. Now, we're running out of options. Disagree--other ways of handling the challenge; how thermally efficient are the rooms in the first place, and are there other ways of handling heat issues? Cables with a cutout in tiles allows air to escape in areas that don't provide cooling. Josh notes the diversity between carriers at 40w/sq/ft vs hosting providers at 400w/sq/ft is making engineering decisions challenging. It's not about power really anymore, we can get power, it's about cooling now. Dealing with space in wrong terms--watts/sq ft, vs requirements of each rack. Charge customers based on the cooling requirements? If you try to cool 15kw per cabinet, you still have limits of how many cfm you can move through a given space. At some point, the air flow vertically through the rack starts to starve. What about a dual push-pull air system that pushes from the bottom and pulls from the top. Q: Randy, IIJ, question from the audience. He'll put as hot stuff in there as he can cool, because he wants the power, that's life. Problem is cooling; over 3kw/4kw over current level, the wind tunnel effect gets painful. the option of putting water in the cabinets is a dealbreaker for many people. Fact is, most facilities can't even handle 3kw per square meter; any build that can't meet that today is unrealistic. That's 300+ w per sq. ft. Josh has some cabinets at NOTA; Akamai is at 386w/sq/ft, they can cool it with huge hot aisle behind it, and around carriers at 40w/sq/ft. Those are the densest cabinets they have. IDCs need to build them and charge a realistic amount; people will burn as hot as possible, since they need to move more and more data. Raise plenums higher, move more air, air coming up side of rack and across. Currently, equinix is building 4kw per cabinet, planning for that in 2007. A cabinet is about 2sq meters, so still not at the density Randy's looking for. Starting to separate high density users from medium and low density users. Q: GNI, Derek ? datacenter in SF, 1008 IBM blade servers, 2500 sq ft, ping pong table, soda machine in surrounding areas. need 2500w/sq/ft to deliver the same cabinet space. 20kw per cabinet is what they can deliver in cabinets. He's paying for 100 cabinets, have 12 installed. he's still netting efficiency for it. still gets 3% better efficiency, still beats 84 1u pizza box servers. If IDCs could keep up with that, could keep physical space requirements more reasonable. The costs are exponential for more density. Up to a year lead time for 2MW generators, we're pushing the envelope on that. It is an exponential increase. Budget trauma when those costs get passed on. Let the demand stimulate ingenuity. The internet industry in general is short sighted. 22 million blade servers installed, where they will be located. Q: BillNorton. one other dimension. Life span for new datacenters is 10-40 year timeframe, so it's hard to adjust midlife to hugely different power and cooling demands.
From a technology point of view, CMOS was last great quantum leap, need another great quantum leap before we get relief on the cooling footprint. Randy is right, the cooling architecture isn't optimal. CRS1, 20% of power goes to fans to move air past convoluted air paths. Spreading out the equipment is a mitigation. multi-chassis systems will help with that. Sun, Juniper, do you see power continuing to grow linearly, or flatten out? As they go to 40gig or 100gig, the power and heat will continue to grow; more gates, more heat, more power. We'll hit a wall soon.
Cisco and Juniper agree, it's 6/6/06, take note, world! 20 year shelf life for datacenter, look at where they were 20 years ago. 10w/sq/foot back in 1986. We've greatly increased the amount of work that can be done since that time. Will machines continue to do the same amount of work, or will we flatten out on the machine capability curve? Element of geographic progression as you double. Nobody will ever need more than 20kw per rack! (Dan Golding) Running into some roadblocks; 100M gate ASICs, packing so much power into a single chip, may not be linear since can't move that much heat out of the chip from the point source into the system. Q: Patrick from Akamai--mcmurta base, south pole? As a brighter light, spot of hope, hottest colo in Terremark, finding they don't need more power, coming to less power. Running into getting enough spindles, the processors are getting faster and drawing less power. 40 amps per rack, used to be non-full, now able to fill them more completely. not sure if everyone is seeing this, but their power consumption is going down. Not all doom and gloom, but for next 12 months, at least somewhat lucky. Chip multicores is a good leap that can help for a bit; like Sun's Niagra multicore chips or chip multithreading, only about 50% of power is used by real processing power, rest is ancillary power. Q: Rob Seastrom; BillN danced around the question; seen it happen before. MAE-EAST, mark 3, additional liebert challengers tucked in....if one builds a datacenter to 4kw/meter^2, how long will that be premium space vs no-longer-up-to-par. Does 20 year colo life even make sense? Is the run rate steep enough that the number is just one we're fooling ourselves over? Josh: none of them are running at this density. it's the server density that hurts; carriers aren't as much of the pain. Separation of infrastructure most likely; Voice, carriers, etc. there, and separate datacenters where server floors exist. 20 years from now--will it be obsolete? Yes, probably. they'll keep doing what they can to help service their customers. Q: Joel K, from ? -- what's coming in network equipment to help cut power? throttle back linecards that aren't running at full bore? A: If you make it automatic, service providers would consider it; but from the bandwidth demand growth, the exponential growth--technology isn't keeping up, it's plateauing. Multithreaded ideas, turn off idle logic portions, incremental improvements, they're one-shot efforts, won't really help fix the slope of the curve. May just accept that we need more space, period. High speed/low speed fans, only kick up to high speed during thermal extremes. Again, both Cisco and Juniper have explored suppressing some gear, but customers still want 50ms protect gear response, so they can't really shut down. Even making heat sinks to move the heat is getting challenging! Force10 also talked to customers about it; in order to do turning off portions and then turn back on, incurs latency, buffer some packets, etc; people can be sensitive to jitter and latency. Pushback has been fairly large from other sources. Q: Rick Wesson--to colo vendors--when will heat/BTUs be a part of of the charges? And to server vendors, when will heat be a listed component upon which vendors make sales? A: IDCs don't charge based on heat load. Power as proxy for heat right now. Cooling overhead is wrapped into cost of sq ft and power; costs from utilities have been going up 30% due to oil prices going up. Might make it easier to add that charge for customers. Hardware vendors are certainly seeing power/heat limitations in RFPs. Building smaller systems with fewer slots to meet those RFPs. Customers asking for gbits/kw now from network gear. Sun notes that total cost of ownership, power may cost more to run the server than the cost of the server itself. Q: Lane Patterson, equinix. T640, if you redesigned it today, what fraction of power would it use today compared to past? Do they engineer gear to see how much they can pack into the same power/heat footprint? But customers are also asking for more and more capacity, less likely to pay for holding the line at same power/heat as previous generation. Cisco reiterates that we're running out of tricks; we can hold the line for a product generation, but after that, we're out of luck. We may need to shift architecture of pops going forward. Why not build in cheaper places, and backhaul? Q: Jared Mauch, NTT--huge customer demand; no vendors are proving interfaces greater than 40gig; not for next 3 years at least will there be faster links; backhauling from remote locations requires aggregating more and more traffic; if link speeds aren't increasing, backhaul isn't practical. As media companies continue deciding they can sell movies, music, and the like online, we may start hitting the wall; demand on all sides is growing, and we're running out of ways to address these challenges. Q: Avi Freedman. Talks to people doing lots of very dense disk solutions. Rackable solutions working on high density storage racks using laptop drives. 48 disks for you starts to generate a lot of heat; thumper product? 4u, 196 laptop disk rack unit? For people who need lots of spindles, lots of IOPS. A: can't talk about those products, they showed up in Jonathan's blog, but don't exist yet. There are always going to be limitations, the vendor will expect you can run the box in the location you're going to put it; that is, box has requirements, need to make sure customers are installing boxes in areas where the thermal issues are being considered. Q: phil, rosenthal, ISprime. people on the panel are pretty good, not the worst offenders. You need to hit the 1u server people where most power is being wasted; Dell 1650 vs Dell 1850; processor time, sitting at 90% idle on both system for bottom of line servers, do we need lower end CPUs on server lines so the CPUs won't be idle. A: why not use fewer machines, but have them do more work each? Virtualization might help us a bit in these areas, where we get more efficient use of the servers already in place. Equinix notes neutral current dropped a lot, people using 208V instead of 120V, generates less heat to the datacenter as well. Q: Randy; on left, crew singing I want my p2p. will always have max heat in the rack; servers and router vendors will keep working as hard as they can to do what they can. They had to leave 30% of their datacenter empty and build a bistro in it because they couldn't handle the heat budget. The IDC vendors [sorry, missed the comment] Q: Tim Elisio, new metric? To what extent is standardization, like using larger, more efficient ower supplies, or more efficient fans, cooling systems, etc. helping? A: IEEE meetings talking about some standardization, get some savings; the economies of scale helps make more efficient products on the standardized products. Telecom/router industry is working to old standards; may need to re-think what airflow standards are, for example. More dollars in a particular area helps push research and development in that direction. Juniper notes internals can be optimized, but the external plant and interfaces therein need better standardization to get economies of scale. Everyone's using multispeed fans, use them when you need them. 3 orgs, SPEC, ECL forum/EPA (energystar for servers and blade servers), and GreenGrid. Will see benchmarks coming out; will start asking all vendors to start compete on how much work they can do per how much power sucked and heat generated. Make the hardware vendors compete on how efficiently they use power and generate heat; we can then decide with dollars on who will win. Helps motivate people to optimize on the axis they care about. But are vendors talking to each other about how they can use standardized gear and standardized facilities designs more efficiently? ASHRAY?, heating/refrigeration group puts specs on how machines should be cooled (front to back, etc). But vendors don't want to help each other compete because that hurts their business. Dan Golding, 30 seconds, what would each person like to see the folks on other side do to help. Brian: asked vendors to have more efficient power supplies, more efficient systems that generate less heat for them to dissipate. Equinix--challenged by power density; customers don't understand, they want to put in smallest cage as possible. Need people to understand heat load better! Asks customers to use blanks in unused rack space to isolate cold and hot aisles. Too much leakage from cold aisles to hot aisles. Put blanks in!! Josh. Everyone building hot datacenters; would like to see vendors come into IDCs, test them in real world environments, put them in labs, see how they stand up to environment, test glycol taps, water taps, etc. Building servers is faster than building datacenters! Hardware vendor; PGE, worked with them to measure the increased efficiency, blanks DO help!! Education, amongst each other and customers. Watts per sq ft is crazy, do it on a per rack basis, makes it easier for customers to understand the limitations. Force10, if IDC groups got together, if there was a forum or group they could work with; right now, everyone has different requirements. Otherwise, always doing multiple tradeoffs, if there were a more general consensus, easier to engineer for. Cisco--good point, get IDCs and service providers to meet with vendors, come up with a next generation facility architecture to aim and build for. Hopefully make cooling and airflow easier, reduce the amount of power used by fans. Juniper--sees RFPs from customers, environment specs are very diverse; would be good to have common standards for customers to aim for; also, update some outdated nomeclature, use common terminology. Michael, Josh, Rob, Brian, thanks to all the panelists, Steve Feldman, we've slipped by 15 minutes, start at 2:15, everything will slip thereafter LUNCH!
Dan Golding, 30 seconds, what would each person like to see the folks on other side do to help.
Did anybody mention cogeneration? At least some of that waste heat is turned into electricity which reduces the total consumption off the grid. http://www.polarpowerinc.com/products/generators/cogenset.htm And what about risks? As you increase the active cooling of your IDC, you reduce the ability to survive power outages. This is another reason to separate hot customers from cool. In the event of a power outage, your cool customers who have better heat engineering do not have to share fate with the lazy hot customers. Here I am referring to server customers who do have choices to use more efficient hardware, improve the power efficiency of their software, and do things like server virtualisation to reduce the CPU count in your IDC. The embedded systems industry has plenty of experience in reducing power consumption through both hardware and software improvements. --Michael Dillon
Great notes! Very educational, I'm filing them all away. Too bad APC wasn't a part of this discussion, they've got some novel systems and are really focused on these issues. Regards, Thomas Leavitt On Thu, 2006-06-08 at 07:28 -0700, Matthew Petach wrote: > (ok, one more set of notes and then off to sit in traffic for an hour on > the way to work... --Matt) > > > 2006.06.06 Power and Cooling panel > Dan Golding, Tier1 research, moderator > > Hot Time in the Big IDC > Cooling, Power, and the Data Center > > 3 IDC vendors, 4 hardware vendors > Michael Laudon, force10 > Jay Park, equinix > Rob Snevely, Sun > Josh Snowhorn, terremark > David Tsiang, cisco > Brad Turner, juniper > Brian Young, S&D > > The power and cooling crisis > internet datacenters are getting full > most of the slack capacity has been used up > devices are using more and more power > low power density - routers, full sized servers > medium power density - 1u servers, switches > high power density - blade servers > Many data centers are full at 70-80% floor space > utilized > North America IDC occupancy is around 50% > most sought-after space is around 70% > > full when power and cooling capacity is used up, > floor space is vacant but can't be used. > > There is a relationship between power and cooling > devices are not 100% efficient > I^2R losses means that power becomes heat > (conservation of energy) > heat must be dissipated > The ability to dissipate heat with normal cooling > technologies is hitting the wall > need new techniques > > Some quick rules of thumb > a rack or cabinet is a standard unit of space > from 30-40sqft per rack > power is measured in watts > many facilities do around 80-100w/sqft; at 30sqft > per rack, that's about 3kw/rack > high > > how did we get here? > what is current situation > where are we going? > [dang, he's flying through his slides!!] > > Hardware engineers > T-series hardware engineer for Juniper > CRS-1 hardware > E-series > datacenter design issues for Sun, > there were other hardware vendors who were not > interested in showing up, these people were brave > for coming up here! > > Josh snowhorn, IDC planner > Jay Park, electrial engineer for equinix > Brian Young, S&D cage design specialist > > What do the IDC vendors feel the current situation > is in terms of power/cooling, how did we get here? > > Josh--designed datacenters at 100w/sq/ft, more than > enough for the carriers; the server guys hit 100w/sqft > in a quarter rack. you could cannabalize some power > and cooling, but still ran out of cooling. > Now spend hundreds of millions to make 200wsqft > datacenters, or higher. > > Now, to hardware vendors--why are their boxes > using up so much electricity, putting out so > much heat? > What are economics behind increasing density > and heat load? > > From high-end router space--it's been simple, the > bandwidth demand has grown faster than the power > efficiency can keep up with. In the past, had > the ability to improve keep up, do power spins about > every 2 years, half power; but now bandwidth is > doubling every year, but takes two years to drop > power in half. We've been loosing at this game > for a while, and running out of room on the voltage > scale; 90nm is down at 1v, can't go much lower, > since diode drop is at 0.7v; at 65nm, it's still > at 1v, there's no big hammer anymore for power > efficiency. Need to pull some tricks out, but > may need to do clock gating, may get some 20-30% > efficiency gains, but not much more that can be > pulled out of the bag now. > > Newton was right; you can do some tricks, but no > magic. Chip multithreading is one area they're > trying to squeeze more performance out of; don't > replicate ancillary ASICs for each core. Also > can more easily share memory, and nobody has a > 100% efficient power supply, so you lose some > power there too. > > More and more getting squeezed in each rack. > > Also a drive on cost; amortizing costs over > space and capability. > reducing costs per port is a big driver. > > And customers are pushing for more and more > density, since the cost of real-estate is getting > so high, since each square foot costs so high. > In Ginza, $120/sq ft for space. > > If you go to places where realestate is cheap, > easier/cheaper to just build really big rooms, > and let power dissipate more naturally. > > IDC people agree, some cities are just crazy > in real-estate costs. But for those in suburban > areas, cost of real-estate isn't so expensive. > 3kw per blade server, put a few in a rack, you > hit nearly 10kw in a rack. Soon, will need > direct chilled water in the rack to cool them. > But chilled water mixed with other colocation > and lower density cabinets is very challenging > to build. > But need to have enclosed space to handle local > chilled water coolers in localized racks. > 20 years ago at IBM, nobody wanted chilled water > in their hardware. Now, we're running out of > options. > > Disagree--other ways of handling the challenge; > how thermally efficient are the rooms in the > first place, and are there other ways of handling > heat issues? > Cables with a cutout in tiles allows air to escape > in areas that don't provide cooling. > > Josh notes the diversity between carriers at 40w/sq/ft > vs hosting providers at 400w/sq/ft is making engineering > decisions challenging. > > It's not about power really anymore, we can get power, > it's about cooling now. > > Dealing with space in wrong terms--watts/sq ft, vs > requirements of each rack. Charge customers based > on the cooling requirements? > > If you try to cool 15kw per cabinet, you still have > limits of how many cfm you can move through a given > space. At some point, the air flow vertically through > the rack starts to starve. > What about a dual push-pull air system that pushes from > the bottom and pulls from the top. > > Q: Randy, IIJ, question from the audience. He'll put as > hot stuff in there as he can cool, because he wants > the power, that's life. > Problem is cooling; over 3kw/4kw over current level, > the wind tunnel effect gets painful. > the option of putting water in the cabinets is a > dealbreaker for many people. > Fact is, most facilities can't even handle 3kw per > square meter; any build that can't meet that today > is unrealistic. > That's 300+ w per sq. ft. > Josh has some cabinets at NOTA; Akamai is at 386w/sq/ft, > they can cool it with huge hot aisle behind it, and > around carriers at 40w/sq/ft. > Those are the densest cabinets they have. > IDCs need to build them and charge a realistic amount; > people will burn as hot as possible, since they need > to move more and more data. > Raise plenums higher, move more air, air coming up > side of rack and across. > > Currently, equinix is building 4kw per cabinet, > planning for that in 2007. A cabinet is about > 2sq meters, so still not at the density Randy's > looking for. > Starting to separate high density users from > medium and low density users. > > Q: GNI, Derek ? datacenter in SF, 1008 IBM blade > servers, 2500 sq ft, ping pong table, soda > machine in surrounding areas. > need 2500w/sq/ft to deliver the same cabinet space. > 20kw per cabinet is what they can deliver in > cabinets. > He's paying for 100 cabinets, have 12 installed. > he's still netting efficiency for it. still gets > 3% better efficiency, still beats 84 1u pizza box > servers. If IDCs could keep up with that, could > keep physical space requirements more reasonable. > > The costs are exponential for more density. > Up to a year lead time for 2MW generators, we're > pushing the envelope on that. It is an exponential > increase. > Budget trauma when those costs get passed on. > Let the demand stimulate ingenuity. > The internet industry in general is short sighted. > 22 million blade servers installed, where they will > be located. > > Q: BillNorton. one other dimension. Life span for > new datacenters is 10-40 year timeframe, so it's hard > to adjust midlife to hugely different power and > cooling demands. > > From a technology point of view, CMOS was last > great quantum leap, need another great quantum > leap before we get relief on the cooling footprint. > Randy is right, the cooling architecture isn't > optimal. > CRS1, 20% of power goes to fans to move air past > convoluted air paths. > Spreading out the equipment is a mitigation. > multi-chassis systems will help with that. > Sun, Juniper, do you see power continuing > to grow linearly, or flatten out? > As they go to 40gig or 100gig, the power and > heat will continue to grow; more gates, more > heat, more power. We'll hit a wall soon. > > Cisco and Juniper agree, it's 6/6/06, take > note, world! > > 20 year shelf life for datacenter, look at where > they were 20 years ago. 10w/sq/foot back in 1986. > We've greatly increased the amount of work that > can be done since that time. Will machines > continue to do the same amount of work, or will > we flatten out on the machine capability curve? > Element of geographic progression as you double. > > Nobody will ever need more than 20kw per rack! > (Dan Golding) > > Running into some roadblocks; 100M gate ASICs, > packing so much power into a single chip, may > not be linear since can't move that much > heat out of the chip from the point source into > the system. > > Q: Patrick from Akamai--mcmurta base, south pole? > As a brighter light, spot of hope, hottest colo > in Terremark, finding they don't need more power, > coming to less power. Running into getting enough > spindles, the processors are getting faster and > drawing less power. 40 amps per rack, used to be > non-full, now able to fill them more completely. > not sure if everyone is seeing this, but their > power consumption is going down. > Not all doom and gloom, but for next 12 months, > at least somewhat lucky. > > Chip multicores is a good leap that can help for > a bit; like Sun's Niagra multicore chips or chip > multithreading, only about 50% of power is used by > real processing power, rest is ancillary power. > > Q: Rob Seastrom; BillN danced around the question; > seen it happen before. MAE-EAST, mark 3, additional > liebert challengers tucked in....if one builds a > datacenter to 4kw/meter^2, how long will that be > premium space vs no-longer-up-to-par. Does 20 > year colo life even make sense? Is the run rate > steep enough that the number is just one we're > fooling ourselves over? > Josh: none of them are running at this density. it's > the server density that hurts; carriers aren't as much > of the pain. > Separation of infrastructure most likely; Voice, carriers, > etc. there, and separate datacenters where server floors > exist. > 20 years from now--will it be obsolete? Yes, probably. > they'll keep doing what they can to help service their > customers. > > Q: Joel K, from ? -- what's coming in network equipment > to help cut power? throttle back linecards that > aren't running at full bore? > A: If you make it automatic, service providers would > consider it; but from the bandwidth demand growth, > the exponential growth--technology isn't keeping > up, it's plateauing. Multithreaded ideas, turn > off idle logic portions, incremental improvements, > they're one-shot efforts, won't really help fix > the slope of the curve. > May just accept that we need more space, period. > High speed/low speed fans, only kick up to high > speed during thermal extremes. Again, both Cisco > and Juniper have explored suppressing some gear, > but customers still want 50ms protect gear response, > so they can't really shut down. > Even making heat sinks to move the heat is getting > challenging! > Force10 also talked to customers about it; in order > to do turning off portions and then turn back on, > incurs latency, buffer some packets, etc; people > can be sensitive to jitter and latency. > Pushback has been fairly large from other sources. > > Q: Rick Wesson--to colo vendors--when will heat/BTUs be > a part of of the charges? And to server vendors, when > will heat be a listed component upon which vendors make > sales? > A: IDCs don't charge based on heat load. > Power as proxy for heat right now. > Cooling overhead is wrapped into cost of sq ft and power; > costs from utilities have been going up 30% due to > oil prices going up. > Might make it easier to add that charge for customers. > Hardware vendors are certainly seeing power/heat > limitations in RFPs. > Building smaller systems with fewer slots to meet those > RFPs. > Customers asking for gbits/kw now from network gear. > Sun notes that total cost of ownership, power may > cost more to run the server than the cost of the > server itself. > > Q: Lane Patterson, equinix. T640, if you redesigned it > today, what fraction of power would it use today > compared to past? > Do they engineer gear to see how much they can pack > into the same power/heat footprint? > But customers are also asking for more and more > capacity, less likely to pay for holding the line > at same power/heat as previous generation. > Cisco reiterates that we're running out of tricks; > we can hold the line for a product generation, but > after that, we're out of luck. We may need to > shift architecture of pops going forward. > Why not build in cheaper places, and backhaul? > > Q: Jared Mauch, NTT--huge customer demand; no vendors > are proving interfaces greater than 40gig; not for > next 3 years at least will there be faster links; > backhauling from remote locations requires aggregating > more and more traffic; if link speeds aren't increasing, > backhaul isn't practical. > As media companies continue deciding they can > sell movies, music, and the like online, we may > start hitting the wall; demand on all sides is > growing, and we're running out of ways to address > these challenges. > > Q: Avi Freedman. Talks to people doing lots of very > dense disk solutions. Rackable solutions working > on high density storage racks using laptop drives. > 48 disks for you starts to generate a lot of heat; > thumper product? > 4u, 196 laptop disk rack unit? For people who need > lots of spindles, lots of IOPS. > A: can't talk about those products, they showed > up in Jonathan's blog, but don't exist yet. > There are always going to be limitations, the > vendor will expect you can run the box in the > location you're going to put it; that is, box > has requirements, need to make sure customers > are installing boxes in areas where the thermal > issues are being considered. > > Q: phil, rosenthal, ISprime. > people on the panel are pretty good, not the worst > offenders. You need to hit the 1u server people > where most power is being wasted; Dell 1650 vs > Dell 1850; processor time, sitting at 90% idle > on both system for bottom of line servers, do > we need lower end CPUs on server lines so the > CPUs won't be idle. > A: why not use fewer machines, but have them do > more work each? Virtualization might help us a > bit in these areas, where we get more efficient > use of the servers already in place. > > Equinix notes neutral current dropped a lot, > people using 208V instead of 120V, generates > less heat to the datacenter as well. > > Q: Randy; on left, crew singing I want my p2p. > will always have max heat in the rack; servers and > router vendors will keep working as hard as they > can to do what they can. > They had to leave 30% of their datacenter empty > and build a bistro in it because they couldn't > handle the heat budget. > The IDC vendors [sorry, missed the comment] > > Q: Tim Elisio, new metric? > To what extent is standardization, like using > larger, more efficient ower supplies, or more > efficient fans, cooling systems, etc. helping? > A: IEEE meetings talking about some standardization, > get some savings; the economies of scale helps make > more efficient products on the standardized products. > Telecom/router industry is working to old standards; > may need to re-think what airflow standards are, > for example. > More dollars in a particular area helps push > research and development in that direction. > Juniper notes internals can be optimized, but > the external plant and interfaces therein need > better standardization to get economies of scale. > Everyone's using multispeed fans, use them when > you need them. > 3 orgs, SPEC, ECL forum/EPA (energystar for servers > and blade servers), and GreenGrid. > Will see benchmarks coming out; will start asking > all vendors to start compete on how much work they > can do per how much power sucked and heat generated. > Make the hardware vendors compete on how efficiently > they use power and generate heat; we can then decide > with dollars on who will win. > > Helps motivate people to optimize on the axis they > care about. > > But are vendors talking to each other about how they > can use standardized gear and standardized facilities > designs more efficiently? > ASHRAY?, heating/refrigeration group puts specs on how > machines should be cooled (front to back, etc). > But vendors don't want to help each other compete > because that hurts their business. > > Dan Golding, 30 seconds, what would each person like > to see the folks on other side do to help. > > Brian: asked vendors to have more efficient power > supplies, more efficient systems that generate less > heat for them to dissipate. > Equinix--challenged by power density; customers > don't understand, they want to put in smallest cage > as possible. Need people to understand heat load > better! > Asks customers to use blanks in unused rack space > to isolate cold and hot aisles. Too much leakage > from cold aisles to hot aisles. Put blanks in!! > Josh. Everyone building hot datacenters; would like > to see vendors come into IDCs, test them in real > world environments, put them in labs, see how they > stand up to environment, test glycol taps, water taps, > etc. Building servers is faster than building > datacenters! > > Hardware vendor; PGE, worked with them to measure > the increased efficiency, blanks DO help!! > Education, amongst each other and customers. > Watts per sq ft is crazy, do it on a per rack > basis, makes it easier for customers to understand > the limitations. > Force10, if IDC groups got together, if there was > a forum or group they could work with; right now, > everyone has different requirements. Otherwise, > always doing multiple tradeoffs, if there were > a more general consensus, easier to engineer for. > Cisco--good point, get IDCs and service providers > to meet with vendors, come up with a next generation > facility architecture to aim and build for. Hopefully > make cooling and airflow easier, reduce the amount > of power used by fans. > Juniper--sees RFPs from customers, environment specs > are very diverse; would be good to have common > standards for customers to aim for; also, update > some outdated nomeclature, use common terminology. > > Michael, Josh, Rob, Brian, thanks to all the panelists, > Steve Feldman, we've slipped by 15 minutes, start at 2:15, > everything will slip thereafter > > LUNCH! -- Thomas Leavitt <thomas@thomasleavitt.org> - 831-295-3917 Godmoma's Forge, LLC - www.godmomasforge.com - Web and graphic design made spiffy - Encrypted public key at http://www.thomasleavitt.org/thomas.asc Download GnuPG (including for Windows) at http://www.gnupg.org/download/ to read .asc attachment (encrypted signature)
participants (3)
-
Matthew Petach -
Michael.Dillon＠btradianz.com -
Thomas Leavitt