8 Jun
2006
8 Jun
'06
2:35 p.m.
Great notes! Very educational, I'm filing them all away. Too bad APC wasn't a part of this discussion, they've got some novel systems and are really focused on these issues. Regards, Thomas Leavitt On Thu, 2006-06-08 at 07:28 -0700, Matthew Petach wrote: > (ok, one more set of notes and then off to sit in traffic for an hour on > the way to work... --Matt) > > > 2006.06.06 Power and Cooling panel > Dan Golding, Tier1 research, moderator > > Hot Time in the Big IDC > Cooling, Power, and the Data Center > > 3 IDC vendors, 4 hardware vendors > Michael Laudon, force10 > Jay Park, equinix > Rob Snevely, Sun > Josh Snowhorn, terremark > David Tsiang, cisco > Brad Turner, juniper > Brian Young, S&D > > The power and cooling crisis > internet datacenters are getting full > most of the slack capacity has been used up > devices are using more and more power > low power density - routers, full sized servers > medium power density - 1u servers, switches > high power density - blade servers > Many data centers are full at 70-80% floor space > utilized > North America IDC occupancy is around 50% > most sought-after space is around 70% > > full when power and cooling capacity is used up, > floor space is vacant but can't be used. > > There is a relationship between power and cooling > devices are not 100% efficient > I^2R losses means that power becomes heat > (conservation of energy) > heat must be dissipated > The ability to dissipate heat with normal cooling > technologies is hitting the wall > need new techniques > > Some quick rules of thumb > a rack or cabinet is a standard unit of space > from 30-40sqft per rack > power is measured in watts > many facilities do around 80-100w/sqft; at 30sqft > per rack, that's about 3kw/rack > high > > how did we get here? > what is current situation > where are we going? > [dang, he's flying through his slides!!] > > Hardware engineers > T-series hardware engineer for Juniper > CRS-1 hardware > E-series > datacenter design issues for Sun, > there were other hardware vendors who were not > interested in showing up, these people were brave > for coming up here! > > Josh snowhorn, IDC planner > Jay Park, electrial engineer for equinix > Brian Young, S&D cage design specialist > > What do the IDC vendors feel the current situation > is in terms of power/cooling, how did we get here? > > Josh--designed datacenters at 100w/sq/ft, more than > enough for the carriers; the server guys hit 100w/sqft > in a quarter rack. you could cannabalize some power > and cooling, but still ran out of cooling. > Now spend hundreds of millions to make 200wsqft > datacenters, or higher. > > Now, to hardware vendors--why are their boxes > using up so much electricity, putting out so > much heat? > What are economics behind increasing density > and heat load? > > From high-end router space--it's been simple, the > bandwidth demand has grown faster than the power > efficiency can keep up with. In the past, had > the ability to improve keep up, do power spins about > every 2 years, half power; but now bandwidth is > doubling every year, but takes two years to drop > power in half. We've been loosing at this game > for a while, and running out of room on the voltage > scale; 90nm is down at 1v, can't go much lower, > since diode drop is at 0.7v; at 65nm, it's still > at 1v, there's no big hammer anymore for power > efficiency. Need to pull some tricks out, but > may need to do clock gating, may get some 20-30% > efficiency gains, but not much more that can be > pulled out of the bag now. > > Newton was right; you can do some tricks, but no > magic. Chip multithreading is one area they're > trying to squeeze more performance out of; don't > replicate ancillary ASICs for each core. Also > can more easily share memory, and nobody has a > 100% efficient power supply, so you lose some > power there too. > > More and more getting squeezed in each rack. > > Also a drive on cost; amortizing costs over > space and capability. > reducing costs per port is a big driver. > > And customers are pushing for more and more > density, since the cost of real-estate is getting > so high, since each square foot costs so high. > In Ginza, $120/sq ft for space. > > If you go to places where realestate is cheap, > easier/cheaper to just build really big rooms, > and let power dissipate more naturally. > > IDC people agree, some cities are just crazy > in real-estate costs. But for those in suburban > areas, cost of real-estate isn't so expensive. > 3kw per blade server, put a few in a rack, you > hit nearly 10kw in a rack. Soon, will need > direct chilled water in the rack to cool them. > But chilled water mixed with other colocation > and lower density cabinets is very challenging > to build. > But need to have enclosed space to handle local > chilled water coolers in localized racks. > 20 years ago at IBM, nobody wanted chilled water > in their hardware. Now, we're running out of > options. > > Disagree--other ways of handling the challenge; > how thermally efficient are the rooms in the > first place, and are there other ways of handling > heat issues? > Cables with a cutout in tiles allows air to escape > in areas that don't provide cooling. > > Josh notes the diversity between carriers at 40w/sq/ft > vs hosting providers at 400w/sq/ft is making engineering > decisions challenging. > > It's not about power really anymore, we can get power, > it's about cooling now. > > Dealing with space in wrong terms--watts/sq ft, vs > requirements of each rack. Charge customers based > on the cooling requirements? > > If you try to cool 15kw per cabinet, you still have > limits of how many cfm you can move through a given > space. At some point, the air flow vertically through > the rack starts to starve. > What about a dual push-pull air system that pushes from > the bottom and pulls from the top. > > Q: Randy, IIJ, question from the audience. He'll put as > hot stuff in there as he can cool, because he wants > the power, that's life. > Problem is cooling; over 3kw/4kw over current level, > the wind tunnel effect gets painful. > the option of putting water in the cabinets is a > dealbreaker for many people. > Fact is, most facilities can't even handle 3kw per > square meter; any build that can't meet that today > is unrealistic. > That's 300+ w per sq. ft. > Josh has some cabinets at NOTA; Akamai is at 386w/sq/ft, > they can cool it with huge hot aisle behind it, and > around carriers at 40w/sq/ft. > Those are the densest cabinets they have. > IDCs need to build them and charge a realistic amount; > people will burn as hot as possible, since they need > to move more and more data. > Raise plenums higher, move more air, air coming up > side of rack and across. > > Currently, equinix is building 4kw per cabinet, > planning for that in 2007. A cabinet is about > 2sq meters, so still not at the density Randy's > looking for. > Starting to separate high density users from > medium and low density users. > > Q: GNI, Derek ? datacenter in SF, 1008 IBM blade > servers, 2500 sq ft, ping pong table, soda > machine in surrounding areas. > need 2500w/sq/ft to deliver the same cabinet space. > 20kw per cabinet is what they can deliver in > cabinets. > He's paying for 100 cabinets, have 12 installed. > he's still netting efficiency for it. still gets > 3% better efficiency, still beats 84 1u pizza box > servers. If IDCs could keep up with that, could > keep physical space requirements more reasonable. > > The costs are exponential for more density. > Up to a year lead time for 2MW generators, we're > pushing the envelope on that. It is an exponential > increase. > Budget trauma when those costs get passed on. > Let the demand stimulate ingenuity. > The internet industry in general is short sighted. > 22 million blade servers installed, where they will > be located. > > Q: BillNorton. one other dimension. Life span for > new datacenters is 10-40 year timeframe, so it's hard > to adjust midlife to hugely different power and > cooling demands. > > From a technology point of view, CMOS was last > great quantum leap, need another great quantum > leap before we get relief on the cooling footprint. > Randy is right, the cooling architecture isn't > optimal. > CRS1, 20% of power goes to fans to move air past > convoluted air paths. > Spreading out the equipment is a mitigation. > multi-chassis systems will help with that. > Sun, Juniper, do you see power continuing > to grow linearly, or flatten out? > As they go to 40gig or 100gig, the power and > heat will continue to grow; more gates, more > heat, more power. We'll hit a wall soon. > > Cisco and Juniper agree, it's 6/6/06, take > note, world! > > 20 year shelf life for datacenter, look at where > they were 20 years ago. 10w/sq/foot back in 1986. > We've greatly increased the amount of work that > can be done since that time. Will machines > continue to do the same amount of work, or will > we flatten out on the machine capability curve? > Element of geographic progression as you double. > > Nobody will ever need more than 20kw per rack! > (Dan Golding) > > Running into some roadblocks; 100M gate ASICs, > packing so much power into a single chip, may > not be linear since can't move that much > heat out of the chip from the point source into > the system. > > Q: Patrick from Akamai--mcmurta base, south pole? > As a brighter light, spot of hope, hottest colo > in Terremark, finding they don't need more power, > coming to less power. Running into getting enough > spindles, the processors are getting faster and > drawing less power. 40 amps per rack, used to be > non-full, now able to fill them more completely. > not sure if everyone is seeing this, but their > power consumption is going down. > Not all doom and gloom, but for next 12 months, > at least somewhat lucky. > > Chip multicores is a good leap that can help for > a bit; like Sun's Niagra multicore chips or chip > multithreading, only about 50% of power is used by > real processing power, rest is ancillary power. > > Q: Rob Seastrom; BillN danced around the question; > seen it happen before. MAE-EAST, mark 3, additional > liebert challengers tucked in....if one builds a > datacenter to 4kw/meter^2, how long will that be > premium space vs no-longer-up-to-par. Does 20 > year colo life even make sense? Is the run rate > steep enough that the number is just one we're > fooling ourselves over? > Josh: none of them are running at this density. it's > the server density that hurts; carriers aren't as much > of the pain. > Separation of infrastructure most likely; Voice, carriers, > etc. there, and separate datacenters where server floors > exist. > 20 years from now--will it be obsolete? Yes, probably. > they'll keep doing what they can to help service their > customers. > > Q: Joel K, from ? -- what's coming in network equipment > to help cut power? throttle back linecards that > aren't running at full bore? > A: If you make it automatic, service providers would > consider it; but from the bandwidth demand growth, > the exponential growth--technology isn't keeping > up, it's plateauing. Multithreaded ideas, turn > off idle logic portions, incremental improvements, > they're one-shot efforts, won't really help fix > the slope of the curve. > May just accept that we need more space, period. > High speed/low speed fans, only kick up to high > speed during thermal extremes. Again, both Cisco > and Juniper have explored suppressing some gear, > but customers still want 50ms protect gear response, > so they can't really shut down. > Even making heat sinks to move the heat is getting > challenging! > Force10 also talked to customers about it; in order > to do turning off portions and then turn back on, > incurs latency, buffer some packets, etc; people > can be sensitive to jitter and latency. > Pushback has been fairly large from other sources. > > Q: Rick Wesson--to colo vendors--when will heat/BTUs be > a part of of the charges? And to server vendors, when > will heat be a listed component upon which vendors make > sales? > A: IDCs don't charge based on heat load. > Power as proxy for heat right now. > Cooling overhead is wrapped into cost of sq ft and power; > costs from utilities have been going up 30% due to > oil prices going up. > Might make it easier to add that charge for customers. > Hardware vendors are certainly seeing power/heat > limitations in RFPs. > Building smaller systems with fewer slots to meet those > RFPs. > Customers asking for gbits/kw now from network gear. > Sun notes that total cost of ownership, power may > cost more to run the server than the cost of the > server itself. > > Q: Lane Patterson, equinix. T640, if you redesigned it > today, what fraction of power would it use today > compared to past? > Do they engineer gear to see how much they can pack > into the same power/heat footprint? > But customers are also asking for more and more > capacity, less likely to pay for holding the line > at same power/heat as previous generation. > Cisco reiterates that we're running out of tricks; > we can hold the line for a product generation, but > after that, we're out of luck. We may need to > shift architecture of pops going forward. > Why not build in cheaper places, and backhaul? > > Q: Jared Mauch, NTT--huge customer demand; no vendors > are proving interfaces greater than 40gig; not for > next 3 years at least will there be faster links; > backhauling from remote locations requires aggregating > more and more traffic; if link speeds aren't increasing, > backhaul isn't practical. > As media companies continue deciding they can > sell movies, music, and the like online, we may > start hitting the wall; demand on all sides is > growing, and we're running out of ways to address > these challenges. > > Q: Avi Freedman. Talks to people doing lots of very > dense disk solutions. Rackable solutions working > on high density storage racks using laptop drives. > 48 disks for you starts to generate a lot of heat; > thumper product? > 4u, 196 laptop disk rack unit? For people who need > lots of spindles, lots of IOPS. > A: can't talk about those products, they showed > up in Jonathan's blog, but don't exist yet. > There are always going to be limitations, the > vendor will expect you can run the box in the > location you're going to put it; that is, box > has requirements, need to make sure customers > are installing boxes in areas where the thermal > issues are being considered. > > Q: phil, rosenthal, ISprime. > people on the panel are pretty good, not the worst > offenders. You need to hit the 1u server people > where most power is being wasted; Dell 1650 vs > Dell 1850; processor time, sitting at 90% idle > on both system for bottom of line servers, do > we need lower end CPUs on server lines so the > CPUs won't be idle. > A: why not use fewer machines, but have them do > more work each? Virtualization might help us a > bit in these areas, where we get more efficient > use of the servers already in place. > > Equinix notes neutral current dropped a lot, > people using 208V instead of 120V, generates > less heat to the datacenter as well. > > Q: Randy; on left, crew singing I want my p2p. > will always have max heat in the rack; servers and > router vendors will keep working as hard as they > can to do what they can. > They had to leave 30% of their datacenter empty > and build a bistro in it because they couldn't > handle the heat budget. > The IDC vendors [sorry, missed the comment] > > Q: Tim Elisio, new metric? > To what extent is standardization, like using > larger, more efficient ower supplies, or more > efficient fans, cooling systems, etc. helping? > A: IEEE meetings talking about some standardization, > get some savings; the economies of scale helps make > more efficient products on the standardized products. > Telecom/router industry is working to old standards; > may need to re-think what airflow standards are, > for example. > More dollars in a particular area helps push > research and development in that direction. > Juniper notes internals can be optimized, but > the external plant and interfaces therein need > better standardization to get economies of scale. > Everyone's using multispeed fans, use them when > you need them. > 3 orgs, SPEC, ECL forum/EPA (energystar for servers > and blade servers), and GreenGrid. > Will see benchmarks coming out; will start asking > all vendors to start compete on how much work they > can do per how much power sucked and heat generated. > Make the hardware vendors compete on how efficiently > they use power and generate heat; we can then decide > with dollars on who will win. > > Helps motivate people to optimize on the axis they > care about. > > But are vendors talking to each other about how they > can use standardized gear and standardized facilities > designs more efficiently? > ASHRAY?, heating/refrigeration group puts specs on how > machines should be cooled (front to back, etc). > But vendors don't want to help each other compete > because that hurts their business. > > Dan Golding, 30 seconds, what would each person like > to see the folks on other side do to help. > > Brian: asked vendors to have more efficient power > supplies, more efficient systems that generate less > heat for them to dissipate. > Equinix--challenged by power density; customers > don't understand, they want to put in smallest cage > as possible. Need people to understand heat load > better! > Asks customers to use blanks in unused rack space > to isolate cold and hot aisles. Too much leakage > from cold aisles to hot aisles. Put blanks in!! > Josh. Everyone building hot datacenters; would like > to see vendors come into IDCs, test them in real > world environments, put them in labs, see how they > stand up to environment, test glycol taps, water taps, > etc. Building servers is faster than building > datacenters! > > Hardware vendor; PGE, worked with them to measure > the increased efficiency, blanks DO help!! > Education, amongst each other and customers. > Watts per sq ft is crazy, do it on a per rack > basis, makes it easier for customers to understand > the limitations. > Force10, if IDC groups got together, if there was > a forum or group they could work with; right now, > everyone has different requirements. Otherwise, > always doing multiple tradeoffs, if there were > a more general consensus, easier to engineer for. > Cisco--good point, get IDCs and service providers > to meet with vendors, come up with a next generation > facility architecture to aim and build for. Hopefully > make cooling and airflow easier, reduce the amount > of power used by fans. > Juniper--sees RFPs from customers, environment specs > are very diverse; would be good to have common > standards for customers to aim for; also, update > some outdated nomeclature, use common terminology. > > Michael, Josh, Rob, Brian, thanks to all the panelists, > Steve Feldman, we've slipped by 15 minutes, start at 2:15, > everything will slip thereafter > > LUNCH! -- Thomas Leavitt <thomas@thomasleavitt.org> - 831-295-3917 Godmoma's Forge, LLC - www.godmomasforge.com - Web and graphic design made spiffy - Encrypted public key at http://www.thomasleavitt.org/thomas.asc Download GnuPG (including for Windows) at http://www.gnupg.org/download/ to read .asc attachment (encrypted signature)