RE: 24x7 Support Strategies
[Sorry, I have a hosed copy of #*@! Outlook, which crashes whenever I tell it to prefix earlier comments with >] -----Original Message----- From: William F. Maton Sotomayor [mailto:wmaton@ryouko.imsb.nrc.ca] Sent: Thursday, June 14, 2007 8:14 AM To: Howard C. Berkowitz Subject: RE: 24x7 Support Strategies On Thu, 14 Jun 2007, Howard C. Berkowitz wrote:
This topic interests me very much, and I had a BOF about staff development at the Montreal meeting in 1999. I remember some of the details, and,
while
I am no longer generally doing course development, I have some pretty strong ideas of what reasonably constitutes a proper training sandbox for a major ISP.
Actually, it's BOFs like these that I would like to see revisited from time to time by NANOG. The 'big' BOF I always wanted to see revisited in great detail was the data center one from the late 1990's. [Howard C. Berkowitz] I remember that one well. As an aside, while I recognize it would be a whole new level of effort, I'd really like to see -- and will work on if needed -- some NANOG documents other than copies of presentations. The data center ones were very good, and Sean Donelan has a couple of great tables that pull together a huge amount of information on data/carrier centers. A related area that might well be worth revisiting is cooling. IIRC, it was someone from Google, at the Intel developer conference, who said that their power and HVAC costs were rapidly approaching the cost of their servers. He laid down a challenge for chipmakers to be more energy-efficient. My understanding is that at a certain level of scaling, chilled water becomes very efficient. A few municipal areas have chilled water utilities, while in others, you may need to build your own system. One of the immediate questions will be to what extent you can use liquid chilling in racks or even components, and when the best will be a rack heat exchanger or chilled air input coming from the chilled water. This is also a topic for operation in disasters, which is where my St. Louis talk morphed. While I don't have all the details and would welcome them, I understand that New Orleans made a hard decision during Katrina, which probably would not have been made if there weren't contingency plans for it. A telco tandem/toll office was one of the last ones operating in an area, and it needed chilled water. Some of the chilled water distribution had been knocked out. The city diverted a fire engine, even though there were fires, to act as an interim pump so they continued to have long-distance telephony.
A related area that might well be worth revisiting is cooling. IIRC, it was someone from Google, at the Intel developer conference, who said that their power and HVAC costs were rapidly approaching the cost of their servers. He laid down a challenge for chipmakers to be more energy-efficient.
And what about all those diesel generators? How many of them are capable of running on vegetable oil rather than diesel oil? I regularly walk past a building in London that reeks because of the diesel fuel tanks in the basement. You have to wonder about the safety of storing large amounts of petroleum oil in the centers of major cities when vegetable oil is safer, and more carbon-friendly. But back to chips and heat generation. Has anyone instrumented some of these servers (and their software) to figure out how much heat various functions generate? A few months ago, I walked away from my desk to get a cup of tea, and stretch my legs for a few minutes. I was away about 25 minutes and when I came back, my laptop had a scorched smell coming out of it (probably dust on the cpu chip). I closed all my apps, shut it down, waited 5 minutes to let it cool and restarted it. After a while, I noticed the smell again, shortly after I restarted an Oracle client install that I hadn't completed during the earlier incident. This install wasn't completing because I didn't have the right information to identify which database servers to connect to and it was spinning its wheels. This particular application was generating so much heat that the dust on the CPU chip was being scorched. I closed the app, and the smell faded. So, how do we know that the heat generated by all this software on all these servers is actually providing any value at all to either the data center owners or the server owners? In order to know this, we need to measure what is generating the heat, then change the software to remove bad behaviors. Since a lot of these servers are running open source software, this is not as hard to implement as you might think. I suspect that you would need to make special modifications to the hardware of a server to install temperature and current measuring devices in key locations and feed all this data into a separate machine for analysis. Also, I expect that the embedded system industry with their experience of building low-power consumption devices, might be able to help out. --Michael Dillon
Didn't we discuss the need for standard water connectors not so long ago? Water over Ethernet?
On 2007/06/14 15:11, michael.dillon@bt.com wrote:
I suspect that you would need to make special modifications to the hardware of a server to install temperature and current measuring devices in key locations
The vast majority of modern machines have working temperature sensors; some just one or two (e.g. cpu and case temperature), others have a huge range. If you're unlucky and use an OS which doesn't provide access and monitoring of these as part of the standard installation, third-party software is usually available. Non-CPU-bound servers can often benefit from enabling power-saving features too (e.g. speedstep, powernow/cool'n'quiet). In some cases this is very simple and offers significant reduction in power consumption and heat generation.
But back to chips and heat generation. Has anyone instrumented some of these servers (and their software) to figure out how much heat various functions generate?
It seems that someone has done just that. A list member sent me a private reply pointing me to http://www.linuxpowertop.org/ I still think that special hardware that can work with profiling data to target specific blocks of code inside the process would also be useful to identify best programming practices. However, many programmers debug their code quite effectively without profilers by means of print statements, and this powertop tool can be leveraged by a programmer with source code available. If nothing else, code two variations of a program and compare powertop results. --Michael Dillon
On Jun 14, 2007, at 10:11 AM, <michael.dillon@bt.com> <michael.dillon@bt.com> wrote
And what about all those diesel generators? How many of them are capable of running on vegetable oil rather than diesel oil? I regularly walk past a building in London that reeks because of the diesel fuel tanks in the basement. You have to wonder about the safety of storing large amounts of petroleum oil in the centers of major cities when vegetable oil is safer, and more carbon-friendly.
Vegetable oil can degrade much faster than diesel. What you really want is a large pond at the top of a hill, and another large pond at the bottom of the same hill. When utility prices are low, pump the water to the upper pond. When power is needed, have your installed hydropower setup allow water to flow through the turbine from the upper pond to the lower pond. In a city, a very large tank located at the top of the building and an equally large one in the bottom with a pipe between the two, should suffice. Remember that the "head" or height difference is a large factor in determining how much power a hydro setup can generate. Patrick Giagnocavo patrick@zill.net
On Thu, 14 Jun 2007, Patrick Giagnocavo wrote:
Vegetable oil can degrade much faster than diesel.
What you really want is a large pond at the top of a hill, and another large pond at the bottom of the same hill.
When utility prices are low, pump the water to the upper pond. When power is needed, have your installed hydropower setup allow water to flow through the turbine from the upper pond to the lower pond.
In a city, a very large tank located at the top of the building and an equally large one in the bottom with a pipe between the two, should suffice. Remember that the "head" or height difference is a large factor in determining how much power a hydro setup can generate. I just wanted to give a little bit more perspective on above: 1 liter of diesel fuel contains approx 10000WH.
1 liter of water pumped 100 meters up has a potential energy of .272WH It takes a *lot* of water to provide a measurable difference for a datacenter of any significant size... -alex
OK guys..... I get it...... you are not cert fans. I did mention that when I interview candidates, the most knowledgeable gets the position. Did I mention that I can take an engine or transmission apart, then put it back together? ( or have it on a bench unseen and reassemble it... just a hobby ) We got way off track here.... I'll end this thread now..... Enjoy ! Regards BF -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Alex Pilosov Sent: Thursday, June 14, 2007 8:28 PM To: Patrick Giagnocavo Cc: nanog@nanog.org Subject: Re: 24x7 Support Strategies On Thu, 14 Jun 2007, Patrick Giagnocavo wrote:
Vegetable oil can degrade much faster than diesel.
What you really want is a large pond at the top of a hill, and another large pond at the bottom of the same hill.
When utility prices are low, pump the water to the upper pond. When power is needed, have your installed hydropower setup allow water to flow through the turbine from the upper pond to the lower pond.
In a city, a very large tank located at the top of the building and an equally large one in the bottom with a pipe between the two, should suffice. Remember that the "head" or height difference is a large factor in determining how much power a hydro setup can generate. I just wanted to give a little bit more perspective on above: 1 liter of
diesel fuel contains approx 10000WH. 1 liter of water pumped 100 meters up has a potential energy of .272WH It takes a *lot* of water to provide a measurable difference for a datacenter of any significant size... -alex
participants (7)
-
Alex Pilosov -
Alexander Harrowell -
Farrell,Bob -
Howard C. Berkowitz -
michael.dillon＠bt.com -
Patrick Giagnocavo -
Stuart Henderson