Hot weather and power outages continue
Due to the hot weather in many parts of the US, there have been various power outages. Some large outages have been caused by severe storms, but mostly the heat has just overloaded power distribution equipment. The good news so far is the Net has shown very few disruptions due to the heat and some multi-day power outages. The major ISP access providers have been indicating about average numbers of outages in their networks, i.e. even during "normal" times, there are usually a few tens of thousands of lines down nationwide. While its expected for individual customers to go down during power outages, usually because the customer does not have local backup power, it is less common for major web sites and co-location centers to experience downtime during power outages. MySpace.com reported it was down tonight due to power problems, with a temporary page up now. MySpace.com is hosted at a few Internet co-location data centers (Equinix and CWIE).
On Mon, Jul 24, 2006 at 02:22:26AM -0400, Sean Donelan wrote:
Due to the hot weather in many parts of the US, there have been various power outages. Some large outages have been caused by severe storms, but mostly the heat has just overloaded power distribution equipment.
The good news so far is the Net has shown very few disruptions due to the heat and some multi-day power outages. The major ISP access providers have been indicating about average numbers of outages in their networks, i.e. even during "normal" times, there are usually a few tens of thousands of lines down nationwide.
I'm surprised nobody said anything about the (apparently regional) utility outage in NoVA on Saturday. Equinix Ashburn was running on generator for several hours, but apparently the SAVVIS facility down the road a few miles in Sterling (old Exodus facility) didn't fare so well. The latest story I heard was that they lost 14 out of 16 chillers and customers had to send techs in in shifts because it reached over 130F inside. Come on Sean, this "very few disruptions" stuff is below your usual standards. The least you can do to help us pass the time in this damn heat is to recount a few good stories about routers you could scramble eggs on. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
I'm surprised nobody said anything about the (apparently regional) utility outage in NoVA on Saturday.
There have been several around the country. Secaucus, NJ, various cities near Los Angeles and the SF Bay Area, not to mention the power quality goes to hell when the grid gets very hot. I've been a bit surprised we haven't heard about more equipment failures.
Equinix Ashburn was running on generator for several hours, but apparently the SAVVIS facility down the road a few miles in Sterling (old Exodus facility) didn't fare so well. The latest story I heard was that they lost 14 out of 16 chillers and customers had to send techs in in shifts because it reached over 130F inside.
I've always been a fan of being able to force 100% economizer and chiller loop bypass emergency operation; it won't keep you "cool" but will help keep your data center from turning into an Easy-Bake Oven(tm). But that failure operating mode is rarely part of the standard HVAC programming. You are probably still going to need to replace some toasted disks and chips afterwards. There is a reason why NEBS equipment costs more.
Come on Sean, this "very few disruptions" stuff is below your usual standards. The least you can do to help us pass the time in this damn heat is to recount a few good stories about routers you could scramble eggs on. :)
I remember back in the old days, when all we had were AGS+ routers in wiring closets without any air conditioning .... We had to carry them uphill, both ways.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Sean Donelan Sent: 24 July 2006 13:27 To: nanog@merit.edu Subject: Re: Hot weather and power outages continue
I've always been a fan of being able to force 100% economizer and chiller loop bypass emergency operation; it won't keep you "cool" but will help keep your data center from turning into an Easy-Bake Oven(tm). But that failure operating mode is rarely part of the standard HVAC programming.
Sean, Can you elaborate on what you mean by " force 100% economizer and chiller loop bypass emergency operation" Thanks, Sam
My assumption is that it means "it isn't going to keep things cold, but it will keep the air flowing to prevent a 'server sauna'". -brandon On 7/25/06, Sam Stickland <sam_mailinglists@spacething.org> wrote:
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Sean Donelan Sent: 24 July 2006 13:27 To: nanog@merit.edu Subject: Re: Hot weather and power outages continue
I've always been a fan of being able to force 100% economizer and chiller loop bypass emergency operation; it won't keep you "cool" but will help keep your data center from turning into an Easy-Bake Oven(tm). But that failure operating mode is rarely part of the standard HVAC programming.
Sean,
Can you elaborate on what you mean by " force 100% economizer and chiller loop bypass emergency operation"
Thanks,
Sam
-- Brandon Galbraith Email: brandon.galbraith@gmail.com AIM: brandong00 Voice: 630.400.6992 "A true pirate starts drinking before the sun hits the yard-arm. Yarrrr. --thelost"
Brandon Galbraith wrote:
My assumption is that it means "it isn't going to keep things cold, but it will keep the air flowing to prevent a 'server sauna'".
On 7/25/06, Sam Stickland <sam_mailinglists@spacething.org> wrote:
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Sean Donelan Sent: 24 July 2006 13:27 To: nanog@merit.edu Subject: Re: Hot weather and power outages continue
I've always been a fan of being able to force 100% economizer and chiller loop bypass emergency operation; it won't keep you "cool" but will help keep your data center from turning into an Easy-Bake Oven(tm). But that failure operating mode is rarely part of the standard HVAC programming.
Sean,
Can you elaborate on what you mean by " force 100% economizer and chiller loop bypass emergency operation"
Thanks,
Sam
When I was in charge of such things, there was a way to circulate evaporator tower water n the chilled water loop to remove some of the heat, if you had enough power to run the two pumps. -- Requiescas in pace o email Ex turpi causa non oritur actio http://members.cox.net/larrysheldon/
On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
Come on Sean, this "very few disruptions" stuff is below your usual standards. The least you can do to help us pass the time in this damn heat is to recount a few good stories about routers you could scramble eggs on. :)
there is a funny story of some dial devices on fire, and still passing packets...
"Christopher L. Morrow" <christopher.morrow@verizonbusiness.com> writes:
On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
Come on Sean, this "very few disruptions" stuff is below your usual standards. The least you can do to help us pass the time in this damn heat is to recount a few good stories about routers you could scramble eggs on. :)
there is a funny story of some dial devices on fire, and still passing packets...
and an equally funny story of said devices being held up in customs in a particular european country because they said "TNT" on the outside of their crates... ---rob
Robert E.Seastrom wrote:
"Christopher L. Morrow" <christopher.morrow@verizonbusiness.com> writes:
Come on Sean, this "very few disruptions" stuff is below your usual standards. The least you can do to help us pass the time in this damn heat is to recount a few good stories about routers you could scramble eggs on. :)
On Mon, 24 Jul 2006, Richard A Steenbergen wrote: there is a funny story of some dial devices on fire, and still passing packets...
and an equally funny story of said devices being held up in customs in a particular european country because they said "TNT" on the outside of their crates...
I ordered a new personal PC back in March(?) from Lenovo (discount overstock offering). Everything shipped immediately but was delayed in transit, due to a "Live Entity" inspection hold placed on it by the US FDA. The packing list included an item identified as "mouse" (it was right under the item "keyboard"). I'm waiting for nVidia or ATI to come out with a next-gen product named "Nuclear XForce" or "Plutonium Wonder". :-) -Jim P.
On Mon, 24 Jul 2006, Robert E. Seastrom wrote:
"Christopher L. Morrow" <christopher.morrow@verizonbusiness.com> writes:
On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
Come on Sean, this "very few disruptions" stuff is below your usual standards. The least you can do to help us pass the time in this damn heat is to recount a few good stories about routers you could scramble eggs on. :)
there is a funny story of some dial devices on fire, and still passing packets...
and an equally funny story of said devices being held up in customs in a particular european country because they said "TNT" on the outside of their crates...
Now, where was youtube and portable digital camcorders back when these things were going on? This is an excellent application for all that super surveillance the DHS wants to do all the time! :) life-bloopers.com!
--On July 24, 2006 2:22:26 AM -0400 Sean Donelan <sean@donelan.com> wrote:
While its expected for individual customers to go down during power outages, usually because the customer does not have local backup power, it is less common for major web sites and co-location centers to experience downtime during power outages.
Except if you're in Qwest territory. Apparently they don't put any battery backup at their mini-DSLAMs and such. Every time we lose power, I'm still up, but the DSL signal goes away. Haven't checked dialtone, but I keep meaning too during the next outage. Now I know it's not exactly fair singling out Qwest, because I'll bet Verizon and others share the same thing, and I'm pretty sure it's just their ADSL service and not the voice service (I haven't checked though) it's still becoming more and more common that as an individual user your connection to the internet, unless you're paying for something other than ADSL or Cable, will be just as affected by local power outages.
Indeed, my RoadRunner connection is the same way. All of my stuff stays up, but "teh Interweb is broken." I'm guessing that they (DSL/CableCo's) find it too cost-prohibitive to roll out UPSes to the customer aggregation points. Suprisingly, my cable TV goes out as well when the power goes, so it might just be more than the CMTS that's going out. -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Michael Loftis Sent: Monday, July 24, 2006 16:20 To: nanog@merit.edu Subject: Re: Hot weather and power outages continue --On July 24, 2006 2:22:26 AM -0400 Sean Donelan <sean@donelan.com> wrote:
While its expected for individual customers to go down during power outages, usually because the customer does not have local backup power, it is less common for major web sites and co-location centers to experience downtime during power outages.
Except if you're in Qwest territory. Apparently they don't put any battery backup at their mini-DSLAMs and such. Every time we lose power, I'm still up, but the DSL signal goes away. Haven't checked dialtone, but I keep meaning too during the next outage. Now I know it's not exactly fair singling out Qwest, because I'll bet Verizon and others share the same thing, and I'm pretty sure it's just their ADSL service and not the voice service (I haven't checked though) it's still becoming more and more common that as an individual user your connection to the internet, unless you're paying for something other than ADSL or Cable, will be just as affected by local power outages.
While hardwired (fiber/coax/copper) aggregation points usually don't have backup power on them, most cellular towers have either batteries or generators for backup power, correct? -brandon On 7/24/06, William S. Duncanson <caesar@starkreality.com> wrote:
Indeed, my RoadRunner connection is the same way. All of my stuff stays up, but "teh Interweb is broken." I'm guessing that they (DSL/CableCo's) find it too cost-prohibitive to roll out UPSes to the customer aggregation points. Suprisingly, my cable TV goes out as well when the power goes, so it might just be more than the CMTS that's going out.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Michael Loftis Sent: Monday, July 24, 2006 16:20 To: nanog@merit.edu Subject: Re: Hot weather and power outages continue
--On July 24, 2006 2:22:26 AM -0400 Sean Donelan <sean@donelan.com> wrote:
While its expected for individual customers to go down during power outages, usually because the customer does not have local backup power, it is less common for major web sites and co-location centers to experience downtime during power outages.
Except if you're in Qwest territory. Apparently they don't put any battery backup at their mini-DSLAMs and such. Every time we lose power, I'm still up, but the DSL signal goes away. Haven't checked dialtone, but I keep meaning too during the next outage.
Now I know it's not exactly fair singling out Qwest, because I'll bet Verizon and others share the same thing, and I'm pretty sure it's just their ADSL service and not the voice service (I haven't checked though) it's still becoming more and more common that as an individual user your connection to the internet, unless you're paying for something other than ADSL or Cable, will be just as affected by local power outages.
-- Brandon Galbraith Email: brandon.galbraith@gmail.com AIM: brandong00 Voice: 630.400.6992 "A true pirate starts drinking before the sun hits the yard-arm. Yarrrr. --thelost"
On Mon, 24 Jul 2006, William S. Duncanson wrote:
Suprisingly, my cable TV goes out as well when the power goes, so it might just be more than the CMTS that's going out.
It's your neighborhood distribution amp going out, or possibly a smaller amp between the neighborhood amp and your house (...or even one at your house...). With the Digital Phone and similar rollouts, though, they've gotten better about putting battery backups on at least the neighborhood amps...they realized it was pointless to put batteries in the MTA if they don't put them on the amps..;) ...d --- david raistrick http://www.netmeister.org/news/learn2quote.html drais@atlasta.net http://www.expita.com/nomime.html
Our small operation has outfitted our Calix shelves in the field with a minimum 8 hours of run time. If they would run low we would re-charge them with portable generators. We just consider it the cost of doing business. Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Michael Loftis Sent: Monday, July 24, 2006 4:20 PM To: nanog@merit.edu Subject: Re: Hot weather and power outages continue --On July 24, 2006 2:22:26 AM -0400 Sean Donelan <sean@donelan.com> wrote:
While its expected for individual customers to go down during power outages, usually because the customer does not have local backup power, it is less common for major web sites and co-location centers to experience downtime during power outages.
Except if you're in Qwest territory. Apparently they don't put any battery backup at their mini-DSLAMs and such. Every time we lose power, I'm still up, but the DSL signal goes away. Haven't checked dialtone, but I keep meaning too during the next outage. Now I know it's not exactly fair singling out Qwest, because I'll bet Verizon and others share the same thing, and I'm pretty sure it's just their ADSL service and not the voice service (I haven't checked though) it's still becoming more and more common that as an individual user your connection to the internet, unless you're paying for something other than ADSL or Cable, will be just as affected by local power outages.
participants (12)
-
Brandon Galbraith -
Christopher L. Morrow -
david raistrick -
Frank Bulk -
Jim Popovitch -
Laurence F. Sheldon, Jr. -
Michael Loftis -
Richard A Steenbergen -
Robert E.Seastrom -
Sam Stickland -
Sean Donelan -
William S. Duncanson