San Francisco power outages imminent
I've seen a few messages forwarded from friends at various SF companies, especially South of Market, saying that PG&E has been calling them and asking that they reduce power utilization. Word is they're not capable of handling the current demand (it is the second day of what feels like a serious heat wave, with temperatures over 90 degrees all over the Bay Area), and power may be shut off on a rotating basis, a block or a few blocks at a time, if the load cannot be reduced. There're a lot of "big name" companies in that area, including Looksmart, Critical Path, Organic Online, Topica, and many, many more. Not sure how many of them have production servers in/near their offices, but it's not uncommon for startups (or recently-past-startup-phase companies) to stupidly have their production machines rely on being able to exchange data with their offices, so who knows what might happen. Mmm, fun! Any word on how San Jose is holding up? Got a lot of data- centers there.... -- J.D. Falk "Practice random funk and senseless disco moves." <jdfalk@cybernothing.org> -- Koz
On Wed, 14 Jun 2000, J.D. Falk wrote:
I've seen a few messages forwarded from friends at various SF companies, especially South of Market, saying that PG&E has been calling them and asking that they reduce power utilization.
Any word on how San Jose is holding up? Got a lot of data- centers there....
i was told that san jose was hearing the same thing.
J.D. Falk wrote:
Any word on how San Jose is holding up? Got a lot of data- centers there....
Similar reports coming from folks in downtown SJ, but no confirmed outages at just shy of 15:00 PDT. (Either because they haven't occured to the folks I have contact with, or because they can't tell me due to their servers going offline...) -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://www.lightbearer.com/~lucifer KF6WAY (Tech) - 146.475 MHz (FM/Phone)
I heard from some that the outages would begin around 15:00 PDT. Apparantly it was on the radio this afternoon. -j On Wed, Jun 14, 2000 at 02:50:05PM -0700, lucifer@lightbearer.com wrote:
J.D. Falk wrote:
Any word on how San Jose is holding up? Got a lot of data- centers there....
Similar reports coming from folks in downtown SJ, but no confirmed outages at just shy of 15:00 PDT. (Either because they haven't occured to the folks I have contact with, or because they can't tell me due to their servers going offline...) -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://www.lightbearer.com/~lucifer KF6WAY (Tech) - 146.475 MHz (FM/Phone)
-- /--------------------------/ Jason Legate \------------------------\ | jlegate@sitesmith.com | SiteSmith, Inc. | | 24x7 Call Center | http://www.sitesmith.com | | 888.898.7667 | PGP Key ID - 0xA855AAC3 | +---------------------------------+--------------------------------+ | Fingerprint - 2D5F 87A0 26E6 A65B 6837 D100 FB54 A972 A855 AAC3 | \------------------------------------------------------------------/
Jason Legate wrote:
I heard from some that the outages would begin around 15:00 PDT. Apparantly it was on the radio this afternoon.
And, as of 15:05, at least one major data center has gone off-grid and switched to generators... (Name withheld to allow their own folks to say it, since it's not a disruption of service) Less confirmable reports of outages in Cupertino, and a number of buildings voluntarily off-grid in downtown SJ, ranging up to South SF. -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://www.lightbearer.com/~lucifer KF6WAY (Tech) - 146.475 MHz (FM/Phone)
I don't know if you refer to the same place, but De Anza in Cupertino is shut off from Stevens Creek almost all the way to 85.
Less confirmable reports of outages in Cupertino, and a number of buildings voluntarily off-grid in downtown SJ, ranging up to South SF.
Thus far, we are still ok over here in Santa Clara (we have that separate grid), but I don't know if that will last. -- /--------------------------/ Jason Legate \------------------------\ | jlegate@sitesmith.com | SiteSmith, Inc. | | 24x7 Call Center | http://www.sitesmith.com | | 888.898.7667 | PGP Key ID - 0xA855AAC3 | +---------------------------------+--------------------------------+ | Fingerprint - 2D5F 87A0 26E6 A65B 6837 D100 FB54 A972 A855 AAC3 | \------------------------------------------------------------------/
Yo All! The imon.com data center in Milpitas is on backup power as well. RGDFS GARY On Wed, 14 Jun 2000 lucifer@lightbearer.com wrote:
Jason Legate wrote:
I heard from some that the outages would begin around 15:00 PDT. Apparantly it was on the radio this afternoon.
And, as of 15:05, at least one major data center has gone off-grid and switched to generators... (Name withheld to allow their own folks to say it, since it's not a disruption of service)
Less confirmable reports of outages in Cupertino, and a number of buildings voluntarily off-grid in downtown SJ, ranging up to South SF.
--------------------------------------------------------------------------- Gary E. Miller Rellim 20340 Empire Ave, Suite E-3, Bend, OR 97701 gem@rellim.com Tel:+1(541)382-8588 Fax: +1(541)382-8676
On Wed, Jun 14, 2000 at 03:01:40PM -0700, Jason Legate wrote:
I heard from some that the outages would begin around 15:00 PDT. Apparantly it was on the radio this afternoon.
-j
On Wed, Jun 14, 2000 at 02:50:05PM -0700, lucifer@lightbearer.com wrote:
J.D. Falk wrote:
Any word on how San Jose is holding up? Got a lot of data- centers there....
Similar reports coming from folks in downtown SJ, but no confirmed outages at just shy of 15:00 PDT. (Either because they haven't occured to the folks I have contact with, or because they can't tell me due to their servers going offline...) --
Sitting in Foster City and we also got warned around 3pm PT and then again at 3:30pm PT. Its 5pm now, nothing has happen so far. But yes, its hot and I expect something to happen (*pat diesel generators*) -- Regards, Ulf. --------------------------------------------------------------------- Ulf Zimmermann, 1525 Pacific Ave., Alameda, CA-94501, #: 510-769-2936 Alameda Networks, Inc. | http://www.Alameda.net | Fax#: 510-521-5073
On 06/14/00, "J.D. Falk" <jdfalk@cybernothing.org> wrote:
Word is they're not capable of handling the current demand (it is the second day of what feels like a serious heat wave, with temperatures over 90 degrees all over the Bay Area), and power may be shut off on a rotating basis, a block or a few blocks at a time, if the load cannot be reduced.
Okay, just received word that they're doing it: rolling black- outs for 90 minutes at a time, around the Bay Area (not just in the city.) http://www.sfgate.com/cgi-bin/article.cgi?file=/gate/archive/2000/06/14/powe... It'll be interesting to see what kind of outages this causes; for one thing, Sean's power may go out. If power goes out in the NOC and there's nobody around to monitor it.... -- J.D. Falk "Laughter is the sound Product Manager that knowledge makes when it's born." Mail Abuse Prevention System LLC -- The Cluetrain Manifesto
It'll be interesting to see what kind of outages this causes; for one thing, Sean's power may go out.
If power goes out in the NOC and there's nobody around to monitor it....
http://www.caiso.com/awe/systemstatus.html is pretty interesting reading at this point; so is CAISO document E-509, at the bottom of http://www.caiso.com/thegrid/operations/opsdoc/emergency/ , which describes what a "No Touch Day" actually is. I'm beginning to see how Sean gets so much inside info; all it takes is to post to NANOG a bit, and people tell you all sorts of things. For example, someone from Exodus just told me that they "haven't yet made any changes." One would hope that this is because they have more than 90 minutes of backup power. Some of the downtown San Jose office space (such as the Bank of America building) is being "quiesced" for the rest of the day, because it is deemed non-critical. -- J.D. Falk "Laughter is the sound Product Manager that knowledge makes when it's born." Mail Abuse Prevention System LLC -- The Cluetrain Manifesto
Think of it as an acid test of everyone's backup systems. :-) More than one has failed due to a bad battery, out of feul, failures in the UPS whilst the generator kicks in.. and, of course, because there was no generator or ups to begin with..
On 06/14/00, "J.D. Falk" <jdfalk@cybernothing.org> wrote:
Word is they're not capable of handling the current demand (it is the second day of what feels like a serious heat wave, with temperatures over 90 degrees all over the Bay Area), and power may be shut off on a rotating basis, a block or a few blocks at a time, if the load cannot be reduced.
Okay, just received word that they're doing it: rolling black- outs for 90 minutes at a time, around the Bay Area (not just in the city.)
http://www.sfgate.com/cgi-bin/article.cgi?file=/gate/archive/2000/06/14/powe...
It'll be interesting to see what kind of outages this causes; for one thing, Sean's power may go out.
If power goes out in the NOC and there's nobody around to monitor it....
-- J.D. Falk "Laughter is the sound Product Manager that knowledge makes when it's born." Mail Abuse Prevention System LLC -- The Cluetrain Manifesto
---------------------------------------------------------------------- Wayne Bouchard [Immagine Your ] web@typo.org [Company Name Here] Network Engineer http://www.typo.org/~web/resume.html ----------------------------------------------------------------------
On Wed, Jun 14, 2000 at 03:38:35PM -0700, Wayne Bouchard wrote:
Think of it as an acid test of everyone's backup systems. :-)
More than one has failed due to a bad battery, out of feul, failures in the UPS whilst the generator kicks in.. and, of course, because there was no generator or ups to begin with..
I'm not in SJ, but we recently had yet another power outage, and it was fun watching the fear in the building's maintenance manager's eyes when he said "well, you've got a full load of fuel" and our hardware manager replied "do you know that, or are you guessing?" Heh heh.
On Wed, Jun 14, 2000 at 03:38:35PM -0700, Wayne Bouchard wrote:
Think of it as an acid test of everyone's backup systems. :-)
More than one has failed due to a bad battery, out of feul, failures in the UPS whilst the generator kicks in.. and, of course, because there was no generator or ups to begin with..
I'm not in SJ, but we recently had yet another power outage, and it was fun watching the fear in the building's maintenance manager's eyes when he said "well, you've got a full load of fuel" and our hardware manager replied "do you know that, or are you guessing?"
Heh heh.
Actually, thats another question I am somewhat interested in.. what are the policies of various people on testing and excercising generators and other backup equipment? How many customers ask about such things before locating equipment vs just asking "do you have backup power?" ---------------------------------------------------------------------- Wayne Bouchard [Immagine Your ] web@typo.org [Company Name Here] Network Engineer http://www.typo.org/~web/resume.html ----------------------------------------------------------------------
On Wed, Jun 14, 2000 at 06:30:32PM -0700, Wayne Bouchard wrote:
Actually, thats another question I am somewhat interested in.. what are the policies of various people on testing and excercising generators and other backup equipment? How many customers ask about such things before locating equipment vs just asking "do you have backup power?"
We are very conscious about it, although at some of our data centers we have to rely upon outside folks. In one instance, an individual was signing off every week that he checked the switch, but it turned out that for months he was just signing the sheet and not actually testing anything. In his new job the only thing he has to check is whether or not the customer would like fries. :-) We're building a new main data center that is tornado-proof. Nothing short of a nuke should be able to cause us power issues. Unless Sean Donelan gets ahold of a fusion-powered backhoe...
> Actually, thats another question I am somewhat interested in.. what > are the policies of various people on testing and excercising > generators and other backup equipment? We run our gensets for five minutes on line gas, and five minutes on tank gas, each week, and test a hot cut-over (exercising the batteries and transfer switches as well) once a month. > How many customers ask about > such things before locating equipment vs just asking "do you have > backup power?" Only once that I can recall. -Bill
Sez "J.D. Falk" <jdfalk@mail-abuse.org>
It'll be interesting to see what kind of outages this causes; for one thing, Sean's power may go out.
I'd be disappointed in him if it did.
If power goes out in the NOC and there's nobody around to monitor it....
Thats what batteries and generators are for, right? You did plan for power outages at all of your mission-critical facilities, didn't you? I wonder how many of us know the "major data center" that Joel Baker referred to being on generators :) S | | Stephen Sprunk, K5SSS, CCIE #3723 :|: :|: Network Design Consultant, HCOE :|||: :|||: 14875 Landmark Blvd #400; Dallas, TX .:|||||||:..:|||||||:. Email: ssprunk@cisco.com
Power outages today in San Jose and Cupertino starting at 0100pm PDT I was back at work and didn't notice when it ended. "J.D. Falk" wrote:
I've seen a few messages forwarded from friends at various SF companies, especially South of Market, saying that PG&E has been calling them and asking that they reduce power utilization.
Word is they're not capable of handling the current demand (it is the second day of what feels like a serious heat wave, with temperatures over 90 degrees all over the Bay Area), and power may be shut off on a rotating basis, a block or a few blocks at a time, if the load cannot be reduced.
There're a lot of "big name" companies in that area, including Looksmart, Critical Path, Organic Online, Topica, and many, many more. Not sure how many of them have production servers in/near their offices, but it's not uncommon for startups (or recently-past-startup-phase companies) to stupidly have their production machines rely on being able to exchange data with their offices, so who knows what might happen.
Mmm, fun!
Any word on how San Jose is holding up? Got a lot of data- centers there....
-- J.D. Falk "Practice random funk and senseless disco moves." <jdfalk@cybernothing.org> -- Koz
-- Thank you; |--------------------------------| | Thinking is a learned process. | | ICANN member @large | | Gigabit over IP, ieee 802.17 | |--------------------------------| Henry R. Linneweh
participants (12)
-
Bill Woodcock -
Christian Nielsen -
Gary E. Miller -
Henry R. Linneweh -
J.D. Falk -
J.D. Falk -
Jason Legate -
lucifer＠lightbearer.com -
Shawn McMahon -
Stephen Sprunk -
Ulf Zimmermann -
Wayne Bouchard