As it finally cools down, and things appear to be back online, here on the late coast... According to CAISO's website, the state-wide grid only reached a "Stage One Electrical Emergency", and no stats that I can find detail the Bay area other than the activation of voluntary interrupt disconnections. According to both PG&E and the media, a Stage Three emergency was declared in the afternoon, resulting in rolling blackouts for 1.5 to 2 hours per block section. The Palo Alto and Santa Clara grids (separate from PG&E) also went to a Stage Three emergency state. All press releases and reports indicate that this was done at the request of CAISO. Cupertino and possibly some of Sunnyvale around De Anza, from Stevens Creek to I-85, including the Apple campus, were under blackout from 13:30 to 15:30. This was before the media informed people of rolling blackouts, and I have not yet heard a clear answer as to whether this was scheduled (though the nature of the timing, and it's coincidence with later blackouts, suggests that it may have been the first block outage). There was one unconfirmed report of an outage of unknown scope, area and duration occuring at 14:30. Many confirmed reports of widespread outages began to come in at roughly 15:00, from the following areas: San Jose, Santa Clara, Palo Alto, Foster City (no confirmed outage, only warnings), Fremont, the China Basin, and parts of San Francisco. At least one major data center disconnected from the grid and went to generator power, prior to the rolling blackouts. The motivation was not stated, but it seems likely that they were requested to do so as part of the voluntary interruption efforts. No interruption of service was seen. 2 ISP failures were reported; no explicit "ISP alive and well" notices came, but the number of people affected by the area, and still able to transit traffic, would indicate a fair number of service providers still up and flinging packets hither and yon. Sean Donelan was not heard from, and we're all hoping the power gnomes didn't get him :) Tomorrows peak power drain is predicted to be somewhat *above* todays peak was predicted for - so, barring a miracle cooling effect in the Bay, you can probably expect more of the same sort of thing tomorrow. Probably with less folks posting about it - unless/until someone's supply gives out and something big falls over, anyway. -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://www.lightbearer.com/~lucifer
At 22:46 -0700 6/14/00, lucifer@lightbearer.com wrote:
There was one unconfirmed report of an outage of unknown scope, area and duration occuring at 14:30.
Downtown Sunnyvale around Carroll Street (including PacBell SNVCA01 and us) lost power (on one phase only) yesterday at 12:35 PDT. PG&E brought it back at 15:35. PG&E said that our area is using too much power and we blew the breaker on that phase. The odd thing was we were still getting about 27 VAC on that phase with the breaker "blown". Of course, the problem started again at 09:21 PDT and is ongoing. Generators (tested weekly) are your friend. I'm sure burning gas does wonders to help out the "Spare the Air" effort. I wonder when CA will require stoplights to have battery backup. I imagine that is more likely to happen than having intelligent drivers. We had one ambulance roll yesterday due to the "if the stoplight is not lit, hit the gas!" effect. Jim Browne jbrowne@jbrowne.com "Reading these menus is like trying to read an article in Wired." - Dan Pape on the menus at El Balazo, Haight Street, SF
When you lose a part of a multi-phase circuit, you usually get bleed through enduser devices that are still getting power through the other phases. Your 27VAC was probably bleen through a (or more likely several) three-phase motors that did not have phase-outage protection circuit breakers. They probably had stopped spinning and were slowly cooking themselves. A building I once worked at lost one of their phases and after an hour the fire alarm went off and we had to evacuate the whole 10 stories. A "temporary" compressor in a design model shop did'nt have adequate protection and cooked all it's lubrication, filling the shop with smoke. Apparently, phase-outage protection is not required in many installations where it should be. One data-center in the same building still had some three phase powered mini-computers (mini in name only) and they lost almost half their hardware. JMH Jim Browne wrote:
Downtown Sunnyvale around Carroll Street (including PacBell SNVCA01 and us) lost power (on one phase only) yesterday at 12:35 PDT. PG&E brought it back at 15:35. PG&E said that our area is using too much power and we blew the breaker on that phase. The odd thing was we were still getting about 27 VAC on that phase with the breaker "blown".
-- John Hall <j.hall@f5.com> F5 Networks, Inc. Senior Test Engineer 206-505-0800 One of the large consolations for experiencing anything unpleasant is the knowledge that one can communicate it. -- Joyce Carol Oates
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 2000-06-16-15:40:11 John Hall:
When you lose a part of a multi-phase circuit, you usually get bleed through enduser devices that are still getting power through the other phases.
Your story is a lot flashier than mine, but I had a really memorable experience with 3-phase bleedover myself, once upon a time. I was working at a research lab in the Duke Medical Center, in a small building they called the Old Laundry Building, because it was. It had a really ancient and heavy-duty building power distribution panel; it had several 3-phase circuits running various big items; and it had of course loads of regular power circuits. Including our machine room. Well, the building power panel breaker for the leg that carried our machine room died. Actually I think it might have been a gigantic fuse that blew, it's been so long I don't remember. Anyway, our machine room was suddenly being powered by bleedthrough. Turns out those 3-phase-powered appliances had a pretty darned significant impedance, so our voltage available was suddenly remarkably sensitive to how much load we applied to it. Unload the circuits and the available voltage rose right up to about nominal, comfortably over 110VAC or so; load everything on and the voltage dropped down to something way way too low, like 40-50VAC or some such, again the details are lost to me, I'm sorry. Well, much of our machine room load was things like big SMD drives - --- Fujitsu Eagles, Hitachi DK-815s, and the like. Weighed a ton, drank power, and had sufficiently smart power supplies to take themselves offline and spin down when the power dropped too far. At which point the available power climbed up again, and they came on again. They were cycling up and down every minute or so until we managed to shut everything off. I think we only lost a couple of power supplies and one HDA, if I recall correctly. Gear sure was tougher then. Today I'd expect that kind of silly-buggers games to let the magic smoke out of every box in a machine room, with the possible rare exception of something indestructable like a 2511, too mean to die. And of course laptops, which ties in nicely to another thread I've committed hereabouts:-). - -Bennett -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.0 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE5SpR8L6KAps40sTYRAjx2AJ9cFyczUkWqnDKbd0Mv6H52TMg4NgCaAr9O IEhUZyKjwQshKGd2QnDfw+c= =KRwh -----END PGP SIGNATURE-----
lucifer@lightbearer.com writes:
At least one major data center disconnected from the grid and went to generator power, prior to the rolling blackouts. The motivation was not stated, but it seems likely that they were requested to do so as part of the voluntary interruption efforts. No interruption of service was seen.
In a conversation with an MFS engineer today I mentioned the problems in the Bay area. He previously worked for SNET (Southern New England Telephone, now part of SBC) for many years and said it was standard practice for them to disconnect COs from the grid and run off generator power for hours or even a day or two during the summer season. I was under the impression that these power shortages were the result of less over-engineering on the part of power companies due to deregulation but maybe this isn't such a new situation after all?
The PG&E claim was they did not have the capacity. I think the problem is more like not having the manpower to bring up extra capacity at peak points, they can't even fix the local stuff for the residential areas and as of last night 13,000 people still had no power since Tuesday. We believe this was really retaliation for them not gettting their rate increase from the PUC and quite frankly people are not to happy about this type of behavior from a business. gwright@thebiz.net wrote:
lucifer@lightbearer.com writes:
At least one major data center disconnected from the grid and went to generator power, prior to the rolling blackouts. The motivation was not stated, but it seems likely that they were requested to do so as part of the voluntary interruption efforts. No interruption of service was seen.
In a conversation with an MFS engineer today I mentioned the problems in the Bay area. He previously worked for SNET (Southern New England Telephone, now part of SBC) for many years and said it was standard practice for them to disconnect COs from the grid and run off generator power for hours or even a day or two during the summer season.
I was under the impression that these power shortages were the result of less over-engineering on the part of power companies due to deregulation but maybe this isn't such a new situation after all?
-- Thank you; |--------------------------------| | Thinking is a learned process. | | ICANN member @large | | Gigabit over IP, ieee 802.17 | |--------------------------------| Henry R. Linneweh
participants (6)
-
Bennett Todd -
gwright＠thebiz.net -
Henry R. Linneweh -
Jim Browne -
John Hall -
lucifer＠lightbearer.com