POWER: San Mateo/San Francisco power outage report
Howdy, I'm back from vacation. By popular demand, a summary of the San Francisco/San Mateo power outage on December 8, 1998. At 8:15am workers making changes to 115-kilovolt lines at the San Mateo substation failed to properly ground the system. When they turned on the power, it caused a low frequency condition and tripped protective breakers on the substation. It also caused a cascade effect, tripping the Hunters Point and Potrero power plants. This isolated San Mateo County and northward to parts of San Francisco City from the electrical grid primary power sources. Aside from the human error, the system worked "as designed to protect the rest of the grid." Approximately 375,000 customers were without power, including San Francisco Airport (SFO), the Pacific Stock Exchange, traffic lights, and cable cars. BART (subway) used emergency power to bring trains to the next station, and then shutdown. Since the electrical grid was isolated, PG&E had to "blackstart" the system on a circuit by circuit basis. The SFO airport power was restored at 9:10AM. The San Francisco Office of Emergency Services was activated at 9:22AM according to EDIS. At 1:00PM about 200,000 customers were still without power. PG&E estimated that full restoration would be completed by 2:15PM, but some customers were still without power through the afternoon. No major injuries have been reported directly attributable to the power failure. No major Internet provider reported any network problems directly attributable to the power failure. Since this outage was caused by a PG&E error, customers may make claims for some types of damages. Other damages may be covered by private insurance. http://www.pge.com/resources/claim_form/claim_form.html In other news: On December 7, 1998 PG&E donated $87,000 to San Fransico's Emergency Services Communications Center to install extra back up electrical facilities for the center. The new facility will work with PG&E during major service interruptions to establish restoration priorities. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation
breakers on the substation. It also caused a cascade effect, tripping the Hunters Point and Potrero power plants.
grid primary power sources. Aside from the human error, the system worked "as designed to protect the rest of the grid."
I suggest that causing a cascade effect and increasing the area of an outage isn't a good way to design a system.
On Wed, 9 Dec 1998, Jon Zeeff wrote:
breakers on the substation. It also caused a cascade effect, tripping the Hunters Point and Potrero power plants.
grid primary power sources. Aside from the human error, the system worked "as designed to protect the rest of the grid."
I suggest that causing a cascade effect and increasing the area of an outage isn't a good way to design a system.
I would agree, that is why I am worried about year 2K power issues. I think for the most part everything is ok with most of the larger providers, but I have talked to several smaller energy providers who will not be ready for y2k. The systems that may/will (depending on who you talk to) fail are normally not that big of a deal to restart after they trip. The problem is that this could cause a cascading failure that would take down most of the national grid. When you have something like that happen it is not that easy to bring everything back on-line. As far as I know we have never had the entire national grid fail, we have had large sections (say New York, 13 north western states, etc.) of the grid fail because of cascading failures that were caused by very small problems. If you are a ISP, you better have your generators ready, I think you are going to need to use them. -- Check out the new CLEC mailing list at http://www.robotics.net/clec
<> Nathan Stratton Telecom & ISP Consulting www.robotics.net nathan@robotics.net
Unnamed Administration sources reported that Nathan Stratton said:
As far as I know we have never had the entire national grid fail, we have had large sections (say New York, 13 north western states, etc.) of the grid fail because of cascading failures that were caused by very small problems.
That is because there really is NO "national grid"..... I'm not a power engineer, but my understanding is there are 6-12 regional grids. Keeping things in sync across the country qualifies as "really hard" at the least. Some regionals are interconnected with DC {The Pacific Intertie, for one} lines for this very reason. This topic at least, I assume, that presidential infrastructure protection committee has looked at in detail. I have far more faith they can grok what to do there than with BGP grief/router flaps/etc. This is topic creeping so I'll close with one thought. Power is hard because: unlike TCP/IP bits, when things fall down, you gotta get RID of the stuff blowing back in your face NOW, or lots of things get turned to smoke. Suppose that when your big box of bits stutters; the box feeding you in LA didn't just stop, but halted & caught fire...and that meant YOUR box, that was picking itself back up, just did likewise. -- A host is a host from coast to coast.................wb8foz@nrk.com & no one will talk to a host that's close........[v].(301) 56-LINUX Unless the host (that isn't close).........................pob 1433 is busy, hung or dead....................................20915-1433
The North American Electric Reliability Council has a good deal of info online at http://www.nerc.com In particular, NERC's Y2K report to the DOE is available at: ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/y2kreport-doe.pdf
On Wed, Dec 09, 1998 at 11:19:46AM -0500, David Lesher wrote:
This is topic creeping so I'll close with one thought. Power is hard because: unlike TCP/IP bits, when things fall down, you gotta get RID of the stuff blowing back in your face NOW, or lots of things get turned to smoke. Suppose that when your big box of bits stutters; the box feeding you in LA didn't just stop, but halted & caught fire...and that meant YOUR box, that was picking itself back up, just did likewise.
Hint: AT&T, SS7, 1990. Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Buy copies of The New Hackers Dictionary. The Suncoast Freenet Give them to all your friends. Tampa Bay, Florida http://www.ccil.org/jargon/ +1 813 790 7592
On Wed, Dec 09, 1998 at 06:57:54PM -0500, Jay R. Ashworth wrote:
On Wed, Dec 09, 1998 at 11:19:46AM -0500, David Lesher wrote:
This is topic creeping so I'll close with one thought. Power is hard because: unlike TCP/IP bits, when things fall down, you gotta get RID of the stuff blowing back in your face NOW, or lots of things get turned to smoke. Suppose that when your big box of bits stutters; the box feeding you in LA didn't just stop, but halted & caught fire...and that meant YOUR box, that was picking itself back up, just did likewise.
Hint: AT&T, SS7, 1990.
David Lesher has kindly pointed out to me that I had a backhoe fade of the brain shortly before posting this message. Nevermind. Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Buy copies of The New Hackers Dictionary. The Suncoast Freenet Give them to all your friends. Tampa Bay, Florida http://www.ccil.org/jargon/ +1 813 790 7592
I just wanted to clarify something, I got a few email messages from people who said I was one of the nuts who is saying the world would fall apart Y2K, I think everything will work out. I do not recommend anyone fill baggies full of dried food, buy a gun, or build a house underground. I think we will get though this ok. I was just trying to point out that some of the power systems in the US are very fragile and that it does not take a lot to bring the system down. My recommendation on having your generator ready is something all ISPs should do no matter what year it is. I do think we will have power outages, but the systems WILL come back online and thinks WILL get back to normal. This is a operational issue and we need to make sure our systems are protected. -- Check out the new CLEC mailing list at http://www.robotics.net/clec
<> Nathan Stratton Telecom & ISP Consulting www.robotics.net nathan@robotics.net
On Wed, 9 Dec 1998, Nathan Stratton wrote:
I would agree, that is why I am worried about year 2K power issues. I think for the most part everything is ok with most of the larger providers, but I have talked to several smaller energy providers who will not be ready for y2k. The systems that may/will (depending on who you talk to) fail are normally not that big of a deal to restart after they trip. The problem is that this could cause a cascading failure that would take down most of the national grid. When you have something like that happen it is not that easy to bring everything back on-line. As far as I know we have never had the entire national grid fail, we have had large sections (say New York, 13 north western states, etc.) of the grid fail because of cascading failures that were caused by very small problems.
If you are a ISP, you better have your generators ready, I think you are going to need to use them.
-- Check out the new CLEC mailing list at http://www.robotics.net/clec
<> Nathan Stratton Telecom & ISP Consulting www.robotics.net nathan@robotics.net
On Wed, 9 Dec 1998, Nathan Stratton wrote:
I just wanted to clarify something, I got a few email messages from people who said I was one of the nuts who is saying the world would fall apart Y2K, I think everything will work out. I do not recommend anyone fill baggies full of dried food, buy a gun, or build a house underground. I think we will get though this ok.
While I fully agree with Nathan, having appropriate supplies around to be able to survive a week or so without power or heat or water or being able to buy food is probably a good thing. Regardless if it is January 1, 2000 or not. - Forrest W. Christian (forrestc@imach.com) ---------------------------------------------------------------------- iMach, Ltd., P.O. Box 5749, Helena, MT 59604 http://www.imach.com Solutions for your high-tech problems. (406)-442-6648 ----------------------------------------------------------------------
Network Engineering != Power Grid/Generation Engineering. What you know about building a network does not apply to building a power generation and distribution system. Imagine what you might design if a large BGP flap could physically destroy a POP. A small variance in voltage or frequency can cause significant damage not only to the transmission and generation facilities, but also to end user equipment. When you lose or gain significant load on a distribution system, it causes the frequency to rise or fall. When the frequency varies significantly from 60Hz, you begin to catastrphically destroy generator sets and sensitive transmission gear. So, trip circuits are set to identify a frequency change that is significant and to trip the gen sets off-line. At this point if you don't isolate the problem, it becomes a domino efffect. The load generating capacity keeps dropping as the demand remains the same. The only option is to isolate the failed grid area, and then slowly bring the grid back on-line bit by bit. -Chris At 09:04 AM 12/9/98 -0500, Jon Zeeff wrote:
breakers on the substation. It also caused a cascade effect, tripping the Hunters Point and Potrero power plants.
grid primary power sources. Aside from the human error, the system worked "as designed to protect the rest of the grid."
I suggest that causing a cascade effect and increasing the area of an outage isn't a good way to design a system.
Chris A. Icide / chris@skycache.com VP Engineering/Operations SkyCache / www.skycache.com (v) 301-598-0500 x2235
On Wed, Dec 09, 1998 at 09:04:27AM -0500, Jon Zeeff wrote:
grid primary power sources. Aside from the human error, the system worked "as designed to protect the rest of the grid."
I suggest that causing a cascade effect and increasing the area of an outage isn't a good way to design a system.
I suggest that you don't understand power distribution and transmission very well. Just like any other generator, if you load a power station too heavily, the voltage and the frequency will both be pulled down. Since the tolerances on public electricity are something like +-5v and +- .5 Hz., they really have no choice but to break ties when a plant or segment goes out. They really shouldn't cross-tie power on a running basis as much as they probably do... but since the Green folks won't always let them build plants... Cheers, -- jr 'Damn I'm glad I don't run a power company...' a -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Buy copies of The New Hackers Dictionary. The Suncoast Freenet Give them to all your friends. Tampa Bay, Florida http://www.ccil.org/jargon/ +1 813 790 7592
Sean, just a minor note ont he initial cause. They grounded the system just fine, then just forgot to pull the safety grounds before they repowered the line. As for "power grid outages shouldn't spread", there's a rule in that biz, it's always easier/faster to reset 10 circuits that replace something. If you leave a turbine generator up in a situation where output load is whacked, you could be looking at a several week outage. The reason it took them so long to get the plants back on line is they have to do a complete system inspection to make sure the breakers tripped in time. One other thing to remember, there isn't a whole lot of risk working on a router outage. When you're restoring electric service, you have to go slower so you keep the possibility of frying a lineman to a minimum. I spent a few weeks installing power line electronic equipment in a 115kv substation and got a whole new level of respect. You couldn't pay me enough to do that work (but I like their trucks.) jerry
On Wed, Dec 09, 1998 at 07:21:18AM -0800, Jerry Scharf wrote:
One other thing to remember, there isn't a whole lot of risk working on a router outage. When you're restoring electric service, you have to go slower so you keep the possibility of frying a lineman to a minimum. I spent a few weeks installing power line electronic equipment in a 115kv substation and got a whole new level of respect. You couldn't pay me enough to do that work (but I like their trucks.)
jerry
Yeah, your first mistake around that kind of energy is almost always your last. I used to a lot of work with embedded control systems in very-high-power microwave transmitter gear (ie: 25KW stuff). Those things run around with several tens of KV at ~2A (yes, that's AMPS) in the HV drawer. Even "pedestrian" HPA hardware (3kw Cband stuff) has a couple hundred ma in the HV drawer, which at 20kv is more than enough to fry you very dead. If you manage to route that kind of energy through yourself (and it WILL jump air gaps at that voltage) they don't even bother with paramedics - just call the coroner. That ignores the risk from the emitted microwave energy itself, which is substantial as well (directly in the front of the antenna the ERP of these things can be in the range of a couple of MW - more than enough to cause you all kinds of physiological trouble (like death)) And that's NOTHING compared to the energy levels running around in circuits at 115kv substations! -- -- Karl Denninger (karl@denninger.net) http://www.mcs.net/~karl I ain't even *authorized* to speak for anyone other than myself, so give up now on trying to associate my words with any particular organization.
participants (10)
-
Chris A. Icide -
David Lesher -
Forrest W. Christian -
Jay R. Ashworth -
Jerry Scharf -
jzeeff＠verio.net -
Karl Denninger -
Mark Boolootian -
Nathan Stratton -
Sean Donelan