- NANOG - lists.nanog.org

RE: California regulators ordered rolling blackouts
by Roeland Meyer 18 Jan '01

18 Jan '01

If you've every worked for a telco, you'd know all about "affiliate transaction" and how PG&E can't touch that debt with anything other than cash, from the other bucket. The CA PUC would squash them like a bug. The real problem is CalISO management. > -----Original Message----- > From: hardie(a)equinix.com [mailto:hardie@equinix.com] > Sent: Wednesday, January 17, 2001 3:12 PM > To: akyol(a)akyol.org > Cc: nanog(a)merit.edu; rusty(a)hodge.com > Subject: Re: California regulators ordered rolling blackouts > > > > http://www.sfgate.com/cgi-bin/article.cgi?file=/chronicle/arch > ive/2001/01/16/MN184364.DTL > > describes one of the more interesting side notes in all of > this, how PG & E's > corporate restructuring allows them to retain profits. Among > the many comical > notes is that about half of the debt PG & E owes is to itself. > > regards, > Ted Hardie > > > > > > > > Even third world countries aren't as idiotic as California. > There when > > electricity goes off, it usually is because there is none > to be had. In our > > situation, it is manipulation and poor planning to the extreme. > > > > My power was out for at least one hour, it is back now. > > > > Bora > > > > ----- Original Message ----- > > From: "Rusty H. Hodge" <rusty(a)hodge.com> > > To: <nanog(a)merit.edu> > > Sent: Wednesday, January 17, 2001 1:37 PM > > Subject: California regulators ordered rolling blackouts > > > > > > > > > > It has begun. Welcome to the 3rd world. > > > > > > > > > http://www.sfgate.com/cgi-bin/article.cgi?file=/news/archive/2 > 001/01/17/stat > > e1503EST0161.DTL > > > > > > Blackouts hit California as utility financial woes deepen > > > > > > Associated Press, SF Gate Wednesday, January 17, 2001 > > > Breaking News Sections > > > > > > > -------------------------------------------------------------- > ---------- > > > > > > California regulators ordered rolling blackouts Wednesday for the > > > first time in the state's months-long electricity crisis, blaming > > > utility credit problems and a tight national power supply for the > > > scattered outages. > > > > > > The rotating blackouts, expected to affect about 500,000 customers > > > for an hour to 90 minutes, were restricted to Pacific Gas and > > > Electric Co. territory in northern and part of central California, > > > said Stephanie McCorkle, a spokeswoman for the Independent System > > > operator, keeper of the state's power grid. > > > > > > A PG&E spokesman said that about 250,000 people were > already without > > > power from the blackouts that began at 11:41 a.m. There > are reports > > > of outages in San Francisco's Lower Haight, Oakland's > Rockridge area, > > > the Oakland Hills, Orinda, the Peninsula from South San > Francisco to > > > San Mateo, parts of Napa and Sonoma County, downtown San Jose, > > > Cupertino's De Anza College, San Ramon, Santa Cruz, > Benicia and other > > > areas, according to radio and TV reports. > > > > > > There was a report of two students being trapped in an elevator at > > > Hastings School of Law in San Francisco. Some ATMs were > reported out > > > of service in downtown San Francisco. > > > > > > Power was cut to the Cow Palace in Daly City, which is > holding a boat > > > show, according to an administrative worker at the > exhibition hall. > > > The main building remains well-lit from emergency lighting and > > > skylights, she said. > > > > > > KICU-TV, channel 36, was knocked off the air. > > > > > > The blackouts would first affect customers in scattered > areas known > > > as blocks 3 and 4. For security reasons, the precise locations are > > > not released. Consumers can find their block numbers at > the bottom of > > > their power bills. > > > > > > PG&E and state officials urged conservation and said it > was not known > > > yet if blackouts would be ordered for blocks 5, 6 and 7. > Blocks 1 and > > > 2 had power outages last June. > > > > > > BART, fire departments, police stations and hospitals are not > > > affected. Motorists who come to non-functioning traffic signals, > > > including El Camino Real in the San Bruno/San Mateo area > and Lawrence > > > Expressway in San Jose, should treat them as four-way stops. > > > > > > Utilities try to avoid cutting power to blocks with essential > > > services such as hospitals. > > > > > > Terry Winter, president of Cal-ISO, said that a large > power plant on > > > California's Central Coast went down at about 11 a.m., > necessitating > > > the outages. > > > > > > Worry that the state's two largest utilities were on the verge of > > > bankruptcy led some suppliers to withhold power from California, > > > despite an emergency federal order requiring them to sell excess > > > electricity to the state, said Jim Detmers, the ISO's managing > > > director of operations. > > > > > > But Winter said later that he did not believe generators were > > > withholding power. Instead, he said, the main problem is > broken power > > > generating facilities, many of them older plants that > have been run > > > heavily since June. > > > > > > Compounding the problem is a general scarcity of electricity > > > nationally, and a lack of snow and rain in the > > > hydroelectric-dependent Pacific Northwest, Detmers said. > > > > > > ``If you are out in the community and get into an > intersection that > > > is in the blackout, use caution,'' he said. > > > > > > The day began with the third Stage 3 power alert within a week, > > > meaning reserves were close to just 1.5 percent. The > warning marked > > > at least the third time California neared blackouts since > its power > > > woes began last summer. > > > > > > The ISO fended off outages before by temporarily turning off huge > > > state pumps that move water from Northern California to the south, > > > sucking enough power for 600,000 homes, but that wasn't enough > > > Thursday. > > > > > > Suppliers were ``reluctant to provide power to California > because of > > > the financial situation of the utilities,'' Detmers said. > > > > > > He said the ISO wasn't probing whether suppliers were flat-out > > > ignoring Energy Secretary Bill Richardson's emergency > order insisting > > > that any spare power go to California, however. > > > > > > ``We're just trying to get the power delivered,'' he said. > > > > > > Adding to the problems, several power plants that were expected to > > > return to full operation after repairs did not, Detmers said. > > > > > > On Tuesday, Southern California Edison declared itself > unable to pay > > > hundreds of millions in wholesale electricity bills, and > it and PG&E, > > > the state's largest utilities, took another hit on Wall Street. > > > > > > SoCal Edison, which serves 11 million people, said it > cannot pay $596 > > > million in bills for wholesale energy and debt service, including > > > $215 million to the California Power Exchange. > > > > > > The Power Exchange was considering whether to make the utility buy > > > its power elsewhere and an electricity supplier > threatened to force > > > SoCal Edison into bankruptcy if it failed to pay its bills. > > > > > > The default prompted Standard & Poor's and Moody's to > downgrade the > > > credit ratings of SoCal Edison and PG&E to junk-bond status. > > > > > > The credit agency said SoCal Edison's delinquency also > tainted PG&E. > > > With just $500 million in cash left as of Jan. 10, PG&E faces due > > > dates on bills totaling $1 billion during the first two weeks of > > > February. > > > > > > Between them, PG&E and SoCal Edison have lost at least > $10 billion in > > > wholesale energy costs. A rate freeze imposed as the > state phases in > > > deregulation has blocked them from passing on higher > wholesale costs > > > to their customers. > > > > > > Wholesale power prices have risen dramatically since > June, in part of > > > because of a hot summer and a cold winter. In 1999, they averaged > > > perhaps 3.5 cents a kilowatt. Now, they are running about > 30 cents, > > > and sometimes far higher. > > > > > > Demand has remained high, supplies are strapped because > no new power > > > plants have been built in the state in recent years and > imports are > > > tight because other states are fighting over the power. > > > > > > In addition, spiraling prices for natural gas are forcing power > > > plants to raise their prices. Most power plants are fired > by natural > > > gas. > > > > > > On Tuesday, unusually high demand for natural gas, due in part to > > > cold weather, led San Diego Gas and Electric to cut > supplies to two > > > power plants, contributing to the state's Stage 3 alert. > > > > > > The utility said there was plenty of natural gas, but not enough > > > space in the pipeline to meet its customers' needs. To > maintain the > > > supply for its home and small-business users, the utility cut the > > > flow to the two power plants and six large industrial customers. > > > > > > The state avoided rolling blackouts after huge state > pumps that move > > > water from Northern California to the south were turned off > > > temporarily, conserving enough electricity to power 600,000 homes, > > > said Kellan Fluckiger, the ISO's chief operating officer. > > > > > > Joel Nelsen, president of California Citrus Mutual, spent > Tuesday on > > > the phone with Central Valley lawmakers and the governor's office > > > trying to ensure that orange growers wouldn't face outages as they > > > tried to protect crops from a cold snap. > > > > > > ``We're terribly exposed,'' said Nelsen, who heads a trade > > > association of 800 growers. ``The loss of power for a short time > > > could wreak untold damage on our crop.'' > > > > > > > > >

3 2

Re: Favorites (Re: UUNET peering policy)
by Richard A. Steenbergen 18 Jan '01

18 Jan '01

On Wed, Jan 17, 2001 at 05:39:41PM -0700, Danny McPherson wrote: > > And often enough, I've seen MEDs result in worst-exit routing. > That is, routing that was intended to be better than closest-exit > (aka. hot-potato), but turns out to be far worse than best-exit > (I believe some refer to this as cold-potato *8^/) because an > adjacent AS in unable to correctly convey the optimal entry point > into their AS via MEDs, be it because of GOOD aggregation, or > uncluefully derived MED values. I believe Avi Freedman would call that "mashed-potato" routing. -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/humble PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

1 0

Re: California regulators ordered rolling blackouts
by matthew＠tycho.net 18 Jan '01

18 Jan '01

Oh, we equate power failure and rain here in Santa Cruz, California as well :) It is strangely comforting to know that we're all in the same boat, albeit for different reasons. -matthew kaufman Tycho Networks/DSL.net matthew(a)tycho.net Original message <01b401c08120$36173100$9eb431c6@oemcomputer> From: "Alexei Roudnev" <alex(a)relcom.net> Date: Jan 17, 23:28 Subject: Re: California regulators ordered rolling blackouts > > There is big difference. In Russia, all data centers etc have a reliablr power > and, if located in such region as Vladivostok, have independent power supply; > power failure is common event (such as rain) for them and rollout blackouts does > not interrupt their work; in California people used to believe to the good news > and any failure became disaster.

1 0

Re: Favorites (Re: UUNET peering policy)
by Danny McPherson 18 Jan '01

18 Jan '01

> Honoring MEDs is not Black and White. I dont know one large network that > honors all MEDs from every peer. For the most part, I would say that most > providers do 'warm' routing. it just depends on what degree of warm you > can live with. Of course, the amount of brokenness involved with honoring MEDs is questionable anyways. For example, you can't compare MEDs from different ASs and attain any reasonable result. For that matter, the result may very well be persistent route oscillation. And often enough, I've seen MEDs result in worst-exit routing. That is, routing that was intended to be better than closest-exit (aka. hot-potato), but turns out to be far worse than best-exit (I believe some refer to this as cold-potato *8^/) because an adjacent AS in unable to correctly convey the optimal entry point into their AS via MEDs, be it because of GOOD aggregation, or uncluefully derived MED values. Then again, I've seen MEDs work just as intended, allowing a peer network to easily manipulate route/forwarding decsions in a neighboring network in order to alleviate congestion or other issues within their own network. Of course, there are clever ways to make MEDs of benefit, though IMO, they do more harm than good overall. -danny

1 0

Favorites (Re: UUNET peering policy)
by Sean Donelan 17 Jan '01

17 Jan '01

On Sun, 14 January 2001, Paul Vixie wrote: > sean(a)donelan.com (Sean Donelan) writes: > > If you look at Abovenet's traffic graphs, you'll notice Abovenet has a wide > > variety of traffic balances with different providers. Some in Abovenet's > > favor (such as 3:1 with Sprint, 5:1 with Teleglobe) and some in the other > > provider's favor (such as 1:3 with Exodus). ... > > "Favor"? What, precisely, connotes "favor" in this regard? Sending more, or > receiving more? And: why? Which side of the debate do you want to take? The traditional arguement is a network composed mostly of a few large data centers, with lots of servers sending traffic is getting a "free ride" on the network which built out nationwide and has POPs in every LATA. UUNET deserves a return on its investment on all those wholesale dialup POPs and circuits to underserved rural areas. Abovenet is just cream skimming in a few large metro areas, while UUNET does the hard work of carrying that extra traffic imbalance. Abovenet selling "cheap" bandwidth because it doesn't have the cost of delievering the traffic that UUNET has to pay. The opposite side is Abovenet has invested a lot into its sites and MFN into its networks. It just choose to do it in a different way than UUNET. Its more expensive to lay fiber in metro areas than rural areas. It costs a lot of money to operate the centers. Whether the traffic is being paid by the millions of $19.95 dialup users on UUNET's wholesale ports or by the hundreds of hosters in Abovenet's sites, the traffic is paid.

13 12

Re: California regulators ordered rolling blackouts
by matthew＠tycho.net 17 Jan '01

17 Jan '01

> It has begun. Welcome to the 3rd world. >From CNN's article today about the MIR space station, we have this timely quote: "Cash-strapped Russia, where whole regions are regularly plunged into darkness due to power cuts..." (http://www.cnn.com/2001/TECH/space/01/17/mir.launch.reut/index.html) -matthew kaufman Tycho Networks/DSL.net matthew(a)tycho.net

1 0

Re: California hits stage 3 power emergency
by Sean Donelan 17 Jan '01

17 Jan '01

On Wed, 17 January 2001, Matt Clauson wrote: > PG&E started with "block 3" (look on your power bill for what block number > you are) -- assuming 1 hour per blackout, and they go through the block > numbers in sequence, you have an idea of when you'll get hit. Anyone have > an idea of how many PG&E blocks there are? PG&E has divided its service area into 14 blocks of 550 megawatts. Other utility companies, such as City of Palo Alto ( http://www.cpau.com ), have their own blocks. If you share a circuit with a "critical action" (i.e. hospital, fire station) you will not have a rolling blackout code. The utility avoids interrupting service to those facilities. However, they may still experience an uncontrolled blackout if the grid fails. All Internet facilities in California, which I checked, appear to be operating normally. Calling around, some folks are or have been on emergency generator power, but this has not affected customer service so far.

1 0

message from California Governor's Office of Emergency Services
by Sean Donelan 17 Jan '01

17 Jan '01

From: Governor's Office of Emergency Services SACRAMENTO-In response to the Cal ISO declaration of a Stage 3 Electrical Emergency issued today at 0145 thru 2400, the State OES has issued the following message to all California emergency services personnel. 'Critical Information Please notify Emergency Services Managers, Fire and Law Enforcement agencies: Emergency services personnel (Law-Enforcement, Fire, EMS, and Local Offices of Emergency Services) throughout California should be advised that the California Independent System Operator, the entity that coordinates statewide flow of electrical supply, did declare a Stage 3 Emergency effective today's date at 0145 thru 2400HRS. At this time PG&E is dropping firm load of 500 mega watts in Northern California (Rolling Black-outs). We do not have information on specific areas effected. All Law Enforcement, Fire, EMS, and Offices of Emergency Services management personnel should be notified of this message, as incoming call volumes and requests for assistance may increase during this time. Local energy suppliers may be able to provide more detailed information on potential or actual local impacts.'

1 0

RE: California regulators ordered rolling blackouts
by Roeland Meyer 17 Jan '01

17 Jan '01

Next question, is the State going to over-rule the AQMD to get generators back online? > -----Original Message----- > From: Bora Akyol [mailto:akyol@akyol.org] > Sent: Wednesday, January 17, 2001 1:50 PM > To: nanog(a)merit.edu; Rusty H. Hodge > Subject: Re: California regulators ordered rolling blackouts > > > > Even third world countries aren't as idiotic as California. There when > electricity goes off, it usually is because there is none to > be had. In our > situation, it is manipulation and poor planning to the extreme. > > My power was out for at least one hour, it is back now. > > Bora > > ----- Original Message ----- > From: "Rusty H. Hodge" <rusty(a)hodge.com> > To: <nanog(a)merit.edu> > Sent: Wednesday, January 17, 2001 1:37 PM > Subject: California regulators ordered rolling blackouts > > > > > > It has begun. Welcome to the 3rd world. > > > > > http://www.sfgate.com/cgi-bin/article.cgi?file=/news/archive/2 > 001/01/17/stat > e1503EST0161.DTL > > > > Blackouts hit California as utility financial woes deepen > > > > Associated Press, SF Gate Wednesday, January 17, 2001 > > Breaking News Sections > > > > > -------------------------------------------------------------- > ---------- > > > > California regulators ordered rolling blackouts Wednesday for the > > first time in the state's months-long electricity crisis, blaming > > utility credit problems and a tight national power supply for the > > scattered outages. > > > > The rotating blackouts, expected to affect about 500,000 customers > > for an hour to 90 minutes, were restricted to Pacific Gas and > > Electric Co. territory in northern and part of central California, > > said Stephanie McCorkle, a spokeswoman for the Independent System > > operator, keeper of the state's power grid. > > > > A PG&E spokesman said that about 250,000 people were already without > > power from the blackouts that began at 11:41 a.m. There are reports > > of outages in San Francisco's Lower Haight, Oakland's > Rockridge area, > > the Oakland Hills, Orinda, the Peninsula from South San Francisco to > > San Mateo, parts of Napa and Sonoma County, downtown San Jose, > > Cupertino's De Anza College, San Ramon, Santa Cruz, Benicia > and other > > areas, according to radio and TV reports. > > > > There was a report of two students being trapped in an elevator at > > Hastings School of Law in San Francisco. Some ATMs were reported out > > of service in downtown San Francisco. > > > > Power was cut to the Cow Palace in Daly City, which is > holding a boat > > show, according to an administrative worker at the exhibition hall. > > The main building remains well-lit from emergency lighting and > > skylights, she said. > > > > KICU-TV, channel 36, was knocked off the air. > > > > The blackouts would first affect customers in scattered areas known > > as blocks 3 and 4. For security reasons, the precise locations are > > not released. Consumers can find their block numbers at the > bottom of > > their power bills. > > > > PG&E and state officials urged conservation and said it was > not known > > yet if blackouts would be ordered for blocks 5, 6 and 7. > Blocks 1 and > > 2 had power outages last June. > > > > BART, fire departments, police stations and hospitals are not > > affected. Motorists who come to non-functioning traffic signals, > > including El Camino Real in the San Bruno/San Mateo area > and Lawrence > > Expressway in San Jose, should treat them as four-way stops. > > > > Utilities try to avoid cutting power to blocks with essential > > services such as hospitals. > > > > Terry Winter, president of Cal-ISO, said that a large power plant on > > California's Central Coast went down at about 11 a.m., necessitating > > the outages. > > > > Worry that the state's two largest utilities were on the verge of > > bankruptcy led some suppliers to withhold power from California, > > despite an emergency federal order requiring them to sell excess > > electricity to the state, said Jim Detmers, the ISO's managing > > director of operations. > > > > But Winter said later that he did not believe generators were > > withholding power. Instead, he said, the main problem is > broken power > > generating facilities, many of them older plants that have been run > > heavily since June. > > > > Compounding the problem is a general scarcity of electricity > > nationally, and a lack of snow and rain in the > > hydroelectric-dependent Pacific Northwest, Detmers said. > > > > ``If you are out in the community and get into an intersection that > > is in the blackout, use caution,'' he said. > > > > The day began with the third Stage 3 power alert within a week, > > meaning reserves were close to just 1.5 percent. The warning marked > > at least the third time California neared blackouts since its power > > woes began last summer. > > > > The ISO fended off outages before by temporarily turning off huge > > state pumps that move water from Northern California to the south, > > sucking enough power for 600,000 homes, but that wasn't enough > > Thursday. > > > > Suppliers were ``reluctant to provide power to California because of > > the financial situation of the utilities,'' Detmers said. > > > > He said the ISO wasn't probing whether suppliers were flat-out > > ignoring Energy Secretary Bill Richardson's emergency order > insisting > > that any spare power go to California, however. > > > > ``We're just trying to get the power delivered,'' he said. > > > > Adding to the problems, several power plants that were expected to > > return to full operation after repairs did not, Detmers said. > > > > On Tuesday, Southern California Edison declared itself unable to pay > > hundreds of millions in wholesale electricity bills, and it > and PG&E, > > the state's largest utilities, took another hit on Wall Street. > > > > SoCal Edison, which serves 11 million people, said it > cannot pay $596 > > million in bills for wholesale energy and debt service, including > > $215 million to the California Power Exchange. > > > > The Power Exchange was considering whether to make the utility buy > > its power elsewhere and an electricity supplier threatened to force > > SoCal Edison into bankruptcy if it failed to pay its bills. > > > > The default prompted Standard & Poor's and Moody's to downgrade the > > credit ratings of SoCal Edison and PG&E to junk-bond status. > > > > The credit agency said SoCal Edison's delinquency also tainted PG&E. > > With just $500 million in cash left as of Jan. 10, PG&E faces due > > dates on bills totaling $1 billion during the first two weeks of > > February. > > > > Between them, PG&E and SoCal Edison have lost at least $10 > billion in > > wholesale energy costs. A rate freeze imposed as the state phases in > > deregulation has blocked them from passing on higher wholesale costs > > to their customers. > > > > Wholesale power prices have risen dramatically since June, > in part of > > because of a hot summer and a cold winter. In 1999, they averaged > > perhaps 3.5 cents a kilowatt. Now, they are running about 30 cents, > > and sometimes far higher. > > > > Demand has remained high, supplies are strapped because no new power > > plants have been built in the state in recent years and imports are > > tight because other states are fighting over the power. > > > > In addition, spiraling prices for natural gas are forcing power > > plants to raise their prices. Most power plants are fired by natural > > gas. > > > > On Tuesday, unusually high demand for natural gas, due in part to > > cold weather, led San Diego Gas and Electric to cut supplies to two > > power plants, contributing to the state's Stage 3 alert. > > > > The utility said there was plenty of natural gas, but not enough > > space in the pipeline to meet its customers' needs. To maintain the > > supply for its home and small-business users, the utility cut the > > flow to the two power plants and six large industrial customers. > > > > The state avoided rolling blackouts after huge state pumps that move > > water from Northern California to the south were turned off > > temporarily, conserving enough electricity to power 600,000 homes, > > said Kellan Fluckiger, the ISO's chief operating officer. > > > > Joel Nelsen, president of California Citrus Mutual, spent Tuesday on > > the phone with Central Valley lawmakers and the governor's office > > trying to ensure that orange growers wouldn't face outages as they > > tried to protect crops from a cold snap. > > > > ``We're terribly exposed,'' said Nelsen, who heads a trade > > association of 800 growers. ``The loss of power for a short time > > could wreak untold damage on our crop.'' > > > >

1 0

California hits stage 3 power emergency
by Dalvenjah FoxFire 17 Jan '01

17 Jan '01

I haven't seen Sean post about this yet, so figured I would. California's been getting hit with a nasty storm since yesterday that's taken out some more major power plants; the CAISO declared a Stage 3 (rolling blackouts all around) power emergency just before 9 AM PST today. See http://www.caiso.com/SystemStatus.html for pretty graphs. Have you test-run your backup generators and transfer switches lately? }:> -dalvenjah -- Dalvenjah FoxFire (aka Sven Nielsen) In cyberspace, noone knows you're Founder, the DALnet IRC Network wearing pointy ears. e-mail: dalvenjah(a)dal.net WWW: http://www.dal.net/~dalvenjah/ whois: SN90 Try DALnet! http://www.dal.net/

4 3