Favorites (Re: UUNET peering policy)
On Sun, 14 January 2001, Paul Vixie wrote:
sean@donelan.com (Sean Donelan) writes:
If you look at Abovenet's traffic graphs, you'll notice Abovenet has a wide variety of traffic balances with different providers. Some in Abovenet's favor (such as 3:1 with Sprint, 5:1 with Teleglobe) and some in the other provider's favor (such as 1:3 with Exodus). ...
"Favor"? What, precisely, connotes "favor" in this regard? Sending more, or receiving more? And: why?
Which side of the debate do you want to take? The traditional arguement is a network composed mostly of a few large data centers, with lots of servers sending traffic is getting a "free ride" on the network which built out nationwide and has POPs in every LATA. UUNET deserves a return on its investment on all those wholesale dialup POPs and circuits to underserved rural areas. Abovenet is just cream skimming in a few large metro areas, while UUNET does the hard work of carrying that extra traffic imbalance. Abovenet selling "cheap" bandwidth because it doesn't have the cost of delievering the traffic that UUNET has to pay. The opposite side is Abovenet has invested a lot into its sites and MFN into its networks. It just choose to do it in a different way than UUNET. Its more expensive to lay fiber in metro areas than rural areas. It costs a lot of money to operate the centers. Whether the traffic is being paid by the millions of $19.95 dialup users on UUNET's wholesale ports or by the hundreds of hosters in Abovenet's sites, the traffic is paid.
The argument that "outbound is bad" was particularly sound when most national backbones were 1) leased and 2) small. That's not really the case anymore. With lots of bandwidth, this paradigm should really start to shift - who cares if someone if dumping outbound on you, and you have to backhaul it across the country? There would seem to be no actual economic reason to justify traffic ratios - except, of course the desire to discourage peering and encourage transit, but not removing a barrier that once was meaningful, and now is deprecated. I don't feel sympathy for the example of many dialup POPs and a few large data centers. There is a value for both parties in the exchange - content is delivered to customers faster. With the cost of long-haul taking a dive, one might think this would have more impact than a traffic ratio. - Daniel Golding On 15 Jan 2001, Sean Donelan wrote:
On Sun, 14 January 2001, Paul Vixie wrote:
sean@donelan.com (Sean Donelan) writes:
If you look at Abovenet's traffic graphs, you'll notice Abovenet has a wide variety of traffic balances with different providers. Some in Abovenet's favor (such as 3:1 with Sprint, 5:1 with Teleglobe) and some in the other provider's favor (such as 1:3 with Exodus). ...
"Favor"? What, precisely, connotes "favor" in this regard? Sending more, or receiving more? And: why?
Which side of the debate do you want to take?
The traditional arguement is a network composed mostly of a few large data centers, with lots of servers sending traffic is getting a "free ride" on the network which built out nationwide and has POPs in every LATA.
UUNET deserves a return on its investment on all those wholesale dialup POPs and circuits to underserved rural areas. Abovenet is just cream skimming in a few large metro areas, while UUNET does the hard work of carrying that extra traffic imbalance. Abovenet selling "cheap" bandwidth because it doesn't have the cost of delievering the traffic that UUNET has to pay.
The opposite side is Abovenet has invested a lot into its sites and MFN into its networks. It just choose to do it in a different way than UUNET. Its more expensive to lay fiber in metro areas than rural areas. It costs a lot of money to operate the centers. Whether the traffic is being paid by the millions of $19.95 dialup users on UUNET's wholesale ports or by the hundreds of hosters in Abovenet's sites, the traffic is paid.
On 15 Jan 2001, Sean Donelan wrote:
On Sun, 14 January 2001, Paul Vixie wrote:
sean@donelan.com (Sean Donelan) writes:
If you look at Abovenet's traffic graphs, you'll notice Abovenet has a wide variety of traffic balances with different providers. Some in Abovenet's favor (such as 3:1 with Sprint, 5:1 with Teleglobe) and some in the other provider's favor (such as 1:3 with Exodus). ...
"Favor"? What, precisely, connotes "favor" in this regard? Sending more, or receiving more? And: why?
Which side of the debate do you want to take?
The traditional arguement is a network composed mostly of a few large data centers, with lots of servers sending traffic is getting a "free ride" on the network which built out nationwide and has POPs in every LATA.
UUNET deserves a return on its investment on all those wholesale dialup POPs and circuits to underserved rural areas. Abovenet is just cream skimming in a few large metro areas, while UUNET does the hard work of carrying that extra traffic imbalance. Abovenet selling "cheap" bandwidth because it doesn't have the cost of delievering the traffic that UUNET has to pay.
The opposite side is Abovenet has invested a lot into its sites and MFN into its networks. It just choose to do it in a different way than UUNET. Its more expensive to lay fiber in metro areas than rural areas. It costs a lot of money to operate the centers. Whether the traffic is being paid by the millions of $19.95 dialup users on UUNET's wholesale ports or by the hundreds of hosters in Abovenet's sites, the traffic is paid.
Warning: the following is oversimplified: I think what it comes down to, and what it has come down to at least since the inception of hosting companies who spew large amounts of traffic back at the access networks, is who gets paid twice for carrying the traffic? Does UUNET get paid twice for carrying the traffic, once by their customer that pays for dial or leased line access and once by the hosting company that pays for peering because their traffic ratio is off? This seems to be the status quo. The other way around would mean that the hosting company got paid twice for carrying the traffic. -travis
On Mon, 15 Jan 2001, Travis Pugh wrote:
I think what it comes down to, and what it has come down to at least since the inception of hosting companies who spew large amounts of traffic back at the access networks, is who gets paid twice for carrying the traffic? Does UUNET get paid twice for carrying the traffic, once by their customer that pays for dial or leased line access and once by the hosting company that pays for peering because their traffic ratio is off? This seems to be the status quo. The other way around would mean that the hosting company got paid twice for carrying the traffic.
... and if we bring the chicken and the egg syndrome into all of this, then who has most to gain that the internet is fast and efficient, the people who make content, the people who live off of the people who make content, the people who live off of the people consuming content or the people who consume the content? ... and where is the growth? ... and who gains from that? One might think that UUNET in this case would be happy that someone would provide content so that more users would want to buy even more bandwidth so that UUNET would make more money. On the other hand one might also think that UUNET would want their investment to last as long as possible and that upgrading is more expensive than making it possible for people to buy more bandwidth. Whatever is most important, I do not know. I know it's easy to argue for both ways. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Mon, 15 Jan 2001, Travis Pugh wrote:
Warning: the following is oversimplified:
I think what it comes down to, and what it has come down to at least since the inception of hosting companies who spew large amounts of traffic back at the access networks, is who gets paid twice for carrying the traffic? Does UUNET get paid twice for carrying the traffic, once by their customer that pays for dial or leased line access and once by the hosting company that pays for peering because their traffic ratio is off? This seems to be the status quo. The other way around would mean that the hosting company got paid twice for carrying the traffic.
If I am a generic business, and I connect to UUNET twice, in different locations, and the ONLY traffic I send is between my two offices, should I expect a discount on transit? No. I would expect to pay it at both ends. What is the difference? -- "Do not pound nails into glass", said Tom painstakingly. Charlie Watts cewatts@frontier.net
On Mon, 15 Jan 2001, Charlie Watts wrote:
On Mon, 15 Jan 2001, Travis Pugh wrote:
Warning: the following is oversimplified:
I think what it comes down to, and what it has come down to at least since the inception of hosting companies who spew large amounts of traffic back at the access networks, is who gets paid twice for carrying the traffic? Does UUNET get paid twice for carrying the traffic, once by their customer that pays for dial or leased line access and once by the hosting company that pays for peering because their traffic ratio is off? This seems to be the status quo. The other way around would mean that the hosting company got paid twice for carrying the traffic.
If I am a generic business, and I connect to UUNET twice, in different locations, and the ONLY traffic I send is between my two offices, should I expect a discount on transit?
No. I would expect to pay it at both ends.
What is the difference?
-- "Do not pound nails into glass", said Tom painstakingly. Charlie Watts cewatts@frontier.net
Actually, if you are a generic business, and you connect to UUNET twice, in different locations, and the ONLY traffic you send is between your two offices, you should talk to your salesperson about VPN services on their backbone vs purchasing transit from them. And YES -- you SHOULD expect it to be less expensive. --- John Fraizer EnterZone, Inc
On Mon, Jan 15, 2001 at 10:37:18AM -0800, Sean Donelan wrote:
The traditional arguement is a network composed mostly of a few large data centers, with lots of servers sending traffic is getting a "free ride" on the network which built out nationwide and has POPs in every LATA.
Note, one of the ways AboveNet addresses this issue is by honoring meds. This does not work with all networks, but I will put forth an example: LAX NYC Net 1 (eyeballs): 1---------------2-----user | | | | PeeringA PeeringB | | | | Net 2: (content): server----3---------------4 In a traditional (closest exit) setup the user will make a request of the server, and traffic will flow as follows: user->server: 2->4->3 server->user: 3->1->2 Since there is generally much more traffic (bits) server->user the "eyeball" network incurs a higher bit/mile cost (the long haul from 1->2) in this case. In the pure case (100% eyeball network peering with 100% content network, closest exit) it is safe to say the eyeball network carries a higher cost. Consider what AboveNet does. By honoring meds the following paths occur (abovenet as net 2 in the picture): user->server: 2->4->3 server->user: 3->4->2 The bit/mile cost just shifted to the 3->4 link, paid for by the "server" network. At this point the server network is actually incuring more cost (assuming the pure case, again). This all gets muddied very quickly in the real world, as no network is "100% eyeballs" or "100% content", and there are many other geographical issues involved etc. This is but one tool that can be used to fix the problem of one network incuring more cost than the other to interconnect. To refuse flatly based on ratio, or bit/mile cost without looking for relatively easy alternatives to balance the cost is a bit short sighted. Peering is about "equal value". Consider the pure content provider co-located witht he pure eyeball provider, where neither have a long haul network (eg local providers, similar bit/mile cost on each side). The costs are equally distributed, both get similar "value" from the interconnection (and indeed, can't exist without each other in the pure case), but the ratio may well be 10:1. -- Leo Bicknell - bicknell@ufp.org Systems Engineer - Internetworking Engineer - CCIE 3440 Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On Wed, 17 Jan 2001, Leo Bicknell wrote:
user->server: 2->4->3 server->user: 3->1->2
Consider what AboveNet does. By honoring meds the following paths occur (abovenet as net 2 in the picture):
user->server: 2->4->3 server->user: 3->4->2
This assumes that the network you are peering with sends correct MEDs. This also assumes that one always honors MEDs. What happens during fiber cuts or when you are waiting for an upgrade on your circuits? Do you continue to honor MEDs and hurt the customer? Honoring MEDs is not Black and White. I dont know one large network that honors all MEDs from every peer. For the most part, I would say that most providers do 'warm' routing. it just depends on what degree of warm you can live with. Christian
According to what I've heard from Qwest folks, they honor MEDS. (Taken with a grain of salt, it was part of a sales pitch). On Wed, Jan 17, 2001 at 12:25:50PM -0800, Christian Nielsen wrote:
On Wed, 17 Jan 2001, Leo Bicknell wrote:
user->server: 2->4->3 server->user: 3->1->2
Consider what AboveNet does. By honoring meds the following paths occur (abovenet as net 2 in the picture):
user->server: 2->4->3 server->user: 3->4->2
This assumes that the network you are peering with sends correct MEDs. This also assumes that one always honors MEDs. What happens during fiber cuts or when you are waiting for an upgrade on your circuits? Do you continue to honor MEDs and hurt the customer?
Honoring MEDs is not Black and White. I dont know one large network that honors all MEDs from every peer. For the most part, I would say that most providers do 'warm' routing. it just depends on what degree of warm you can live with.
Christian
-- Marius Strom <marius@marius.org> Professional Geek/Unix System Administrator URL: http://www.marius.org/ http://www.marius.org/marius.pgp 0x55DE53E4 "Never underestimate the bandwidth of a mini-van full of DLT tapes traveling down the highway at 65 miles per hour..." -Andrew Tanenbaum, "Computer Networks"
It has begun. Welcome to the 3rd world. http://www.sfgate.com/cgi-bin/article.cgi?file=/news/archive/2001/01/17/stat... Blackouts hit California as utility financial woes deepen Associated Press, SF Gate Wednesday, January 17, 2001 Breaking News Sections ------------------------------------------------------------------------ California regulators ordered rolling blackouts Wednesday for the first time in the state's months-long electricity crisis, blaming utility credit problems and a tight national power supply for the scattered outages. The rotating blackouts, expected to affect about 500,000 customers for an hour to 90 minutes, were restricted to Pacific Gas and Electric Co. territory in northern and part of central California, said Stephanie McCorkle, a spokeswoman for the Independent System operator, keeper of the state's power grid. A PG&E spokesman said that about 250,000 people were already without power from the blackouts that began at 11:41 a.m. There are reports of outages in San Francisco's Lower Haight, Oakland's Rockridge area, the Oakland Hills, Orinda, the Peninsula from South San Francisco to San Mateo, parts of Napa and Sonoma County, downtown San Jose, Cupertino's De Anza College, San Ramon, Santa Cruz, Benicia and other areas, according to radio and TV reports. There was a report of two students being trapped in an elevator at Hastings School of Law in San Francisco. Some ATMs were reported out of service in downtown San Francisco. Power was cut to the Cow Palace in Daly City, which is holding a boat show, according to an administrative worker at the exhibition hall. The main building remains well-lit from emergency lighting and skylights, she said. KICU-TV, channel 36, was knocked off the air. The blackouts would first affect customers in scattered areas known as blocks 3 and 4. For security reasons, the precise locations are not released. Consumers can find their block numbers at the bottom of their power bills. PG&E and state officials urged conservation and said it was not known yet if blackouts would be ordered for blocks 5, 6 and 7. Blocks 1 and 2 had power outages last June. BART, fire departments, police stations and hospitals are not affected. Motorists who come to non-functioning traffic signals, including El Camino Real in the San Bruno/San Mateo area and Lawrence Expressway in San Jose, should treat them as four-way stops. Utilities try to avoid cutting power to blocks with essential services such as hospitals. Terry Winter, president of Cal-ISO, said that a large power plant on California's Central Coast went down at about 11 a.m., necessitating the outages. Worry that the state's two largest utilities were on the verge of bankruptcy led some suppliers to withhold power from California, despite an emergency federal order requiring them to sell excess electricity to the state, said Jim Detmers, the ISO's managing director of operations. But Winter said later that he did not believe generators were withholding power. Instead, he said, the main problem is broken power generating facilities, many of them older plants that have been run heavily since June. Compounding the problem is a general scarcity of electricity nationally, and a lack of snow and rain in the hydroelectric-dependent Pacific Northwest, Detmers said. ``If you are out in the community and get into an intersection that is in the blackout, use caution,'' he said. The day began with the third Stage 3 power alert within a week, meaning reserves were close to just 1.5 percent. The warning marked at least the third time California neared blackouts since its power woes began last summer. The ISO fended off outages before by temporarily turning off huge state pumps that move water from Northern California to the south, sucking enough power for 600,000 homes, but that wasn't enough Thursday. Suppliers were ``reluctant to provide power to California because of the financial situation of the utilities,'' Detmers said. He said the ISO wasn't probing whether suppliers were flat-out ignoring Energy Secretary Bill Richardson's emergency order insisting that any spare power go to California, however. ``We're just trying to get the power delivered,'' he said. Adding to the problems, several power plants that were expected to return to full operation after repairs did not, Detmers said. On Tuesday, Southern California Edison declared itself unable to pay hundreds of millions in wholesale electricity bills, and it and PG&E, the state's largest utilities, took another hit on Wall Street. SoCal Edison, which serves 11 million people, said it cannot pay $596 million in bills for wholesale energy and debt service, including $215 million to the California Power Exchange. The Power Exchange was considering whether to make the utility buy its power elsewhere and an electricity supplier threatened to force SoCal Edison into bankruptcy if it failed to pay its bills. The default prompted Standard & Poor's and Moody's to downgrade the credit ratings of SoCal Edison and PG&E to junk-bond status. The credit agency said SoCal Edison's delinquency also tainted PG&E. With just $500 million in cash left as of Jan. 10, PG&E faces due dates on bills totaling $1 billion during the first two weeks of February. Between them, PG&E and SoCal Edison have lost at least $10 billion in wholesale energy costs. A rate freeze imposed as the state phases in deregulation has blocked them from passing on higher wholesale costs to their customers. Wholesale power prices have risen dramatically since June, in part of because of a hot summer and a cold winter. In 1999, they averaged perhaps 3.5 cents a kilowatt. Now, they are running about 30 cents, and sometimes far higher. Demand has remained high, supplies are strapped because no new power plants have been built in the state in recent years and imports are tight because other states are fighting over the power. In addition, spiraling prices for natural gas are forcing power plants to raise their prices. Most power plants are fired by natural gas. On Tuesday, unusually high demand for natural gas, due in part to cold weather, led San Diego Gas and Electric to cut supplies to two power plants, contributing to the state's Stage 3 alert. The utility said there was plenty of natural gas, but not enough space in the pipeline to meet its customers' needs. To maintain the supply for its home and small-business users, the utility cut the flow to the two power plants and six large industrial customers. The state avoided rolling blackouts after huge state pumps that move water from Northern California to the south were turned off temporarily, conserving enough electricity to power 600,000 homes, said Kellan Fluckiger, the ISO's chief operating officer. Joel Nelsen, president of California Citrus Mutual, spent Tuesday on the phone with Central Valley lawmakers and the governor's office trying to ensure that orange growers wouldn't face outages as they tried to protect crops from a cold snap. ``We're terribly exposed,'' said Nelsen, who heads a trade association of 800 growers. ``The loss of power for a short time could wreak untold damage on our crop.''
Even third world countries aren't as idiotic as California. There when electricity goes off, it usually is because there is none to be had. In our situation, it is manipulation and poor planning to the extreme. My power was out for at least one hour, it is back now. Bora ----- Original Message ----- From: "Rusty H. Hodge" <rusty@hodge.com> To: <nanog@merit.edu> Sent: Wednesday, January 17, 2001 1:37 PM Subject: California regulators ordered rolling blackouts
It has begun. Welcome to the 3rd world.
http://www.sfgate.com/cgi-bin/article.cgi?file=/news/archive/2001/01/17/stat e1503EST0161.DTL
Blackouts hit California as utility financial woes deepen
Associated Press, SF Gate Wednesday, January 17, 2001 Breaking News Sections
------------------------------------------------------------------------
California regulators ordered rolling blackouts Wednesday for the first time in the state's months-long electricity crisis, blaming utility credit problems and a tight national power supply for the scattered outages.
The rotating blackouts, expected to affect about 500,000 customers for an hour to 90 minutes, were restricted to Pacific Gas and Electric Co. territory in northern and part of central California, said Stephanie McCorkle, a spokeswoman for the Independent System operator, keeper of the state's power grid.
A PG&E spokesman said that about 250,000 people were already without power from the blackouts that began at 11:41 a.m. There are reports of outages in San Francisco's Lower Haight, Oakland's Rockridge area, the Oakland Hills, Orinda, the Peninsula from South San Francisco to San Mateo, parts of Napa and Sonoma County, downtown San Jose, Cupertino's De Anza College, San Ramon, Santa Cruz, Benicia and other areas, according to radio and TV reports.
There was a report of two students being trapped in an elevator at Hastings School of Law in San Francisco. Some ATMs were reported out of service in downtown San Francisco.
Power was cut to the Cow Palace in Daly City, which is holding a boat show, according to an administrative worker at the exhibition hall. The main building remains well-lit from emergency lighting and skylights, she said.
KICU-TV, channel 36, was knocked off the air.
The blackouts would first affect customers in scattered areas known as blocks 3 and 4. For security reasons, the precise locations are not released. Consumers can find their block numbers at the bottom of their power bills.
PG&E and state officials urged conservation and said it was not known yet if blackouts would be ordered for blocks 5, 6 and 7. Blocks 1 and 2 had power outages last June.
BART, fire departments, police stations and hospitals are not affected. Motorists who come to non-functioning traffic signals, including El Camino Real in the San Bruno/San Mateo area and Lawrence Expressway in San Jose, should treat them as four-way stops.
Utilities try to avoid cutting power to blocks with essential services such as hospitals.
Terry Winter, president of Cal-ISO, said that a large power plant on California's Central Coast went down at about 11 a.m., necessitating the outages.
Worry that the state's two largest utilities were on the verge of bankruptcy led some suppliers to withhold power from California, despite an emergency federal order requiring them to sell excess electricity to the state, said Jim Detmers, the ISO's managing director of operations.
But Winter said later that he did not believe generators were withholding power. Instead, he said, the main problem is broken power generating facilities, many of them older plants that have been run heavily since June.
Compounding the problem is a general scarcity of electricity nationally, and a lack of snow and rain in the hydroelectric-dependent Pacific Northwest, Detmers said.
``If you are out in the community and get into an intersection that is in the blackout, use caution,'' he said.
The day began with the third Stage 3 power alert within a week, meaning reserves were close to just 1.5 percent. The warning marked at least the third time California neared blackouts since its power woes began last summer.
The ISO fended off outages before by temporarily turning off huge state pumps that move water from Northern California to the south, sucking enough power for 600,000 homes, but that wasn't enough Thursday.
Suppliers were ``reluctant to provide power to California because of the financial situation of the utilities,'' Detmers said.
He said the ISO wasn't probing whether suppliers were flat-out ignoring Energy Secretary Bill Richardson's emergency order insisting that any spare power go to California, however.
``We're just trying to get the power delivered,'' he said.
Adding to the problems, several power plants that were expected to return to full operation after repairs did not, Detmers said.
On Tuesday, Southern California Edison declared itself unable to pay hundreds of millions in wholesale electricity bills, and it and PG&E, the state's largest utilities, took another hit on Wall Street.
SoCal Edison, which serves 11 million people, said it cannot pay $596 million in bills for wholesale energy and debt service, including $215 million to the California Power Exchange.
The Power Exchange was considering whether to make the utility buy its power elsewhere and an electricity supplier threatened to force SoCal Edison into bankruptcy if it failed to pay its bills.
The default prompted Standard & Poor's and Moody's to downgrade the credit ratings of SoCal Edison and PG&E to junk-bond status.
The credit agency said SoCal Edison's delinquency also tainted PG&E. With just $500 million in cash left as of Jan. 10, PG&E faces due dates on bills totaling $1 billion during the first two weeks of February.
Between them, PG&E and SoCal Edison have lost at least $10 billion in wholesale energy costs. A rate freeze imposed as the state phases in deregulation has blocked them from passing on higher wholesale costs to their customers.
Wholesale power prices have risen dramatically since June, in part of because of a hot summer and a cold winter. In 1999, they averaged perhaps 3.5 cents a kilowatt. Now, they are running about 30 cents, and sometimes far higher.
Demand has remained high, supplies are strapped because no new power plants have been built in the state in recent years and imports are tight because other states are fighting over the power.
In addition, spiraling prices for natural gas are forcing power plants to raise their prices. Most power plants are fired by natural gas.
On Tuesday, unusually high demand for natural gas, due in part to cold weather, led San Diego Gas and Electric to cut supplies to two power plants, contributing to the state's Stage 3 alert.
The utility said there was plenty of natural gas, but not enough space in the pipeline to meet its customers' needs. To maintain the supply for its home and small-business users, the utility cut the flow to the two power plants and six large industrial customers.
The state avoided rolling blackouts after huge state pumps that move water from Northern California to the south were turned off temporarily, conserving enough electricity to power 600,000 homes, said Kellan Fluckiger, the ISO's chief operating officer.
Joel Nelsen, president of California Citrus Mutual, spent Tuesday on the phone with Central Valley lawmakers and the governor's office trying to ensure that orange growers wouldn't face outages as they tried to protect crops from a cold snap.
``We're terribly exposed,'' said Nelsen, who heads a trade association of 800 growers. ``The loss of power for a short time could wreak untold damage on our crop.''
http://www.sfgate.com/cgi-bin/article.cgi?file=/chronicle/archive/2001/01/16... describes one of the more interesting side notes in all of this, how PG & E's corporate restructuring allows them to retain profits. Among the many comical notes is that about half of the debt PG & E owes is to itself. regards, Ted Hardie
Even third world countries aren't as idiotic as California. There when electricity goes off, it usually is because there is none to be had. In our situation, it is manipulation and poor planning to the extreme.
My power was out for at least one hour, it is back now.
Bora
----- Original Message ----- From: "Rusty H. Hodge" <rusty@hodge.com> To: <nanog@merit.edu> Sent: Wednesday, January 17, 2001 1:37 PM Subject: California regulators ordered rolling blackouts
It has begun. Welcome to the 3rd world.
http://www.sfgate.com/cgi-bin/article.cgi?file=/news/archive/2001/01/17/stat e1503EST0161.DTL
Blackouts hit California as utility financial woes deepen
Associated Press, SF Gate Wednesday, January 17, 2001 Breaking News Sections
------------------------------------------------------------------------
California regulators ordered rolling blackouts Wednesday for the first time in the state's months-long electricity crisis, blaming utility credit problems and a tight national power supply for the scattered outages.
The rotating blackouts, expected to affect about 500,000 customers for an hour to 90 minutes, were restricted to Pacific Gas and Electric Co. territory in northern and part of central California, said Stephanie McCorkle, a spokeswoman for the Independent System operator, keeper of the state's power grid.
A PG&E spokesman said that about 250,000 people were already without power from the blackouts that began at 11:41 a.m. There are reports of outages in San Francisco's Lower Haight, Oakland's Rockridge area, the Oakland Hills, Orinda, the Peninsula from South San Francisco to San Mateo, parts of Napa and Sonoma County, downtown San Jose, Cupertino's De Anza College, San Ramon, Santa Cruz, Benicia and other areas, according to radio and TV reports.
There was a report of two students being trapped in an elevator at Hastings School of Law in San Francisco. Some ATMs were reported out of service in downtown San Francisco.
Power was cut to the Cow Palace in Daly City, which is holding a boat show, according to an administrative worker at the exhibition hall. The main building remains well-lit from emergency lighting and skylights, she said.
KICU-TV, channel 36, was knocked off the air.
The blackouts would first affect customers in scattered areas known as blocks 3 and 4. For security reasons, the precise locations are not released. Consumers can find their block numbers at the bottom of their power bills.
PG&E and state officials urged conservation and said it was not known yet if blackouts would be ordered for blocks 5, 6 and 7. Blocks 1 and 2 had power outages last June.
BART, fire departments, police stations and hospitals are not affected. Motorists who come to non-functioning traffic signals, including El Camino Real in the San Bruno/San Mateo area and Lawrence Expressway in San Jose, should treat them as four-way stops.
Utilities try to avoid cutting power to blocks with essential services such as hospitals.
Terry Winter, president of Cal-ISO, said that a large power plant on California's Central Coast went down at about 11 a.m., necessitating the outages.
Worry that the state's two largest utilities were on the verge of bankruptcy led some suppliers to withhold power from California, despite an emergency federal order requiring them to sell excess electricity to the state, said Jim Detmers, the ISO's managing director of operations.
But Winter said later that he did not believe generators were withholding power. Instead, he said, the main problem is broken power generating facilities, many of them older plants that have been run heavily since June.
Compounding the problem is a general scarcity of electricity nationally, and a lack of snow and rain in the hydroelectric-dependent Pacific Northwest, Detmers said.
``If you are out in the community and get into an intersection that is in the blackout, use caution,'' he said.
The day began with the third Stage 3 power alert within a week, meaning reserves were close to just 1.5 percent. The warning marked at least the third time California neared blackouts since its power woes began last summer.
The ISO fended off outages before by temporarily turning off huge state pumps that move water from Northern California to the south, sucking enough power for 600,000 homes, but that wasn't enough Thursday.
Suppliers were ``reluctant to provide power to California because of the financial situation of the utilities,'' Detmers said.
He said the ISO wasn't probing whether suppliers were flat-out ignoring Energy Secretary Bill Richardson's emergency order insisting that any spare power go to California, however.
``We're just trying to get the power delivered,'' he said.
Adding to the problems, several power plants that were expected to return to full operation after repairs did not, Detmers said.
On Tuesday, Southern California Edison declared itself unable to pay hundreds of millions in wholesale electricity bills, and it and PG&E, the state's largest utilities, took another hit on Wall Street.
SoCal Edison, which serves 11 million people, said it cannot pay $596 million in bills for wholesale energy and debt service, including $215 million to the California Power Exchange.
The Power Exchange was considering whether to make the utility buy its power elsewhere and an electricity supplier threatened to force SoCal Edison into bankruptcy if it failed to pay its bills.
The default prompted Standard & Poor's and Moody's to downgrade the credit ratings of SoCal Edison and PG&E to junk-bond status.
The credit agency said SoCal Edison's delinquency also tainted PG&E. With just $500 million in cash left as of Jan. 10, PG&E faces due dates on bills totaling $1 billion during the first two weeks of February.
Between them, PG&E and SoCal Edison have lost at least $10 billion in wholesale energy costs. A rate freeze imposed as the state phases in deregulation has blocked them from passing on higher wholesale costs to their customers.
Wholesale power prices have risen dramatically since June, in part of because of a hot summer and a cold winter. In 1999, they averaged perhaps 3.5 cents a kilowatt. Now, they are running about 30 cents, and sometimes far higher.
Demand has remained high, supplies are strapped because no new power plants have been built in the state in recent years and imports are tight because other states are fighting over the power.
In addition, spiraling prices for natural gas are forcing power plants to raise their prices. Most power plants are fired by natural gas.
On Tuesday, unusually high demand for natural gas, due in part to cold weather, led San Diego Gas and Electric to cut supplies to two power plants, contributing to the state's Stage 3 alert.
The utility said there was plenty of natural gas, but not enough space in the pipeline to meet its customers' needs. To maintain the supply for its home and small-business users, the utility cut the flow to the two power plants and six large industrial customers.
The state avoided rolling blackouts after huge state pumps that move water from Northern California to the south were turned off temporarily, conserving enough electricity to power 600,000 homes, said Kellan Fluckiger, the ISO's chief operating officer.
Joel Nelsen, president of California Citrus Mutual, spent Tuesday on the phone with Central Valley lawmakers and the governor's office trying to ensure that orange growers wouldn't face outages as they tried to protect crops from a cold snap.
``We're terribly exposed,'' said Nelsen, who heads a trade association of 800 growers. ``The loss of power for a short time could wreak untold damage on our crop.''
participants (13)
-
Bora Akyol -
Charlie Watts -
Christian Nielsen -
Daniel L. Golding -
hardie＠equinix.com -
John Fraizer -
Leo Bicknell -
Marius Strom -
Mikael Abrahamsson -
Mr. James W. Laferriere -
Rusty H. Hodge -
Sean Donelan -
Travis Pugh