power failure causes and effects
Most of the early rumors about causes of the power failure have proven incorrect -- fire at a New York City power plant, etc. Most likely is a congestion failure in the Niagara-Mohawk grid, which covers a large part of New York state and has feeders across into Canada. Congestion failures happen when power flows through a particular switching station are high, and a component fails either directly or because of a power surge caused by a failure elsewhere. The imbalance caused by an open or short circuit will then immediately spread through the rest of the grid unless action is taken to disconnect, or point failures occur (resulting in fires and explosions from degradation of power transmission equipment, transformers, etc., not a pretty outcome at all) -- or both. In general, transmission grids are designed from a failsafe perspective, meaning that it is much safer to cause rolling brownouts or blackouts than to let key components such as transmission substations or power plants have a catastrophic failure. Since the entire grid has to be in sync and supply and demand must be in relative parity at all times, the usual strategy in these cases is to isolate the affected area, "island it" by shutting down power ingress and egress (tripping safety breakers at major crossing points), and shutting down power plants in the vicinity that will have stress failures if they don't have sufficient load to balance their output. The problem is that power travels faster than even the highest-speed switching equipment can operate, so the surges causing a cascading failure like this afternoon's can spread very quickly, like ripples in a pond. The weather was hot and humid but completely within range for the time of year, so this has to be counted as a "normal accident." It's likely that whatever component initially failed and triggered the shutdown was within its usual tolerances and simply had an ordinary breakdown. Of course, to spread, a massive outage like this also exposes other weaknesses and hidden dependencies in the system, which might be other physical components, software, operator error, etc. The system is remarkably resilient in most circumstances, which is what five-nines or more is all about. But rust never sleeps, and underinvestment in key transmission corridors in New England, New York north of Manhattan and in parts of the Midwest is no doubt an underlying cause. As to the root cause of that engineering problem -- the answer is politics, some of it congressional, and I will say no more in this forum. Fred Heutte Portland, Oregon energy policy analyst and net geek
On Thu, 14 Aug 2003 19:12:04 -0700 Fred Heutte <aoxomoxoa@sunlightdata.com> wrote: It looks like DC may have had a close call - the University of Maryland was dark for about 1 hour, and I have heard several news reports here stating that the local (DC regional) power grid just managed to decouple from wider East Coast grid in time to avoid a collapse here. Regards Marshall Eubanks
Most of the early rumors about causes of the power failure have proven incorrect -- fire at a New York City power plant, etc.
Most likely is a congestion failure in the Niagara-Mohawk grid, which covers a large part of New York state and has feeders across into Canada.
Congestion failures happen when power flows through a particular switching station are high, and a component fails either directly or because of a power surge caused by a failure elsewhere.
The imbalance caused by an open or short circuit will then immediately spread through the rest of the grid unless action is taken to disconnect, or point failures occur (resulting in fires and explosions from degradation of power transmission equipment, transformers, etc., not a pretty outcome at all) -- or both. In general, transmission grids are designed from a failsafe perspective, meaning that it is much safer to cause rolling brownouts or blackouts than to let key components such as transmission substations or power plants have a catastrophic failure.
Since the entire grid has to be in sync and supply and demand must be in relative parity at all times, the usual strategy in these cases is to isolate the affected area, "island it" by shutting down power ingress and egress (tripping safety breakers at major crossing points), and shutting down power plants in the vicinity that will have stress failures if they don't have sufficient load to balance their output.
The problem is that power travels faster than even the highest-speed switching equipment can operate, so the surges causing a cascading failure like this afternoon's can spread very quickly, like ripples in a pond.
The weather was hot and humid but completely within range for the time of year, so this has to be counted as a "normal accident." It's likely that whatever component initially failed and triggered the shutdown was within its usual tolerances and simply had an ordinary breakdown. Of course, to spread, a massive outage like this also exposes other weaknesses and hidden dependencies in the system, which might be other physical components, software, operator error, etc.
The system is remarkably resilient in most circumstances, which is what five-nines or more is all about. But rust never sleeps, and underinvestment in key transmission corridors in New England, New York north of Manhattan and in parts of the Midwest is no doubt an underlying cause.
As to the root cause of that engineering problem -- the answer is politics, some of it congressional, and I will say no more in this forum.
Fred Heutte
Portland, Oregon energy policy analyst and net geek
I have been hearing on the TV that some places that had power failure have started getting their power back, reporters say hopefully by the morning all of the places where had power failure will back online. Mehmet Akcin ----- Original Message ----- From: "Marshall Eubanks" <tme@multicasttech.com> To: "Fred Heutte" <aoxomoxoa@sunlightdata.com>; <nanog@merit.edu> Sent: Friday, August 15, 2003 12:51 AM Subject: Re: power failure causes and effects
On Thu, 14 Aug 2003 19:12:04 -0700 Fred Heutte <aoxomoxoa@sunlightdata.com> wrote:
It looks like DC may have had a close call - the University of Maryland
was
dark for about 1 hour, and I have heard several news reports here stating that the local (DC regional) power grid just managed to decouple from wider East Coast grid in time to avoid a collapse here.
Regards Marshall Eubanks
Most of the early rumors about causes of the power failure have proven incorrect -- fire at a New York City power plant, etc.
Most likely is a congestion failure in the Niagara-Mohawk grid, which covers a large part of New York state and has feeders across into Canada.
Congestion failures happen when power flows through a particular switching station are high, and a component fails either directly or because of a power surge caused by a failure elsewhere.
The imbalance caused by an open or short circuit will then immediately spread through the rest of the grid unless action is taken to disconnect, or point failures occur (resulting in fires and explosions from degradation of power transmission equipment, transformers, etc., not a pretty outcome at all) -- or both. In general, transmission grids are designed from a failsafe perspective, meaning that it is much safer to cause rolling brownouts or blackouts than to let key components such as transmission substations or power plants have a catastrophic failure.
Since the entire grid has to be in sync and supply and demand must be in relative parity at all times, the usual strategy in these cases is to isolate the affected area, "island it" by shutting down power ingress and egress (tripping safety breakers at major crossing points), and shutting down power plants in the vicinity that will have stress failures if they don't have sufficient load to balance their output.
The problem is that power travels faster than even the highest-speed switching equipment can operate, so the surges causing a cascading failure like this afternoon's can spread very quickly, like ripples in a pond.
The weather was hot and humid but completely within range for the time of year, so this has to be counted as a "normal accident." It's likely that whatever component initially failed and triggered the shutdown was within its usual tolerances and simply had an ordinary breakdown. Of course, to spread, a massive outage like this also exposes other weaknesses and hidden dependencies in the system, which might be other physical components, software, operator error, etc.
The system is remarkably resilient in most circumstances, which is what five-nines or more is all about. But rust never sleeps, and underinvestment in key transmission corridors in New England, New York north of Manhattan and in parts of the Midwest is no doubt an underlying cause.
As to the root cause of that engineering problem -- the answer is politics, some of it congressional, and I will say no more in this forum.
Fred Heutte
Portland, Oregon energy policy analyst and net geek
On Fri, 15 Aug 2003, Mehmet Akcin wrote:
I have been hearing on the TV that some places that had power failure have started getting their power back, reporters say hopefully by the morning all of the places where had power failure will back online.
In NYC it seems like either some places are running out of battery or diesel. For example I see L3 at 111 8th up fine as well as Focal at 32 Old Slip. However one of my upstreams at L3, Hurricane Electric, dropped off for a while. After about 10 minutes they were alive again but rather than the normal path out via He->MFN, they were using a L3 connection. So whatever was tying some of their sites together went dark. Additionnaly, I just got a flood of outage notifications from Covad all RFO "power outage". So either VZ is not running gensets or they shed CLECs first (nahh....). On another note, half of my NY Metro DirecTV stations are gone. I have no idea what the path is from studio -> DTV, but I assume they don't pull it off the air. Charles
Mehmet Akcin
----- Original Message ----- From: "Marshall Eubanks" <tme@multicasttech.com> To: "Fred Heutte" <aoxomoxoa@sunlightdata.com>; <nanog@merit.edu> Sent: Friday, August 15, 2003 12:51 AM Subject: Re: power failure causes and effects
On Thu, 14 Aug 2003 19:12:04 -0700 Fred Heutte <aoxomoxoa@sunlightdata.com> wrote:
It looks like DC may have had a close call - the University of Maryland
was
dark for about 1 hour, and I have heard several news reports here stating that the local (DC regional) power grid just managed to decouple from wider East Coast grid in time to avoid a collapse here.
Regards Marshall Eubanks
Most of the early rumors about causes of the power failure have proven incorrect -- fire at a New York City power plant, etc.
Most likely is a congestion failure in the Niagara-Mohawk grid, which covers a large part of New York state and has feeders across into Canada.
Congestion failures happen when power flows through a particular switching station are high, and a component fails either directly or because of a power surge caused by a failure elsewhere.
The imbalance caused by an open or short circuit will then immediately spread through the rest of the grid unless action is taken to disconnect, or point failures occur (resulting in fires and explosions from degradation of power transmission equipment, transformers, etc., not a pretty outcome at all) -- or both. In general, transmission grids are designed from a failsafe perspective, meaning that it is much safer to cause rolling brownouts or blackouts than to let key components such as transmission substations or power plants have a catastrophic failure.
Since the entire grid has to be in sync and supply and demand must be in relative parity at all times, the usual strategy in these cases is to isolate the affected area, "island it" by shutting down power ingress and egress (tripping safety breakers at major crossing points), and shutting down power plants in the vicinity that will have stress failures if they don't have sufficient load to balance their output.
The problem is that power travels faster than even the highest-speed switching equipment can operate, so the surges causing a cascading failure like this afternoon's can spread very quickly, like ripples in a pond.
The weather was hot and humid but completely within range for the time of year, so this has to be counted as a "normal accident." It's likely that whatever component initially failed and triggered the shutdown was within its usual tolerances and simply had an ordinary breakdown. Of course, to spread, a massive outage like this also exposes other weaknesses and hidden dependencies in the system, which might be other physical components, software, operator error, etc.
The system is remarkably resilient in most circumstances, which is what five-nines or more is all about. But rust never sleeps, and underinvestment in key transmission corridors in New England, New York north of Manhattan and in parts of the Midwest is no doubt an underlying cause.
As to the root cause of that engineering problem -- the answer is politics, some of it congressional, and I will say no more in this forum.
Fred Heutte
Portland, Oregon energy policy analyst and net geek
participants (4)
-
Charles Sprickman -
Fred Heutte -
Marshall Eubanks -
Mehmet Akcin