effects of NYC power outage
Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage late on September 11th. I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data. Thanks! Craig Partridge Chief Scientist, BBN Technologies
On Mon, 22 Jul 2002 08:05:21 -0400 Craig Partridge <craig@aland.bbn.com> wrote:
Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage late on September 11th.
Hello; To be honest, I did not see any BGP or other routing effects from the NYC fire (there were problems on Abilene this weekend, but they were due to a bad router update). My data are presented on http://www.multicasttech.com/status and are fairly coarse-grained, having a 6 hour update cycle. The 9/11 problems actually came starting on 9/13 and 14 when the battery / generator power started running out at 25 Broadway. My understanding is that the biggest problem was the inability to access the facility to refuel. My (multicast-centric) analysis of the 9/11 response was presented at Nanog 23 : http://www.nanog.org/mtg-0110/eubanks.html You should also look at the other two presentations on 9/11 and the Internet at that meeting : http://www.nanog.org/mtg-0110/agenda.html Regards Marshall Eubanks T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : tme@multicasttech.com http://www.multicasttech.com Test your network for multicast : http://www.multicasttech.com/mt/
I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data.
Thanks!
Craig Partridge Chief Scientist, BBN Technologies
You should also look at the other two presentations on 9/11 and the Internet at that meeting :
http://www.nanog.org/mtg-0110/agenda.html BGP stability was normal on 9/11. As we know only
the telephone network suffered more whereas internet remained stable. Their might have been some problems in the access because of the flash crowd problem. A particular slide from the nanog 23 presentation http://www.nanog.org/mtg-0110/ppt/misconfig/sld010.htm shows the behaviour on 9/11. Just observe closely the slide in the above link. It covers a period from a period from 8/1 to 9/26 and there was variation of 40-60 prefixes (between aug and september), except on 9/11 (there was 100 changes.) Only 0.1% of the route table was lost. BGP was more unstable during code red propagation(http://www.renesys.com/projects/bgp_instability/.) A quick peek into both the graphs will make one thing clear: *BGP is robust enough to withstand any extreme congestion.* But the question is: what can be an effective solution for access congestion on days like 9/11? __________________________________________________ Do You Yahoo!? Yahoo! Health - Feel better, live better http://health.yahoo.com
In message <20020722130741.34462.qmail@web20809.mail.yahoo.com>, senthil ayyasa my writes:
BGP stability was normal on 9/11. As we know only the telephone network suffered more whereas internet remained stable. Their might have been some problems in the access because of the flash crowd problem.
I've now seen a lot of data on 9/11 and BGP (and other metrics) and, while final results and interpretation will wait for the NRC report, I will say that the data on reachability and such like varies dramatically, depending on where measured, granularity of measurements and other issues. Thanks! Craig
[NANOG has been bouncing my attempts to reply to this thread for several days, possibly because I quoted the word "u n st a b l e" early on, apparently triggering the "un subs cribe" filter for words that start with "uns" and contain a "b".. If your posts to NANOG have been silently bounced in the past, and your network's operational issues lead you to start your posts with words like "uns tab le" or "uns ol vable" or "uns uit able" or "u nsp eak able," wonder no more. ] At any rate, about two days ago Senthil wrote:
BGP was more [un st a ble] during code red propagation(http://www.renesys.com/projects/bgp_instability/.)
A quick peek into both the graphs will make one thing clear: *BGP is robust enough to withstand any extreme congestion.*
Anyone interested in this might also like to look at our report titled "Internet Routing Behavior on 9/11 and in the Following Weeks." http://www.renesys.com/projects/reports/renesys-030502-NRC-911.pdf Note in particular the minute-by-minute changes in routing table size around critical events on pages 9 through 11. Fine time granularity is important to avoid missing all the interesting features. --jim
A side-note on why 25 Broadway lost power. I am told they had the fuel, but the "Local 3" union worker who was watching the gauges on the generator misread the dials, and a human error caused the generator to run bone dry. --Phil -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Marshall Eubanks Sent: Monday, July 22, 2002 8:27 AM To: Craig Partridge; nanog@merit.edu Subject: Re: effects of NYC power outage On Mon, 22 Jul 2002 08:05:21 -0400 Craig Partridge <craig@aland.bbn.com> wrote:
Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage
late
on September 11th.
Hello; To be honest, I did not see any BGP or other routing effects from the NYC fire (there were problems on Abilene this weekend, but they were due to a bad router update). My data are presented on http://www.multicasttech.com/status and are fairly coarse-grained, having a 6 hour update cycle. The 9/11 problems actually came starting on 9/13 and 14 when the battery / generator power started running out at 25 Broadway. My understanding is that the biggest problem was the inability to access the facility to refuel. My (multicast-centric) analysis of the 9/11 response was presented at Nanog 23 : http://www.nanog.org/mtg-0110/eubanks.html You should also look at the other two presentations on 9/11 and the Internet at that meeting : http://www.nanog.org/mtg-0110/agenda.html Regards Marshall Eubanks T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : tme@multicasttech.com http://www.multicasttech.com Test your network for multicast : http://www.multicasttech.com/mt/
I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data.
Thanks!
Craig Partridge Chief Scientist, BBN Technologies
Nope. The main generator for the 5th floor apparently ran for a while, but the radiator became clogged with garbage floating aroung in the air, and therefore couldn't cool itself, and overheated. They shut it down to prevent it from hurting itself. Fuel was another issue. On Mon, 22 Jul 2002, Phil Rosenthal wrote:
A side-note on why 25 Broadway lost power.
I am told they had the fuel, but the "Local 3" union worker who was watching the gauges on the generator misread the dials, and a human error caused the generator to run bone dry.
--Phil
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Marshall Eubanks Sent: Monday, July 22, 2002 8:27 AM To: Craig Partridge; nanog@merit.edu Subject: Re: effects of NYC power outage
On Mon, 22 Jul 2002 08:05:21 -0400 Craig Partridge <craig@aland.bbn.com> wrote:
Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage
late
on September 11th.
Hello;
To be honest, I did not see any BGP or other routing effects from the NYC fire (there were problems on Abilene this weekend, but they were due to a bad router update). My data are presented on
http://www.multicasttech.com/status
and are fairly coarse-grained, having a 6 hour update cycle.
The 9/11 problems actually came starting on 9/13 and 14 when the battery / generator power started running out at 25 Broadway. My understanding is that the biggest problem was the inability to access the facility to refuel.
My (multicast-centric) analysis of the 9/11 response was presented at Nanog 23 : http://www.nanog.org/mtg-0110/eubanks.html
You should also look at the other two presentations on 9/11 and the Internet at that meeting :
http://www.nanog.org/mtg-0110/agenda.html
Regards Marshall Eubanks
T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : tme@multicasttech.com http://www.multicasttech.com
Test your network for multicast : http://www.multicasttech.com/mt/
I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data.
Thanks!
Craig Partridge Chief Scientist, BBN Technologies
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
I have never seen the final root cause (actually direct cause, we know what the root cause was) report from Telehouse. Although I can understand why Telehouse wouldn't want to say what happened. Between replacing water pumps, reports of contanimation inside and outside the cooling system, fuel delivery delays, etc I'm not certain there was a single cause. From the outside there seemed to be multiple events, each with different direct causes. On Mon, 22 Jul 2002, Alex Rubenstein wrote:
Nope.
The main generator for the 5th floor apparently ran for a while, but the radiator became clogged with garbage floating aroung in the air, and therefore couldn't cool itself, and overheated. They shut it down to prevent it from hurting itself.
Fuel was another issue.
From what I recall, it failed due to a mechanical problem first... then after they fixed it and had it running for sometime, it ran out of fuel.
-Simon On Mon, 22 Jul 2002 11:50:29 -0400, Phil Rosenthal wrote:
A side-note on why 25 Broadway lost power.
I am told they had the fuel, but the "Local 3" union worker who was watching the gauges on the generator misread the dials, and a human error caused the generator to run bone dry.
--Phil
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Marshall Eubanks Sent: Monday, July 22, 2002 8:27 AM To: Craig Partridge; nanog@merit.edu Subject: Re: effects of NYC power outage
On Mon, 22 Jul 2002 08:05:21 -0400 Craig Partridge <craig@aland.bbn.com> wrote:
Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage
late
on September 11th.
Hello;
To be honest, I did not see any BGP or other routing effects from the NYC fire (there were problems on Abilene this weekend, but they were due to a bad router update). My data are presented on
http://www.multicasttech.com/status
and are fairly coarse-grained, having a 6 hour update cycle.
The 9/11 problems actually came starting on 9/13 and 14 when the battery / generator power started running out at 25 Broadway. My understanding is that the biggest problem was the inability to access the facility to refuel.
My (multicast-centric) analysis of the 9/11 response was presented at Nanog 23 : http://www.nanog.org/mtg-0110/eubanks.html
You should also look at the other two presentations on 9/11 and the Internet at that meeting :
http://www.nanog.org/mtg-0110/agenda.html
Regards Marshall Eubanks
T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : tme@multicasttech.com http://www.multicasttech.com
Test your network for multicast : http://www.multicasttech.com/mt/
I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data.
Thanks!
Craig Partridge Chief Scientist, BBN Technologies
Craig, We saw real hits on both Genuity and on NYC Teleglobe on Saturday. Both in latency and in packet loss. Our 9/11 graphs are visible at //order.mids.org/~peter/index.html where I put them following the event and on the NANOG 23 (Oct. 2001) site. Peter ----------------------------------------------------------- Peter H. Salus Chief Knowledge Officer, Matrix NetSystems Ste. 501W 1106 Clayton Lane Austin, TX 78723 +1 512 451-7602 -----------------------------------------------------------
participants (9)
-
Alex Rubenstein -
cowie＠renesys.com -
Craig Partridge -
Marshall Eubanks -
Peter Salus -
Phil Rosenthal -
Sean Donelan -
senthil ayyasamy -
Simon