August 2003 - Test - lists.nanog.org

151 Front St Toronto (Torix)
by Mike Tancsa 15 Aug '03

15 Aug '03

Does anyone know whats up with them ? All my peers are down now and the host for the mailing list is still off the air. ---Mike -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike(a)sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike

3 7

power failure causes and effects
by Fred Heutte 15 Aug '03

15 Aug '03

Most of the early rumors about causes of the power failure have proven incorrect -- fire at a New York City power plant, etc. Most likely is a congestion failure in the Niagara-Mohawk grid, which covers a large part of New York state and has feeders across into Canada. Congestion failures happen when power flows through a particular switching station are high, and a component fails either directly or because of a power surge caused by a failure elsewhere. The imbalance caused by an open or short circuit will then immediately spread through the rest of the grid unless action is taken to disconnect, or point failures occur (resulting in fires and explosions from degradation of power transmission equipment, transformers, etc., not a pretty outcome at all) -- or both. In general, transmission grids are designed from a failsafe perspective, meaning that it is much safer to cause rolling brownouts or blackouts than to let key components such as transmission substations or power plants have a catastrophic failure. Since the entire grid has to be in sync and supply and demand must be in relative parity at all times, the usual strategy in these cases is to isolate the affected area, "island it" by shutting down power ingress and egress (tripping safety breakers at major crossing points), and shutting down power plants in the vicinity that will have stress failures if they don't have sufficient load to balance their output. The problem is that power travels faster than even the highest-speed switching equipment can operate, so the surges causing a cascading failure like this afternoon's can spread very quickly, like ripples in a pond. The weather was hot and humid but completely within range for the time of year, so this has to be counted as a "normal accident." It's likely that whatever component initially failed and triggered the shutdown was within its usual tolerances and simply had an ordinary breakdown. Of course, to spread, a massive outage like this also exposes other weaknesses and hidden dependencies in the system, which might be other physical components, software, operator error, etc. The system is remarkably resilient in most circumstances, which is what five-nines or more is all about. But rust never sleeps, and underinvestment in key transmission corridors in New England, New York north of Manhattan and in parts of the Midwest is no doubt an underlying cause. As to the root cause of that engineering problem -- the answer is politics, some of it congressional, and I will say no more in this forum. Fred Heutte Portland, Oregon energy policy analyst and net geek

4 3

Weird attack or traffic (Was Re: The impending DDoS storm)
by Haesu 15 Aug '03

15 Aug '03

Is anyone else seeing backscatters on your network about windowsupdate.com's IP? Someone who transits through 65.123.21.137 router is sending out lots of packets to 204.79.188.11 (windowsupdate.com) in which its not currently advertised to internet as we speak. Not to mention, packets seem to be source-spoofed to 65.124.16.0/21 (our block), causing backscatter from 65.123.21.137 to our network... Any ideas/or anyone seeing similar effect? Is someone who is administrative to Qwest Communications WASH01-WAN-65-123-21 (NET-65-123-21-0-1) aware of this may be? It looks like a Qwest customer CPE router to me but I dunno.. See below for traffic snapshot.. -hc -- Sincerely, Haesu C. TowardEX Technologies, Inc. WWW: http://www.towardex.com E-mail: haesu(a)towardex.com Cell: (978) 394-2867 k00:50:22.807370 65.123.21.137 > 65.124.23.125: icmp: net 204.79.188.11 unreachable 00:50:22.891672 65.123.21.137 > 65.124.22.48: icmp: net 204.79.188.11 unreachable 00:50:22.979997 65.123.21.137 > 65.124.22.98: icmp: net 204.79.188.11 unreachable 00:50:23.047340 65.123.21.137 > 65.124.22.21: icmp: net 204.79.188.11 unreachable 00:50:23.133616 65.123.21.137 > 65.124.22.72: icmp: net 204.79.188.11 unreachable 00:50:23.520405 65.123.21.137 > 65.124.23.107: icmp: net 204.79.188.11 unreachable 00:50:23.745844 65.123.21.137 > 65.124.22.3: icmp: net 204.79.188.11 unreachable 00:50:23.829309 65.123.21.137 > 65.124.22.54: icmp: net 204.79.188.11 unreachable 00:50:24.493650 65.123.21.137 > 65.124.23.113: icmp: net 204.79.188.11 unreachable 00:50:24.530074 65.123.21.137 > 65.124.23.35: icmp: net 204.79.188.11 unreachable 00:50:24.618082 65.123.21.137 > 65.124.23.86: icmp: net 204.79.188.11 unreachable 00:47:50.611529 65.123.21.137 > 65.124.18.100: icmp: net 204.79.188.11 unreachable 00:47:50.649962 65.123.21.137 > 65.124.17.151: icmp: net 204.79.188.11 unreachable 00:47:50.711865 65.123.21.137 > 65.124.17.124: icmp: net 204.79.188.11 unreachable 00:47:50.756960 65.123.21.137 > 65.124.17.47: icmp: net 204.79.188.11 unreachable 00:47:50.826367 65.123.21.137 > 65.124.20.8: icmp: net 204.79.188.11 unreachable 00:47:52.355967 65.123.21.137 > 65.124.22.126: icmp: net 204.79.188.11 unreachable 00:47:52.587141 65.123.21.137 > 65.124.20.46: icmp: net 204.79.188.11 unreachable 00:47:53.865460 65.123.21.137 > 65.124.22.87: icmp: net 204.79.188.11 unreachable 00:48:05.250757 65.123.21.137 > 65.124.16.1: icmp: net 204.79.188.11 unreachable 00:48:05.713640 65.123.21.137 > 65.124.17.86: icmp: net 204.79.188.11 unreachable 00:48:05.841169 65.123.21.137 > 65.124.17.60: icmp: net 204.79.188.11 unreachable 00:48:06.013042 65.123.21.137 > 65.124.16.33: icmp: net 204.79.188.11 unreachable 00:48:06.549540 65.123.21.137 > 65.124.17.41: icmp: net 204.79.188.11 unreachable 00:48:06.803847 65.123.21.137 > 65.124.17.92: icmp: net 204.79.188.11 unreachable 00:48:06.981930 65.123.21.137 > 65.124.17.15: icmp: net 204.79.188.11 unreachable 00:48:07.277776 65.123.21.137 > 65.124.18.100: icmp: net 204.79.188.11 unreachable 00:48:07.343120 65.123.21.137 > 65.124.18.74: icmp: net 204.79.188.11 unreachable 00:48:07.486285 65.123.21.137 > 65.124.17.47: icmp: net 204.79.188.11 unreachable 00:48:07.569901 65.123.21.137 > 65.124.20.8: icmp: net 204.79.188.11 unreachable 00:48:08.117407 65.123.21.137 > 65.124.18.106: icmp: net 204.79.188.11 unreachable 00:48:08.356732 65.123.21.137 > 65.124.20.41: icmp: net 204.79.188.11 unreachable 00:48:08.637485 65.123.21.137 > 65.124.20.14: icmp: net 204.79.188.11 unreachable 00:48:08.944750 65.123.21.137 > 65.124.22.126: icmp: net 204.79.188.11 unreachable 00:48:08.946623 65.123.21.137 > 65.124.22.49: icmp: net 204.79.188.11 unreachable

2 2

RE: power failure causes and effects
by Mark Segal 15 Aug '03

15 Aug '03

We have power in some pops and our co.. Genie just shut off.. Wow its quiet now.. (fyi our co is in vaughan). mark -- Mark Segal Director, Network Planning FCI Broadband Tel: 905-284-4070 Fax: 416-987-4701 http://www.fcibroadband.com Futureway Communications Inc. is now FCI Broadband -----Original Message----- From: Mehmet Akcin [mailto:mehmet@akcin.net] Sent: August 15, 2003 1:10 AM To: nanog(a)merit.edu Subject: Re: power failure causes and effects I have been hearing on the TV that some places that had power failure have started getting their power back, reporters say hopefully by the morning all of the places where had power failure will back online. Mehmet Akcin ----- Original Message ----- From: "Marshall Eubanks" <tme(a)multicasttech.com> To: "Fred Heutte" <aoxomoxoa(a)sunlightdata.com>; <nanog(a)merit.edu> Sent: Friday, August 15, 2003 12:51 AM Subject: Re: power failure causes and effects > > On Thu, 14 Aug 2003 19:12:04 -0700 > Fred Heutte <aoxomoxoa(a)sunlightdata.com> wrote: > > It looks like DC may have had a close call - the University of > Maryland was > dark for about 1 hour, and I have heard several news reports here > stating that the local (DC regional) power grid just managed to > decouple from wider East Coast grid in time to avoid a collapse here. > > Regards > Marshall Eubanks > > > > > Most of the early rumors about causes of the power failure have > > proven incorrect -- fire at a New York City power plant, etc. > > > > Most likely is a congestion failure in the Niagara-Mohawk grid, > > which covers a large part of New York state and has feeders across > > into Canada. > > > > Congestion failures happen when power flows through a particular > > switching station are high, and a component fails either directly or > > because of a power surge caused by a failure elsewhere. > > > > The imbalance caused by an open or short circuit will then > > immediately spread through the rest of the grid unless action is > > taken to disconnect, or point failures occur (resulting in fires and > > explosions from degradation of power transmission equipment, > > transformers, etc., not a pretty outcome at all) -- or both. In > > general, transmission grids are designed from a failsafe > > perspective, meaning that it is much safer to cause rolling > > brownouts or blackouts than to let key components such as > > transmission substations or power plants have a catastrophic > > failure. > > > > Since the entire grid has to be in sync and supply and demand must > > be in relative parity at all times, the usual strategy in these > > cases is to isolate the affected area, "island it" by shutting down > > power ingress and egress (tripping safety breakers at major crossing > > points), and shutting down power plants in the vicinity that will > > have stress failures if they don't have sufficient load to balance > > their output. > > > > The problem is that power travels faster than even the highest-speed > > switching equipment can operate, so the surges causing a cascading > > failure like this afternoon's can spread very quickly, like ripples > > in a pond. > > > > The weather was hot and humid but completely within > > range for the time of year, so this has to be counted as a "normal > > accident." It's likely that whatever component initially failed and > > triggered the shutdown was within its usual tolerances and simply > > had an ordinary breakdown. Of course, to spread, a massive outage > > like this also exposes other weaknesses and hidden dependencies in > > the system, which might be other physical components, software, > > operator error, etc. > > > > The system is remarkably resilient in most circumstances, which is > > what five-nines or more is all about. But rust never sleeps, and > > underinvestment in key transmission corridors in New England, New > > York north of Manhattan and in parts of the Midwest is no doubt an > > underlying cause. > > > > As to the root cause of that engineering problem -- the answer is > > politics, some of it congressional, and I will say no more in this > > forum. > > > > Fred Heutte > > > > Portland, Oregon > > energy policy analyst and net geek > > >

1 0

Re: 151 Front St Toronto (Torix)
by Mike Tancsa 15 Aug '03

15 Aug '03

Just spoke with the GT/360 help desk. Apparently there were issues with the generators at 151 Front St and the A/C was either broken or just partially functioning. As a result at one point they had shut down their long haul gear as the temperature was approaching dangerous levels. Anyone have any more details ? The intake on my 7204s are at 37C vs the normal 28C. ---Mike At 11:04 PM 14/08/2003 -0400, Mike Tancsa wrote: >Does anyone know whats up with them ? All my peers are down now and the >host for the mailing list is still off the air. > > ---Mike >-------------------------------------------------------------------- >Mike Tancsa, tel +1 519 651 3400 >Sentex Communications, mike(a)sentex.net >Providing Internet since 1994 www.sentex.net >Cambridge, Ontario Canada www.sentex.net/mike

1 0

Thoughts....
by Allen McRay 15 Aug '03

15 Aug '03

Wonder what is going to happen as those millions of machines in the northeast are restarted after a little rest this afternoon and evening. Bet more than a few of them have got a bug just waiting to wake up. Be interesting to see if there is any increase in activity....... Allen

1 0

RE: 151 Front St Toronto (Torix)
by Mark Segal 15 Aug '03

15 Aug '03

They all just came back up... And the Canadians, have managed to bring power to their exchange points, by disconnecting from the American grid ;). Mark -- Mark Segal Director, Network Planning FCI Broadband Tel: 905-284-4070 Fax: 416-987-4701 http://www.fcibroadband.com Futureway Communications Inc. is now FCI Broadband -----Original Message----- From: Mark Segal Sent: August 14, 2003 11:13 PM To: 'Mike Tancsa' Subject: RE: 151 Front St Toronto (Torix) All my peers went down about an hour ago too.. (well the ones that were still up). Mark -- Mark Segal Director, Network Planning FCI Broadband Tel: 905-284-4070 Fax: 416-987-4701 http://www.fcibroadband.com Futureway Communications Inc. is now FCI Broadband -----Original Message----- From: Mike Tancsa [mailto:mike@sentex.net] Sent: August 14, 2003 11:05 PM To: nanog(a)merit.edu Subject: 151 Front St Toronto (Torix) Does anyone know whats up with them ? All my peers are down now and the host for the mailing list is still off the air. ---Mike -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike(a)sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike

1 0

RE: The impending DDoS storm
by Jason Frisvold 15 Aug '03

15 Aug '03

On Wed, 2003-08-13 at 10:55, Ingevaldson, Dan (ISS Atlanta) wrote: > More info: > > -Opens a raw socket and spoofs its source address It *appears* to us through current testing that the source address spoofed is always within the class of the current subnet... So, a spoofing filter that denies all but the local subnet may only be partially affective.. > -Randomizes its source port, but destination is always TCP/80 > -Does one DNS lookup on "windowsupdate.com" and then uses the IP > returned > -The window size is always 16384 (this might be useful) It also looks like there is no throttling at all.. it abuses as much bandwidth as it possibly can... > > Regards, > =============================== > Daniel Ingevaldson > Engineering Manager, X-Force R&D > dsi(a)iss.net > 404-236-3160 > > Internet Security Systems, Inc. > The Power to Protect > http://www.iss.net > =============================== > > > -----Original Message----- > From: Jason Frisvold [mailto:friz@corp.ptd.net] > Sent: Wednesday, August 13, 2003 10:50 AM > To: Ingevaldson, Dan (ISS Atlanta) > Cc: Stephen J. Wilcox; nanog(a)merit.edu > Subject: RE: The impending DDoS storm > > > On Wed, 2003-08-13 at 10:14, Ingevaldson, Dan (ISS Atlanta) wrote: > > It might be somewhat tricky to block TCP/80 going to > > windowsupdate.com. > > I agree... but then, who needs updates anyways.. *grin* > > > Regards, > > =============================== > > Daniel Ingevaldson > > Engineering Manager, X-Force R&D > > dsi(a)iss.net > > 404-236-3160 > > > > Internet Security Systems, Inc. > > The Power to Protect > > http://www.iss.net > > =============================== > > > > > > -----Original Message----- > > From: Stephen J. Wilcox [mailto:steve@telecomplete.co.uk] > > Sent: Wednesday, August 13, 2003 10:38 AM > > To: Jason Frisvold > > Cc: nanog(a)merit.edu > > Subject: Re: The impending DDoS storm > > > > > > > > > > On Wed, 13 Aug 2003, Jason Frisvold wrote: > > > > > All, > > > > > > What is everyone doing, if anything, to prevent the apparent > > upcoming > > > DDoS attack against Microsoft? From what I've been reading, and > > > what > > > I've been told, August 16th is the apparent start date... > > > > > > We're looking for some solution to prevent wasting our network > > > resources transporting this traffic, but at the same time trying to > > > allow legitimate through... > > > > > > So, is anyone planning on doing anything? > > > > See previous discussion on filtering... > > > > > > Other than that experience says if these things turn out to be big > > enough to cause an issue then they quickly burn themselves out anyway > > > > Steve -- --------------------------- Jason H. Frisvold Backbone Engineering Supervisor Penteledata Engineering friz(a)corp.ptd.net RedHat Engineer - RHCE # 807302349405893 Cisco Certified - CCNA # CSCO10151622 MySQL Core Certified - ID# 205982910 --------------------------- "Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world." -- Albert Einstein [1879-1955]

7 7

general? monitoring and statistics
by Jon Torrez 15 Aug '03

15 Aug '03

With the media and others flinging "... Niagara-Mohawk power grid was overloaded." Why are the power grid/$joe_powercompany line capacity stats so hard to come by? One would think a simple http(s)? request would show graphs of current/slightly delayed line capacity publicly. If not, with todays events showing international connectivity and their weaknesss change anything? Of course the technology is avaliable to do this exists but one would have to sit in a certain chair to see it. Sure, that's fine with private / managed service(s). I'm talking the grid here and their capacity buy and sell metrics by watt to $joe_powercompany. What about $joe_powercompany load sharing lines as well? Anyone running a list of URLs that are relative to this?

2 1

Private port numbers?
by Iljitsch van Beijnum 15 Aug '03

15 Aug '03

Be damned if you filter, be damned if you don't. Nice choice. I think it's time that we set aside a range of port numbers for private use. That makes all those services that have no business escaping out in the open extremely easy to filter, while at the same time not impacting any legitimate users. Services could even be assigned two port numbers: a public one and a private one. So I could use port 80 to access the web, but port 32768 + 80 (or whatever) to manage my ADSL modem over HTTP. Applications would just need a few lines of code to try either the public or the private port first (depending on the type of application and possibly some heuristics) and try the other port when there is a destination port unreachable or administratively prohibited message. (Note that what I mean here has nothing to do with what IANA calls "private" ports.)

8 11