RE: Second day of rolling blackouts starts
On Thu, 18 January 2001, "Steven J. Sobol" wrote:
Is anyone seeing lots of routing oddities? I'm not able to get to a lot of sites that I normally can, that are hosted in different places; and I'm wondering if some providers are routing around California outages.
Not that I know of. There is something squirrely going on with the root name servers, but I haven't figured it out if it is just my location or more widespread.
I have noticed east coast routers/providers are getting beat up fairly rough. I am having hard times getting my clients in Europe to see me here in San Diego. Level3 and PSInet are taking a beating hard. -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Sean Donelan Sent: Thursday, January 18, 2001 4:22 PM To: sjsobol@NorthShoreTechnologies.net Cc: nanog@merit.edu Subject: RE: Second day of rolling blackouts starts On Thu, 18 January 2001, "Steven J. Sobol" wrote:
Is anyone seeing lots of routing oddities? I'm not able to get to a lot of sites that I normally can, that are hosted in different places; and I'm wondering if some providers are routing around California outages.
Not that I know of. There is something squirrely going on with the root name servers, but I haven't figured it out if it is just my location or more widespread.
Is your network multicast enabled ? My traceroute to you shows that you home to AS 2548, which is. If so, this might be connected to the RAMEN worm. This is hosing up native multicast but good, so much so that it is affecting routers and causing some unicast problems. I heard, for example, that it is causing 4% packet loss at the Abilene NOC. RAMEN is (for the multicast enabled part of the Internet) effectively a DOS attack. Regards Marshall Eubanks James Harkins wrote:
I have noticed east coast routers/providers are getting beat up fairly rough. I am having hard times getting my clients in Europe to see me here in San Diego. Level3 and PSInet are taking a beating hard.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Sean Donelan Sent: Thursday, January 18, 2001 4:22 PM To: sjsobol@NorthShoreTechnologies.net Cc: nanog@merit.edu Subject: RE: Second day of rolling blackouts starts
On Thu, 18 January 2001, "Steven J. Sobol" wrote:
Is anyone seeing lots of routing oddities? I'm not able to get to a lot of sites that I normally can, that are hosted in different places; and I'm wondering if some providers are routing around California outages.
Not that I know of. There is something squirrely going on with the root name servers, but I haven't figured it out if it is just my location or more widespread.
-- Multicast Technologies, Inc. 10301 Democracy Lane, Suite 201 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : tme@on-the-i.com http://www.on-the-i.com
I'm betting on it being blackout related: Starts getting ugly between Dallas and LA and just goes down hill from there: 9 dfw3-core4-pos7-0.atlas.icix.net (165.117.48.134) [AS 2548] 36 msec 36 msec 36 msec 10 lax2-core2-pos7-0.atlas.icix.net (165.117.48.42) [AS 2548] 68 msec 64 msec 68 msec --- John Fraizer EnterZone, Inc On Thu, 18 Jan 2001, Marshall Eubanks wrote:
Is your network multicast enabled ? My traceroute to you shows that you home to AS 2548, which is.
If so, this might be connected to the RAMEN worm. This is hosing up native multicast but good, so much so that it is affecting routers and causing some unicast problems. I heard, for example, that it is causing 4% packet loss at the Abilene NOC. RAMEN is (for the multicast enabled part of the Internet) effectively a DOS attack.
Regards Marshall Eubanks
James Harkins wrote:
I have noticed east coast routers/providers are getting beat up fairly rough. I am having hard times getting my clients in Europe to see me here in San Diego. Level3 and PSInet are taking a beating hard.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Sean Donelan Sent: Thursday, January 18, 2001 4:22 PM To: sjsobol@NorthShoreTechnologies.net Cc: nanog@merit.edu Subject: RE: Second day of rolling blackouts starts
On Thu, 18 January 2001, "Steven J. Sobol" wrote:
Is anyone seeing lots of routing oddities? I'm not able to get to a lot of sites that I normally can, that are hosted in different places; and I'm wondering if some providers are routing around California outages.
Not that I know of. There is something squirrely going on with the root name servers, but I haven't figured it out if it is just my location or more widespread.
--
Multicast Technologies, Inc. 10301 Democracy Lane, Suite 201 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : tme@on-the-i.com http://www.on-the-i.com
Two people have asked me off list about the RAMEN worm, which affects Linux Redhat distro's. Here is brief description of the worm, and a link to more, from Lucy Lynch at Internet2 / UOregon. The multicast implications : This worm scans a portion of the multicast address space. These scans (packets) are viewed as new multicast sources by a PIM multicast enabled router, which encapsulates them and sends them to its RP. The RP creates MSDP Session Announcements FOR EACH SCAN and floods them to every RP neighbor it has in "nearby" AS's, and those repeat the process. The result is a MSDP packet storm. We have gotten 15,000 SA's a minute. Dealing with these can melt down routers. (We had to reboot a Cisco 7204, for example, which apparently either filled up or fragmented its memory beyond usability.) I think it is fair to say that the question of rate limiting and other DOS filtering in PIM/SSM/MSDP multicast is getting serious attention now. Marshall Eubanks "Lucy E. Lynch" wrote:
a bit more info on ramen here:
http://members.home.net/dtmartin24/ramen_worm.txt
"And now, the contents of that ramen.tgz file: All the binaries are in the archive twice, with RedHat 6.2 and RedHat 7.0 versions. Numerous binaries were not stripped, which makes the job of taking them apart easier."
asp: An xinetd config. file that will start up the fake webserver Used on RedHat 7.0 victim machines. asp62: HTTP/0.9-compatible server that always serves out the file /tmp/ramen.tgz to any request - NOT stripped asp7: RedHat 7-compiled version - NOT stripped bd62.sh: Does the setup (installing wormserver, removing vulnerable programs, adding ftp users) for RedHat 6.2 bd7.sh: Same for RedHat 7.0 getip.sh: Utility script to get the main external IP address hackl.sh: Driver to read the .l file and pass addresses to lh.sh hackw.sh: Driver to read the .w file and pass addresses to wh.sh index.html: HTML document text l62: LPRng format string exploit program - NOT stripped l7: Same but compiled for RedHat 7 - stripped lh.sh: Driver script to execute the LPRng exploit with several different options randb62: Picks a random class-B subnet to scan on - NOT stripped randb7: Same but compiled for RedHat 7 - NOT stripped s62: statdx exploit - NOT stripped s7: Same but compiled for RedHat 7 - stripped scan.sh: get a classB network from randb and run synscan start.sh: Replace any index.html with the one from the worm; run getip; determine if we're RedHat 6.2 or 7.0 and run the appropriate bd*.sh and start*.sh start62.sh: start (backgrounded) scan.sh, hackl.sh, and hackw.sh start7.sh: Same as start62.sh synscan62: Modified synscan tool - records to .w and .l files - stripped synscan7: Same but compiled for RedHat 7 - stripped w62: venglin wu-ftpd exploit - stripped w7: Same but compiled for RedHat 7 - stripped wh.sh: Driver script to call the "s" and "w" binaries against a given target wu62: Apparently only included by mistake. "strings" shows it to be very similar to w62; nowhere is this binary ever invoked.
Lucy E. Lynch Academic User Services Computing Center University of Oregon llynch@darkwing.uoregon.edu (541) 346-1774 Cell: (541) 912-7998 5419127998@mobile.att.net
At 09:43 19/01/01 -0500, Marshall Eubanks wrote:
Two people have asked me off list about the RAMEN worm, which affects Linux Redhat distro's. Here is brief description of the worm, and a link to more, from Lucy Lynch at Internet2 / UOregon.
The multicast implications :
This worm scans a portion of the multicast address space. These scans (packets) are viewed as new multicast sources by a PIM multicast enabled router, which encapsulates them and sends them to its RP. The RP creates MSDP Session Announcements FOR EACH SCAN and floods them to every RP neighbor it has in "nearby" AS's, and those repeat the process. The result is a MSDP packet storm. We have gotten 15,000 SA's a minute. Dealing with these can melt down routers. (We had to reboot a Cisco 7204, for example, which apparently either filled up or fragmented its memory beyond usability.)
I think it is fair to say that the question of rate limiting and other DOS filtering in PIM/SSM/MSDP multicast is getting serious attention now.
I have installed on my multicast tunnels (one to StarTap and the other to Dante/Quantum): rate-limit input access-group 180 128000 30000 30000 conform-action transmit exceed-action drop ! access-list 180 permit tcp any any eq 639 access-list 180 permit udp any any eq 639 access-list 180 deny ip any any IANA has MSDP listed as port 639 - tcp+udp. It appears MSDP is only really TCP: mcast#sho access-l 180 Extended IP access list 180 permit tcp any any eq 639 (1555 matches) permit udp any any eq 639 deny ip any any (37888 matches) and mcast#sho in rate Tunnel1 Mbone tunnel to Dante Input matches: access-group 180 params: 128000 bps, 30000 limit, 30000 extended limit conformed 755 packets, 60044 bytes; action: transmit exceeded 0 packets, 0 bytes; action: drop last packet: 388ms ago, current burst: 0 bytes last cleared 00:07:26 ago, conformed 1000 bps, exceeded 0 bps Tunnel2 Mbone tunnel to Startap Input matches: access-group 180 params: 128000 bps, 30000 limit, 30000 extended limit conformed 909 packets, 148937 bytes; action: transmit exceeded 0 packets, 0 bytes; action: drop last packet: 1048ms ago, current burst: 0 bytes last cleared 00:08:48 ago, conformed 2000 bps, exceeded 0 bps I'll only know tomorrow if I stop getting the constant: Jan 21 14:00:03: %SYS-3-CPUHOG: Task ran for 6300 msec (123/75), process = MSDP Process, PC = 60790390. -Traceback= 60790398 604146B4 604146A0 error messages. I don't know whether 128kb/sec of MSDP is too much or too little. -Hank
Marshall Eubanks
"Lucy E. Lynch" wrote:
a bit more info on ramen here:
http://members.home.net/dtmartin24/ramen_worm.txt
"And now, the contents of that ramen.tgz file: All the binaries are in the archive twice, with RedHat 6.2 and RedHat 7.0 versions. Numerous binaries were not stripped, which makes the job of taking them apart easier."
asp: An xinetd config. file that will start up the fake webserver Used on RedHat 7.0 victim machines. asp62: HTTP/0.9-compatible server that always serves out the file /tmp/ramen.tgz to any request - NOT stripped asp7: RedHat 7-compiled version - NOT stripped bd62.sh: Does the setup (installing wormserver, removing vulnerable programs, adding ftp users) for RedHat 6.2 bd7.sh: Same for RedHat 7.0 getip.sh: Utility script to get the main external IP address hackl.sh: Driver to read the .l file and pass addresses to lh.sh hackw.sh: Driver to read the .w file and pass addresses to wh.sh index.html: HTML document text l62: LPRng format string exploit program - NOT stripped l7: Same but compiled for RedHat 7 - stripped lh.sh: Driver script to execute the LPRng exploit with several different options randb62: Picks a random class-B subnet to scan on - NOT stripped randb7: Same but compiled for RedHat 7 - NOT stripped s62: statdx exploit - NOT stripped s7: Same but compiled for RedHat 7 - stripped scan.sh: get a classB network from randb and run synscan start.sh: Replace any index.html with the one from the worm; run getip; determine if we're RedHat 6.2 or 7.0 and run the appropriate bd*.sh and start*.sh start62.sh: start (backgrounded) scan.sh, hackl.sh, and hackw.sh start7.sh: Same as start62.sh synscan62: Modified synscan tool - records to .w and .l files - stripped synscan7: Same but compiled for RedHat 7 - stripped w62: venglin wu-ftpd exploit - stripped w7: Same but compiled for RedHat 7 - stripped wh.sh: Driver script to call the "s" and "w" binaries against a given target wu62: Apparently only included by mistake. "strings" shows it to be very similar to w62; nowhere is this binary ever invoked.
Lucy E. Lynch Academic User Services Computing Center University of Oregon llynch@darkwing.uoregon.edu (541) 346-1774 Cell: (541) 912-7998 5419127998@mobile.att.net
Cisco is scheduled to have a patch for IOS 12.0(15)S that will all you to limit the number of SA's received from a peer (similar to prefix-limit on bgp session) from what I understand. You should talk to your cisco reps about that ability in your software. - Jared On Sun, Jan 21, 2001 at 03:16:37PM +0200, Hank Nussbacher wrote:
At 09:43 19/01/01 -0500, Marshall Eubanks wrote:
Two people have asked me off list about the RAMEN worm, which affects Linux Redhat distro's. Here is brief description of the worm, and a link to more, from Lucy Lynch at Internet2 / UOregon.
The multicast implications :
This worm scans a portion of the multicast address space. These scans (packets) are viewed as new multicast sources by a PIM multicast enabled router, which encapsulates them and sends them to its RP. The RP creates MSDP Session Announcements FOR EACH SCAN and floods them to every RP neighbor it has in "nearby" AS's, and those repeat the process. The result is a MSDP packet storm. We have gotten 15,000 SA's a minute. Dealing with these can melt down routers. (We had to reboot a Cisco 7204, for example, which apparently either filled up or fragmented its memory beyond usability.)
I think it is fair to say that the question of rate limiting and other DOS filtering in PIM/SSM/MSDP multicast is getting serious attention now.
I have installed on my multicast tunnels (one to StarTap and the other to Dante/Quantum):
rate-limit input access-group 180 128000 30000 30000 conform-action transmit exceed-action drop ! access-list 180 permit tcp any any eq 639 access-list 180 permit udp any any eq 639 access-list 180 deny ip any any
IANA has MSDP listed as port 639 - tcp+udp. It appears MSDP is only really TCP: mcast#sho access-l 180 Extended IP access list 180 permit tcp any any eq 639 (1555 matches) permit udp any any eq 639 deny ip any any (37888 matches)
and
mcast#sho in rate Tunnel1 Mbone tunnel to Dante Input matches: access-group 180 params: 128000 bps, 30000 limit, 30000 extended limit conformed 755 packets, 60044 bytes; action: transmit exceeded 0 packets, 0 bytes; action: drop last packet: 388ms ago, current burst: 0 bytes last cleared 00:07:26 ago, conformed 1000 bps, exceeded 0 bps Tunnel2 Mbone tunnel to Startap Input matches: access-group 180 params: 128000 bps, 30000 limit, 30000 extended limit conformed 909 packets, 148937 bytes; action: transmit exceeded 0 packets, 0 bytes; action: drop last packet: 1048ms ago, current burst: 0 bytes last cleared 00:08:48 ago, conformed 2000 bps, exceeded 0 bps
I'll only know tomorrow if I stop getting the constant: Jan 21 14:00:03: %SYS-3-CPUHOG: Task ran for 6300 msec (123/75), process = MSDP Process, PC = 60790390. -Traceback= 60790398 604146B4 604146A0 error messages. I don't know whether 128kb/sec of MSDP is too much or too little.
-Hank
Marshall Eubanks
"Lucy E. Lynch" wrote:
a bit more info on ramen here:
http://members.home.net/dtmartin24/ramen_worm.txt
"And now, the contents of that ramen.tgz file: All the binaries are in the archive twice, with RedHat 6.2 and RedHat 7.0 versions. Numerous binaries were not stripped, which makes the job of taking them apart easier."
asp: An xinetd config. file that will start up the fake webserver Used on RedHat 7.0 victim machines. asp62: HTTP/0.9-compatible server that always serves out the file /tmp/ramen.tgz to any request - NOT stripped asp7: RedHat 7-compiled version - NOT stripped bd62.sh: Does the setup (installing wormserver, removing vulnerable programs, adding ftp users) for RedHat 6.2 bd7.sh: Same for RedHat 7.0 getip.sh: Utility script to get the main external IP address hackl.sh: Driver to read the .l file and pass addresses to lh.sh hackw.sh: Driver to read the .w file and pass addresses to wh.sh index.html: HTML document text l62: LPRng format string exploit program - NOT stripped l7: Same but compiled for RedHat 7 - stripped lh.sh: Driver script to execute the LPRng exploit with several different options randb62: Picks a random class-B subnet to scan on - NOT stripped randb7: Same but compiled for RedHat 7 - NOT stripped s62: statdx exploit - NOT stripped s7: Same but compiled for RedHat 7 - stripped scan.sh: get a classB network from randb and run synscan start.sh: Replace any index.html with the one from the worm; run getip; determine if we're RedHat 6.2 or 7.0 and run the appropriate bd*.sh and start*.sh start62.sh: start (backgrounded) scan.sh, hackl.sh, and hackw.sh start7.sh: Same as start62.sh synscan62: Modified synscan tool - records to .w and .l files - stripped synscan7: Same but compiled for RedHat 7 - stripped w62: venglin wu-ftpd exploit - stripped w7: Same but compiled for RedHat 7 - stripped wh.sh: Driver script to call the "s" and "w" binaries against a given target wu62: Apparently only included by mistake. "strings" shows it to be very similar to w62; nowhere is this binary ever invoked.
Lucy E. Lynch Academic User Services Computing Center University of Oregon llynch@darkwing.uoregon.edu (541) 346-1774 Cell: (541) 912-7998 5419127998@mobile.att.net
-- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine. END OF LINE | Manager of IP networks built within my own home
participants (6)
-
Hank Nussbacher -
James Harkins -
Jared Mauch -
John Fraizer -
Marshall Eubanks -
Sean Donelan