Here at Merit we are seeing large numbers of Code Red infected hosts. These hosts may be on our regional network MichNet or they may be elsewhere out on the greater Internet. It is the port scanning of random IP address that causes problems, because the scanning in turn is causing network problems due to heavy ARP loads when the local site routers ARP for what turn out to be unused IP addresses. This is an issue when there are large blocks of IP addresses behind a router. It is less of a problem when there is a relatively small number of IP addresses behind a router (say one class C worth). Are others seeing these sorts of problems? What strategies are there for dealing with this? What we've come up with so far is blocking inbound (inbound to the site) port 80 traffic on the LAN interface of the local site router (so outbound over the LAN interface). This prevents the ARP problems. It also gives us some indication of which systems are infected. It has serious undesirable side effects (preventing legitimate Web access) and so we also have to reenable inbound port 80 access for specific IP addresses that we know are Web servers or otherwise not vulnerable to Code Red. None of this solves the problem in any real sense. It just keeps performance reasonable and buys us time to work on or get others folks to work on real solutions. To solve the Code Red problem seems to require patching the vulnerable hosts or taking the vulnerable or infected hosts offline. How long is it going to take to get every Windows NT, Windows 2000, and Windows XP system patched? We may be at this for a long time. I am not looking forward to this. Any ideas for other approaches to the problem? -Jeff Ogden Merit
On Thu, 19 Jul 2001, Jeff Ogden wrote:
How long is it going to take to get every Windows NT, Windows 2000, and Windows XP system patched? We may be at this for a long time. I am not looking forward to this.
Any ideas for other approaches to the problem?
Yes, a "helpful" virus targeting infected hosts, a list of which is easily derived from your weblogs. (I'm not seriously suggesting this, but it could be very effective.)
Patrick Greenwell writes:
On Thu, 19 Jul 2001, Jeff Ogden wrote:
How long is it going to take to get every Windows NT, Windows 2000, and Windows XP system patched? We may be at this for a long time. I am not looking forward to this.
Any ideas for other approaches to the problem?
Well this patch has been out for a month now so I would give it another 3 years before 75% of the machines are patched. I WISH I was kidding. On a side note, it is rumored that Microsoft's own windowsupdate site was defaced. go figure.
] On a side note, it is rumored that Microsoft's own windowsupdate site was ] defaced. go figure. http://www.smartguys.net/mshacked/ -- Rob Thomas http://www.cymru.com/~robt cmn_err(CE_PANIC, "Out of coffee...");
Jeff Ogden wrote:
Here at Merit we are seeing large numbers of Code Red infected hosts. These hosts may be on our regional network MichNet or they may be elsewhere out on the greater Internet. It is the port scanning of random IP address that causes problems, because the scanning in turn is causing network problems due to heavy ARP loads when the local site routers ARP for what turn out to be unused IP addresses. This is an issue when there are large blocks of IP addresses behind a router. It is less of a problem when there is a relatively small number of IP addresses behind a router (say one class C worth). Are others seeing these sorts of problems? What strategies are there for dealing with this?
Reports from our monitoring systems saw the CPU usage jump by somewhere between 150-200% for our core routers today; our current theory is that much of this was caused by excessively short and rapid flows from the probing, causing a lot of new paths to be learned (and rapidly discarded), rather than being able to just switch it through. -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://www.lightbearer.com/~lucifer
> Reports from our monitoring systems saw the CPU usage jump by somewhere > between 150-200% for our core routers today I just got off the phone with the TAC about this, and received the following _preliminary_ advice: 1) If it's not enabled, turn on CEF, to move some of the packet-forwarding load off the processor and into hardware. For some reason, a lot of this traffic is being process-switched, as evidenced by high "IP Input" cpu loads. 2) If you can, put in an ACL which prohibits port-80 traffic destined _to the interfaces of the router itself_. Since the destination IP addresses of the packets which constitute the attack itself are random, many of them will be addressed to your routers, rather than to hosts, and those will _always_ be process switched, if they're not blocked by an inbound ACL. It goes without saying that you should have a "no http server" line in any production router. -Bill
At 11:12 PM 7/19/2001, lucifer@lightbearer.com wrote:
Reports from our monitoring systems saw the CPU usage jump by somewhere between 150-200% for our core routers today; our current theory is that
Web servers that were hit beginning this morning at 11:26:41 EDT have not seen another attempt since 19:49:53. I'm wondering if this because it was coming up on 00:00:00 GMT 20-July-2001. According to the PC-Cillin write up, the 100-thread scan only takes place if the system date is less than 20, but if it's 20-28, it launches it's DOS attack at www1.whitehouse.gov Does anybody really know yet what payloads this thing is carrying?
Dave Stewart wrote:
At 11:12 PM 7/19/2001, lucifer@lightbearer.com wrote:
Reports from our monitoring systems saw the CPU usage jump by somewhere between 150-200% for our core routers today; our current theory is that
Web servers that were hit beginning this morning at 11:26:41 EDT have not seen another attempt since 19:49:53.
I'm wondering if this because it was coming up on 00:00:00 GMT 20-July-2001.
According to the PC-Cillin write up, the 100-thread scan only takes place if the system date is less than 20, but if it's 20-28, it launches it's DOS attack at www1.whitehouse.gov
Does anybody really know yet what payloads this thing is carrying?
That would roughly correspond with the dropoff in CPU usage, here. Not proof, but... reasonably strong circumstantial. I guess we'll see for sure tomorrow, depending on how the traffic stats look. -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://www.lightbearer.com/~lucifer
Does anybody really know yet what payloads this thing is carrying?
eeye had a breakdown of what it does a couple of days back: http://www.eeye.com/html/Research/Advisories/AL20010717.html Steve
On Thu, 19 Jul 2001 lucifer@lightbearer.com wrote:
Reports from our monitoring systems saw the CPU usage jump by somewhere between 150-200% for our core routers today; our current theory is that
One of our downstreams with a /20 had not nullrouted their /20, so any nets not in use bounced back to us via their default route. This caused approx 4-8 megabit of traffic on their line due to all the scanning. After our customer put in a null route for their /20, the traffic problem ceased. The ping-pong routing was causing his 2600 to use a lot of cpu. I do expect most people to null route their nets, but if someone hasn't, this can cause problems due to scanning. -- Mikael Abrahamsson email: swmike@swm.pp.se
Jeff Ogden wrote:
is causing network problems due to heavy ARP loads when the local site routers ARP for what turn out to be unused IP addresses. This is an issue when there are large blocks of IP addresses behind a router. It is less of a problem when there is a relatively small number of IP addresses behind a router (say one class C worth). Are others seeing these sorts of problems? What strategies are there for dealing with this?
If addresses are contiguous, perhaps you could blackhole some of them temporarily. It might be nice if there was a way to take a current ARP table and freeze it. That is, mark all the entries as permanent, then turn off ARP or dump destination IPs not in the ARP table into the bit bucket. As long as the router continues to respond to ARP requests, this might be a short term fix for that type of event. John
Jeff Ogden wrote:
is causing network problems due to heavy ARP loads when the local site routers ARP for what turn out to be unused IP addresses. This is an issue when there are large blocks of IP addresses behind a router. It is less of a problem when there is a relatively small number of IP addresses behind a router (say one class C worth). Are others seeing these sorts of problems? What strategies are there for dealing with this?
Use smaller subnets (possibly vlans etc) ! Steve
Jeff Ogden wrote:
is causing network problems due to heavy ARP loads when the local site routers ARP for what turn out to be unused IP addresses. This is an issue when there are large blocks of IP addresses behind a router. It is less of a problem when there is a relatively small number of IP addresses behind a router (say one class C worth). Are others seeing these sorts of problems? What strategies are there for dealing with this?
Use smaller subnets (possibly vlans etc) !
Steve
I don't clearly see how this will help.
you said you had large numbers of unused IP addresses. split the block down into manageable chunks, send the chunks to the relevant interfaces and route the whole netblock to null your used ips go out to their appropriate networks and the unused ones having nowhere to go get sent to null. So: No ARPs to spare netblocks! by splitting it into subnets you will also reduce the amuont of broadcast traffic on the network, (each bad ip will generate several broadcast arp packets) And: Better network performance, improved bandwidth! Steve On Fri, 20 Jul 2001, Larry Sheldon wrote:
Jeff Ogden wrote:
is causing network problems due to heavy ARP loads when the local site routers ARP for what turn out to be unused IP addresses. This is an issue when there are large blocks of IP addresses behind a router. It is less of a problem when there is a relatively small number of IP addresses behind a router (say one class C worth). Are others seeing these sorts of problems? What strategies are there for dealing with this?
Use smaller subnets (possibly vlans etc) !
Steve
I don't clearly see how this will help.
participants (11)
-
Bill Woodcock -
Dave Stewart -
Jeff Ogden -
John Kristoff -
Larry Sheldon -
lucifer＠lightbearer.com -
Mikael Abrahamsson -
Patrick Greenwell -
Rob Thomas -
Seth M. Kusiak -
Stephen J. Wilcox