TCP session disconnection caused by Code Red?
I'm seeking the collective wisdom of this list to try to prove that I'm not halucinating. For the last few days, our network seems to be basically unreachable from the outside. Most incoming TCP sessions (web requests, incoming mail, telnet sessions, etc.) often fail with a simple "Connection refused" like nobody is listening on that port (but of course there is a server running on the port). It does not matter which service I'm trying to connect to, it also does not matter which server I'm connecting to. It happens randomly, i.e. three successive connections fail, then 5 connections succeed, then 2 connections fail, etc. Outgoing connections (i.e. our customers surfing the web) work just fine - except active FTP, which of course contains an implicit "incoming" TCP session. I've been fighting with this since friday and have done some packet traces. By comparing successful and unsuccesful packet traces I came to the conclusion that my problems are being caused by incoming TCP packets with the RST bit set. So I applied the following access list on the link to our upstream: access-list 170 deny tcp any any rst access-list 170 permit ip any any After doing this, our incoming TCP sessions magically started working. Looking at the packet counters, I see about 20% of our incoming packets are TCP RST packets. Putting the same filter on an internal link, I see about 1% of TCP RST packets. Turning on access list logging, I see that most of the packets are destined for port 80 on unused IP addresses (which are nullrouted) - which I guess is Code Red searching for victims. So now tell me, am I dreaming or has Code Red found a bug in Cisco IOS? :) Or is this some kind of new (or old?) denial of service attack disguising as Code Red? I've reported this to psirt@cisco.com and I'm waiting to see what they come up with. Anyone seen anything like this? I see the same thing with 12.0(14)S2, 12.0(17)S, 12.0(18)S and 12.2(1). Machine is a 7206 VXR, NPE-400, PA-2E3, 2FE I/O controller. Blaz Zupan, Medinet d.o.o, Trzaska 85, SI-2000 Maribor, Slovenia E-mail: blaz@amis.net, Tel: +386-2-320-6320, Fax: +386-2-320-6325
For the last few days, our network seems to be basically unreachable from the outside. Most incoming TCP sessions (web requests, incoming mail, telnet sessions, etc.) often fail with a simple "Connection refused" like nobody is
Your routers are brain dead from the load.. routers that are used to handling a few thousand connections are being asked to handle 10's of thousands. 1 good 1000+ address scan from an ISDN user kills my Lucent/Ascend TNT unless we filter for it.
Your routers are brain dead from the load.. routers that are used to handling a few thousand connections are being asked to handle 10's of thousands. 1 good 1000+ address scan from an ISDN user kills my Lucent/Ascend TNT unless we filter for it.
Hmmm, a 7206 should surely be able to handle more than 600 packets per second or am I wrong here? Our upstream E3 is currently used a maximum of 15Mbps and at peak time we see about 3000 pps on that link. If 20% of that is TCP RST packets, that would be 600 packets per second. And I'm sure somebody else on this list would be noticing this as well, especially with higher speed links. Blaz Zupan, Medinet d.o.o, Trzaska 85, SI-2000 Maribor, Slovenia E-mail: blaz@amis.net, Tel: +386-2-320-6320, Fax: +386-2-320-6325
Hmmm, a 7206 should surely be able to handle more than 600 packets per second or am I wrong here? Our upstream E3 is currently used a maximum of 15Mbps and
It's not the packets per second that seems to kill them, its the amount of arp cache and sessions (figure 600 packets per second, each packet to a different host...Thats a lot of sessions in 5 minutes)
On Mon, 6 Aug 2001, mike harrison wrote:
Hmmm, a 7206 should surely be able to handle more than 600 packets per second or am I wrong here? Our upstream E3 is currently used a maximum of 15Mbps and
It's not the packets per second that seems to kill them, its the amount of arp cache and sessions (figure 600 packets per second, each packet to a different host...Thats a lot of sessions in 5 minutes)
Curious, in that case consider null routing unused blocks, perhaps take the opportunity to improve on subnet and vlan distribution to help the null routing. Steve
It's not the packets per second that seems to kill them, its the amount of arp cache and sessions (figure 600 packets per second, each packet to a different host...Thats a lot of sessions in 5 minutes)
Curious, in that case consider null routing unused blocks, perhaps take the opportunity to improve on subnet and vlan distribution to help the null routing.
That's exactly the case. All the unused IP addresses are nullrouted and most of the traffic was destined for the nullrouted addresses. I don't think a lot of arp activity was going on. Blaz Zupan, Medinet d.o.o, Trzaska 85, SI-2000 Maribor, Slovenia E-mail: blaz@amis.net, Tel: +386-2-320-6320, Fax: +386-2-320-6325
participants (3)
-
Blaz Zupan -
mike harrison -
Stephen J. Wilcox