92 Byte ICMP Blocking Problem
We started blocking 92 Byte ICMP packets on our ingress points on our core backbone routers. This was a recommendation from Cisco to help mitigate the effects of the Nachi worm. Since then, we have been hammered with customer complaints concerning the inability to talk to mail servers and ssh to their servers, as well as other weird network issues, all centering around the time we started blocking 92 Byte ICMP packets. Has anyone else seen this, and if so, is the only resolution to stop the blockage of 92 Byte ICMP Packets..? Thanks Richard
Once upon a time, Richard J.Sears <rsears@adnc.com> said:
Since then, we have been hammered with customer complaints concerning the inability to talk to mail servers and ssh to their servers, as well as other weird network issues, all centering around the time we started blocking 92 Byte ICMP packets.
Has anyone else seen this, and if so, is the only resolution to stop the blockage of 92 Byte ICMP Packets..?
Yes. As soon as we put the policy route map in place, we had some people unable to talk via SSH, SMTP, or POP3. It was random: one person here in the office couldn't SSH to a particular server. He could SSH to other servers, and the rest of us could SSH to the server he could not. We had similar experiences with SMTP and POP3. When we took the policy route map back out, the problems went away. This is with IOS 12.0(25)S1 on a 7513 doing dCEF. We put the policy route map on the FE interface linking this router to the POP core router; this router has MC-T3 interfaces and ethernets to Ascend TNTs and such. The intent was to stop the 92 byte ICMP echos from reaching the Ascend TNTs, since several of them were rebooting constantly. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
In message <20030912175258.GB616832@hiwaay.net>, Chris Adams writes:
Once upon a time, Richard J.Sears <rsears@adnc.com> said:
Since then, we have been hammered with customer complaints concerning the inability to talk to mail servers and ssh to their servers, as well as other weird network issues, all centering around the time we started blocking 92 Byte ICMP packets.
Has anyone else seen this, and if so, is the only resolution to stop the blockage of 92 Byte ICMP Packets..?
Yes. As soon as we put the policy route map in place, we had some people unable to talk via SSH, SMTP, or POP3. It was random: one person here in the office couldn't SSH to a particular server. He could SSH to other servers, and the rest of us could SSH to the server he could not. We had similar experiences with SMTP and POP3. When we took the policy route map back out, the problems went away.
This is with IOS 12.0(25)S1 on a 7513 doing dCEF. We put the policy route map on the FE interface linking this router to the POP core router; this router has MC-T3 interfaces and ethernets to Ascend TNTs and such. The intent was to stop the 92 byte ICMP echos from reaching the Ascend TNTs, since several of them were rebooting constantly.
I wonder if it's a Path MTU problem. Can you turn off Path MTU on some of the affected hosts and see if it solves the problem? --Steve Bellovin, http://www.research.att.com/~smb
Once upon a time, Steven M. Bellovin <smb@research.att.com> said:
In message <20030912175258.GB616832@hiwaay.net>, Chris Adams writes:
Yes. As soon as we put the policy route map in place, we had some people unable to talk via SSH, SMTP, or POP3. It was random: one person here in the office couldn't SSH to a particular server. He could SSH to other servers, and the rest of us could SSH to the server he could not. We had similar experiences with SMTP and POP3. When we took the policy route map back out, the problems went away.
This is with IOS 12.0(25)S1 on a 7513 doing dCEF. We put the policy route map on the FE interface linking this router to the POP core router; this router has MC-T3 interfaces and ethernets to Ascend TNTs and such. The intent was to stop the 92 byte ICMP echos from reaching the Ascend TNTs, since several of them were rebooting constantly.
I wonder if it's a Path MTU problem. Can you turn off Path MTU on some of the affected hosts and see if it solves the problem?
I don't have it in place anymore (because it caused more problems than it fixed), so I can't test this. In any case, the route map only matched 92 byte ICMP echo and ICMP echo-reply packets, which is not what PMTU uses, so it shouldn't have had a problem. Also, I know that the MTU along the path for the person in the office is the same all the way, so PMTU shouldn't come into play there. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
I had the exact same problem. As soon as I turned it on, within minutes I had customers calling that could no longer FTP into Win2k servers and some that couldn't SSH into their Linux servers. I've since turned it off as well. Are there any other known ways to block this? william ----- Original Message ----- From: "Chris Adams" <cmadams@hiwaay.net> To: "Steven M. Bellovin" <smb@research.att.com> Cc: "Nanog" <nanog@nanog.org> Sent: Friday, September 12, 2003 1:32 PM Subject: Re: 92 Byte ICMP Blocking Problem
Once upon a time, Steven M. Bellovin <smb@research.att.com> said:
In message <20030912175258.GB616832@hiwaay.net>, Chris Adams writes:
Yes. As soon as we put the policy route map in place, we had some people unable to talk via SSH, SMTP, or POP3. It was random: one
person
here in the office couldn't SSH to a particular server. He could SSH to other servers, and the rest of us could SSH to the server he could not. We had similar experiences with SMTP and POP3. When we took the policy route map back out, the problems went away.
This is with IOS 12.0(25)S1 on a 7513 doing dCEF. We put the policy route map on the FE interface linking this router to the POP core router; this router has MC-T3 interfaces and ethernets to Ascend TNTs and such. The intent was to stop the 92 byte ICMP echos from reaching the Ascend TNTs, since several of them were rebooting constantly.
I wonder if it's a Path MTU problem. Can you turn off Path MTU on some of the affected hosts and see if it solves the problem?
I don't have it in place anymore (because it caused more problems than it fixed), so I can't test this. In any case, the route map only matched 92 byte ICMP echo and ICMP echo-reply packets, which is not what PMTU uses, so it shouldn't have had a problem. Also, I know that the MTU along the path for the person in the office is the same all the way, so PMTU shouldn't come into play there. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
There were some posts to this list last week about 75xx's dropping packets outside what was specified in the ip policy route map. -- James Edwards Routing and Security jamesh@cybermesa.com At the Santa Fe Office: Internet at Cyber Mesa Store hours: 9-6 Monday through Friday Phone support 365 days till 10 pm via the Santa Fe office: 505-988-9200 or Toll Free: 888-988-2700
I believe it to be true that all policy route traffic is processor switched rather than CEF on the 75xx platform. If so, the 75xx might not be handling all it's being asked to and dropping stuff in a non-deterministic way. * james said:
There were some posts to this list last week about 75xx's dropping packets outside what was specified in the ip policy route map.
-- James Edwards Routing and Security jamesh@cybermesa.com At the Santa Fe Office: Internet at Cyber Mesa Store hours: 9-6 Monday through Friday Phone support 365 days till 10 pm via the Santa Fe office: 505-988-9200 or Toll Free: 888-988-2700
That's really weird. I've been running with route-map nachiworm permit 10 match ip address nachilist match length 92 92 set interface Null0 ip access-list extended nachilist permit icmp any any echo permit icmp any any echo-reply ip policy route-map nachiworm on transit interfaces and the virtual-templates of all our access servers that can do it properly (just blocking echo/echo-reply on the older ones that can't do the policy) and haven't heard about any customer complaints other than "I can't ping" in the places where we've blocked all echo/echo-reply. The routers doing this (7200/7500)'s are all running 12.2(1-3)S. Access servers are running mostly 12.1M or 12.2XB code. On Fri, 12 Sep 2003, William Devine, II wrote:
I had the exact same problem. As soon as I turned it on, within minutes I had customers calling that could no longer FTP into Win2k servers and some that couldn't SSH into their Linux servers. I've since turned it off as well. Are there any other known ways to block this?
----- Original Message ----- From: "Chris Adams" <cmadams@hiwaay.net> To: "Steven M. Bellovin" <smb@research.att.com> Cc: "Nanog" <nanog@nanog.org> Sent: Friday, September 12, 2003 1:32 PM Subject: Re: 92 Byte ICMP Blocking Problem
I don't have it in place anymore (because it caused more problems than it fixed), so I can't test this. In any case, the route map only matched 92 byte ICMP echo and ICMP echo-reply packets, which is not what PMTU uses, so it shouldn't have had a problem. Also, I know that the MTU along the path for the person in the office is the same all the way, so PMTU shouldn't come into play there.
---------------------------------------------------------------------- Jon Lewis *jlewis@lewis.org*| I route System Administrator | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Hi. I've been running with the service policy version and haven't seen any problem either. I did notice that it seems to block DOS traceroutes, however. John John Souvestre - Southern Star - (504) 888-3348 - www.sstar.com -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of jlewis@lewis.org Sent: Saturday, September 13, 2003 10:18 PM To: William Devine, II Cc: Nanog Subject: Re: 92 Byte ICMP Blocking Problem Importance: High That's really weird. I've been running with route-map nachiworm permit 10 match ip address nachilist match length 92 92 set interface Null0 ip access-list extended nachilist permit icmp any any echo permit icmp any any echo-reply ip policy route-map nachiworm on transit interfaces and the virtual-templates of all our access servers that can do it properly (just blocking echo/echo-reply on the older ones that can't do the policy) and haven't heard about any customer complaints other than "I can't ping" in the places where we've blocked all echo/echo-reply. The routers doing this (7200/7500)'s are all running 12.2(1-3)S. Access servers are running mostly 12.1M or 12.2XB code.
Hi Chris, We were having the same exact problem with 4 TNTs that we have. In the end, we shut off ip-route-cache on the TNTs and that fixed the problem with them. Richard On Fri, 12 Sep 2003 12:52:58 -0500 Chris Adams <cmadams@hiwaay.net> wrote:
Once upon a time, Richard J.Sears <rsears@adnc.com> said:
Since then, we have been hammered with customer complaints concerning the inability to talk to mail servers and ssh to their servers, as well as other weird network issues, all centering around the time we started blocking 92 Byte ICMP packets.
Has anyone else seen this, and if so, is the only resolution to stop the blockage of 92 Byte ICMP Packets..?
Yes. As soon as we put the policy route map in place, we had some people unable to talk via SSH, SMTP, or POP3. It was random: one person here in the office couldn't SSH to a particular server. He could SSH to other servers, and the rest of us could SSH to the server he could not. We had similar experiences with SMTP and POP3. When we took the policy route map back out, the problems went away.
This is with IOS 12.0(25)S1 on a 7513 doing dCEF. We put the policy route map on the FE interface linking this router to the POP core router; this router has MC-T3 interfaces and ethernets to Ascend TNTs and such. The intent was to stop the 92 byte ICMP echos from reaching the Ascend TNTs, since several of them were rebooting constantly.
-- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Once upon a time, Richard J.Sears <rsears@adnc.com> said:
We were having the same exact problem with 4 TNTs that we have. In the end, we shut off ip-route-cache on the TNTs and that fixed the problem with them.
We were only seeing it on some of our TNTs for some reason. I didn't turn the route cache completely off, but I did limit the size, and that solved it for us. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
participants (8)
-
Chris Adams -
james -
jlewis＠lewis.org -
John Souvestre -
Richard J.Sears -
Steve Carter -
Steven M. Bellovin -
William Devine, II