ICMP Redirects from residential customer subnets?
Last night I was troubleshooting a strange issue where Apple products (So far just MacOS and Airports) were losing internet connectivity sporadically. Originally I thought it was an IPv6 transition technology causing the problem but the customer couldn't even ping their default GW via v4. To rule out the customer mistyping/giving us wrong information on what they were seeing I attempted to verify IP connectivity from my DHCP server to them. I pinged the IP they had retrieved via DHCP earlier. What I got back were ICMP redirects interspersed with echo replies from the customer I was pinging. The redirects were of the form: "Redirect Host(New nexthop: x.y.z.23)" The nexthop being an IP of the customer I was troubleshooting. Thinking that was very odd I setup an ACL on the vlan serving that subnet to log ICMP redirects. What I found was one IP x.y.z.56 sending redirects to IPs on my network as well as several IPs outside my network. As far as I know there is no legitimate reason for a residential PC or home gateway to send ICMP redirects. There were also a few dozen other IPs on that subnet sending ICMP redirects. A majority of them had 68:7f:74 (Cisco-Linksys) OUIs. There were also some Belkins and one ASUStek OUIs. The 68:7f:74 source MACs were dispersed amongst many customers not all from the same customer. Which leads me to believe there is either a bugged Linksys firmware or an exploited Linksys home gateway causing trouble. Has anyone ever seen something like this before? Is there any reason to see ICMP redirects on a single homed residential subnet? I'm considering adding ICMP redirects to my customer edge ACL unless there is a legitimate purpose for these packets. Thanks -ML
This is expected and will happen if the consumer router receives traffic not destined for it for most consumer devices. In the Ethernet world, it's usually the result of an active MAC falling out of the table (e.g. disconnected) before the ARP entry on the router expires. The default behavior is to flood the unknown packet out every port. On a Cisco switch you would be looking at using something like UUFB (unknown unicast flood blocking). You might want to keep an eye on resource usage on your routers if you're seeing this problem. Without UUFB there is a considerable uptick in ARP and ICMP traffic caused by this behavior, usually driving up CPU. On Wed, May 9, 2012 at 10:19 AM, ML <ml@kenweb.org> wrote:
Last night I was troubleshooting a strange issue where Apple products (So far just MacOS and Airports) were losing internet connectivity sporadically.
Originally I thought it was an IPv6 transition technology causing the problem but the customer couldn't even ping their default GW via v4.
To rule out the customer mistyping/giving us wrong information on what they were seeing I attempted to verify IP connectivity from my DHCP server to them. I pinged the IP they had retrieved via DHCP earlier.
What I got back were ICMP redirects interspersed with echo replies from the customer I was pinging. The redirects were of the form:
"Redirect Host(New nexthop: x.y.z.23)" The nexthop being an IP of the customer I was troubleshooting. Thinking that was very odd I setup an ACL on the vlan serving that subnet to log ICMP redirects. What I found was one IP x.y.z.56 sending redirects to IPs on my network as well as several IPs outside my network. As far as I know there is no legitimate reason for a residential PC or home gateway to send ICMP redirects. There were also a few dozen other IPs on that subnet sending ICMP redirects. A majority of them had 68:7f:74 (Cisco-Linksys) OUIs. There were also some Belkins and one ASUStek OUIs.
The 68:7f:74 source MACs were dispersed amongst many customers not all from the same customer. Which leads me to believe there is either a bugged Linksys firmware or an exploited Linksys home gateway causing trouble.
Has anyone ever seen something like this before?
Is there any reason to see ICMP redirects on a single homed residential subnet? I'm considering adding ICMP redirects to my customer edge ACL unless there is a legitimate purpose for these packets.
Thanks -ML
-- Ray Soucy Epic Communications Specialist Phone: +1 (207) 561-3526 Networkmaine, a Unit of the University of Maine System http://www.networkmaine.net/
I've seen this reported as a bug recently with Cisco/Linksys since the device is responding to frames for which it isn't the destination MAC when it should just discard them like the below case. Not all consumer gateways do this. But absolutely agree it is the ARP/MAC age out mismatch issue that is the likely culprit. Phil On May 9, 2012, at 1:10 PM, Ray Soucy <rps@maine.edu> wrote:
This is expected and will happen if the consumer router receives traffic not destined for it for most consumer devices.
In the Ethernet world, it's usually the result of an active MAC falling out of the table (e.g. disconnected) before the ARP entry on the router expires. The default behavior is to flood the unknown packet out every port. On a Cisco switch you would be looking at using something like UUFB (unknown unicast flood blocking).
You might want to keep an eye on resource usage on your routers if you're seeing this problem. Without UUFB there is a considerable uptick in ARP and ICMP traffic caused by this behavior, usually driving up CPU.
On Wed, May 9, 2012 at 10:19 AM, ML <ml@kenweb.org> wrote:
Last night I was troubleshooting a strange issue where Apple products (So far just MacOS and Airports) were losing internet connectivity sporadically.
Originally I thought it was an IPv6 transition technology causing the problem but the customer couldn't even ping their default GW via v4.
To rule out the customer mistyping/giving us wrong information on what they were seeing I attempted to verify IP connectivity from my DHCP server to them. I pinged the IP they had retrieved via DHCP earlier.
What I got back were ICMP redirects interspersed with echo replies from the customer I was pinging. The redirects were of the form:
"Redirect Host(New nexthop: x.y.z.23)" The nexthop being an IP of the customer I was troubleshooting. Thinking that was very odd I setup an ACL on the vlan serving that subnet to log ICMP redirects. What I found was one IP x.y.z.56 sending redirects to IPs on my network as well as several IPs outside my network. As far as I know there is no legitimate reason for a residential PC or home gateway to send ICMP redirects. There were also a few dozen other IPs on that subnet sending ICMP redirects. A majority of them had 68:7f:74 (Cisco-Linksys) OUIs. There were also some Belkins and one ASUStek OUIs.
The 68:7f:74 source MACs were dispersed amongst many customers not all from the same customer. Which leads me to believe there is either a bugged Linksys firmware or an exploited Linksys home gateway causing trouble.
Has anyone ever seen something like this before?
Is there any reason to see ICMP redirects on a single homed residential subnet? I'm considering adding ICMP redirects to my customer edge ACL unless there is a legitimate purpose for these packets.
Thanks -ML
-- Ray Soucy
Epic Communications Specialist
Phone: +1 (207) 561-3526
Networkmaine, a Unit of the University of Maine System http://www.networkmaine.net/
I've seen this with fixed wireless base radios that are bridges. We had cases where a customer radio would drop offline, so the base radio would drop the mac of the customer router from it's table. New ICMP packets coming from the network would hit the wireless base radio and be flooded out to all the other client radios. Any other customer routers that were Linksys WRT54G and running a specific firmware version (can't remember the version) that had previously learned the IP of the now-missing customer router would redirect the packet back onto the network, reducing the TTL by 1. Since the wireless base would then flood the packet out again, we would see a storm of traffic that would exponentially increase until the TTL hit 0, at which time it would stop. If I remember right, I was able to cause the Linksys to stop this behavior by enabling the multicast filtering feature, although I'm not sure why that did it. Since I couldn't run around to customers and enable it on each one, I ended up filtering all ICMP from the network towards the customers. I researched the issue at the time and never found anyone else who had mentioned it. I didn't even bother to approach Linksys support. Jason On 5/9/2012 1:00 PM, Phil wrote:
I've seen this reported as a bug recently with Cisco/Linksys since the device is responding to frames for which it isn't the destination MAC when it should just discard them like the below case. Not all consumer gateways do this.
But absolutely agree it is the ARP/MAC age out mismatch issue that is the likely culprit.
Phil
On May 9, 2012, at 1:10 PM, Ray Soucy<rps@maine.edu> wrote:
This is expected and will happen if the consumer router receives traffic not destined for it for most consumer devices.
In the Ethernet world, it's usually the result of an active MAC falling out of the table (e.g. disconnected) before the ARP entry on the router expires. The default behavior is to flood the unknown packet out every port. On a Cisco switch you would be looking at using something like UUFB (unknown unicast flood blocking).
You might want to keep an eye on resource usage on your routers if you're seeing this problem. Without UUFB there is a considerable uptick in ARP and ICMP traffic caused by this behavior, usually driving up CPU.
On Wed, May 9, 2012 at 10:19 AM, ML<ml@kenweb.org> wrote:
Last night I was troubleshooting a strange issue where Apple products (So far just MacOS and Airports) were losing internet connectivity sporadically.
Originally I thought it was an IPv6 transition technology causing the problem but the customer couldn't even ping their default GW via v4.
To rule out the customer mistyping/giving us wrong information on what they were seeing I attempted to verify IP connectivity from my DHCP server to them. I pinged the IP they had retrieved via DHCP earlier.
What I got back were ICMP redirects interspersed with echo replies from the customer I was pinging. The redirects were of the form:
"Redirect Host(New nexthop: x.y.z.23)" The nexthop being an IP of the customer I was troubleshooting. Thinking that was very odd I setup an ACL on the vlan serving that subnet to log ICMP redirects. What I found was one IP x.y.z.56 sending redirects to IPs on my network as well as several IPs outside my network. As far as I know there is no legitimate reason for a residential PC or home gateway to send ICMP redirects. There were also a few dozen other IPs on that subnet sending ICMP redirects. A majority of them had 68:7f:74 (Cisco-Linksys) OUIs. There were also some Belkins and one ASUStek OUIs.
The 68:7f:74 source MACs were dispersed amongst many customers not all from the same customer. Which leads me to believe there is either a bugged Linksys firmware or an exploited Linksys home gateway causing trouble.
Has anyone ever seen something like this before?
Is there any reason to see ICMP redirects on a single homed residential subnet? I'm considering adding ICMP redirects to my customer edge ACL unless there is a legitimate purpose for these packets.
Thanks -ML
-- Ray Soucy
Epic Communications Specialist
Phone: +1 (207) 561-3526
Networkmaine, a Unit of the University of Maine System http://www.networkmaine.net/
participants (4)
-
Jason Baugher -
ML -
Phil -
Ray Soucy