All, We're seeing a bit of a weird one on our network at the moment, and wondering if anyone else has seen it. Since Friday we're seeing Apple devices (we believe it's both laptops and iPhones) responding to ARP requests for the default gateway IP with their own MAC address (i.e. ARP spoofing / MITM type attack). We're only seeing it on Apple devices, but what's more strange is that we're only seeing it where those Apple devices are connected to Cisco 1810 and 1815 APs, and where those APs are connected to a Cisco WLC running v8.5 software. If we downgrade the WLC to v8.2 the problem goes away (but v8.2 doesn't support 1815 APs, so we can't roll that out globally). We're engaged with Cisco TAC, but they're trying to deny it's their problem. Apple support are investigating, but aren't admitting to having seen it before. Has anyone else seen or heard of similar issues over the last few days? Many thanks, Simon
Can you post some packet captures? I was a network engineer on the WiFi network at SFO, for both passengers and baggage scanners, with several hundred APs. Several times we were misled by packet captures that seemed to show client traffic causing network problems, such as packet storms, but which ultimately always had some more mundane cause, like a failed DHCP server or flapping switch interface. The particular SFO network I worked on has Juniper switching and Aruba APs, so it’s not directly applicable to your ecosystem. But the complexities of interpreting packet captures may apply. -mel beckman
On Mar 14, 2019, at 5:28 AM, Simon Lockhart <simon@slimey.org> wrote:
All,
We're seeing a bit of a weird one on our network at the moment, and wondering if anyone else has seen it.
Since Friday we're seeing Apple devices (we believe it's both laptops and iPhones) responding to ARP requests for the default gateway IP with their own MAC address (i.e. ARP spoofing / MITM type attack). We're only seeing it on Apple devices, but what's more strange is that we're only seeing it where those Apple devices are connected to Cisco 1810 and 1815 APs, and where those APs are connected to a Cisco WLC running v8.5 software. If we downgrade the WLC to v8.2 the problem goes away (but v8.2 doesn't support 1815 APs, so we can't roll that out globally). We're engaged with Cisco TAC, but they're trying to deny it's their problem. Apple support are investigating, but aren't admitting to having seen it before.
Has anyone else seen or heard of similar issues over the last few days?
Many thanks,
Simon
On Thu Mar 14, 2019 at 12:53:01PM +0000, Mel Beckman wrote:
Can you post some packet captures?
I have some packet captures, but as they're from a live network, I'd rather not post them publicly.
I was a network engineer on the WiFi network at SFO, for both passengers and baggage scanners, with several hundred APs. Several times we were misled by packet captures that seemed to show client traffic causing network problems, such as packet storms, but which ultimately always had some more mundane cause, like a failed DHCP server or flapping switch interface.
Sure - we're rattling every possible other cause we can think of, including using alternative DHCP server software vendor, etc. The only thing that's reliably making the problem go away is running the APs against WLC version 8.2.
The particular SFO network I worked on has Juniper switching and Aruba APs, so it???s not directly applicable to your ecosystem. But the complexities of interpreting packet captures may apply.
I'm the sort of person who has copies of RFCs printed out on his desk. I'm fairly experienced at interpreting packet captures :) Simon
You asked if anyone else has seen this. It’s possibly going on in other networks but nobody is noticing. What symptoms brought the problem to your attention? You can sanitize the packet captures by limiting them to just the headers. The payloads are likely not useful for troubleshooting anyway, since this seems to be a Layer 2 problem. You asked for help, and sanitized packets would help people help you :) -mel
On Mar 14, 2019, at 10:02 AM, Simon Lockhart <simon@slimey.org> wrote:
On Thu Mar 14, 2019 at 12:53:01PM +0000, Mel Beckman wrote:
Can you post some packet captures?
I have some packet captures, but as they're from a live network, I'd rather not post them publicly.
I was a network engineer on the WiFi network at SFO, for both passengers and baggage scanners, with several hundred APs. Several times we were misled by packet captures that seemed to show client traffic causing network problems, such as packet storms, but which ultimately always had some more mundane cause, like a failed DHCP server or flapping switch interface.
Sure - we're rattling every possible other cause we can think of, including using alternative DHCP server software vendor, etc. The only thing that's reliably making the problem go away is running the APs against WLC version 8.2.
The particular SFO network I worked on has Juniper switching and Aruba APs, so it???s not directly applicable to your ecosystem. But the complexities of interpreting packet captures may apply.
I'm the sort of person who has copies of RFCs printed out on his desk. I'm fairly experienced at interpreting packet captures :)
Simon
Right on! https://www.tracewrangler.com/
On Mar 14, 2019, at 13:13, Mel Beckman <mel@beckman.org> wrote:
You asked if anyone else has seen this. It’s possibly going on in other networks but nobody is noticing. What symptoms brought the problem to your attention?
You can sanitize the packet captures by limiting them to just the headers. The payloads are likely not useful for troubleshooting anyway, since this seems to be a Layer 2 problem. You asked for help, and sanitized packets would help people help you :)
-mel
On Mar 14, 2019, at 10:02 AM, Simon Lockhart <simon@slimey.org> wrote:
On Thu Mar 14, 2019 at 12:53:01PM +0000, Mel Beckman wrote:
Can you post some packet captures?
I have some packet captures, but as they're from a live network, I'd rather not post them publicly.
I was a network engineer on the WiFi network at SFO, for both passengers and baggage scanners, with several hundred APs. Several times we were misled by packet captures that seemed to show client traffic causing network problems, such as packet storms, but which ultimately always had some more mundane cause, like a failed DHCP server or flapping switch interface.
Sure - we're rattling every possible other cause we can think of, including using alternative DHCP server software vendor, etc. The only thing that's reliably making the problem go away is running the APs against WLC version 8.2.
The particular SFO network I worked on has Juniper switching and Aruba APs, so it???s not directly applicable to your ecosystem. But the complexities of interpreting packet captures may apply.
I'm the sort of person who has copies of RFCs printed out on his desk. I'm fairly experienced at interpreting packet captures :)
Simon
— J. Hellenthal The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.
On Thu, Mar 14, 2019 at 7:29 AM Simon Lockhart <simon@slimey.org> wrote:
Apple devices, but what's more strange is that we're only seeing it where those Apple devices are connected to Cisco 1810 and 1815 APs, and where those APs are connected to a Cisco WLC running v8.5 software. If we downgrade the WLC to v8.2 the problem goes away (but v8.2 doesn't support 1815 APs, so we
Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy for Wake on Demand --- When a device goes to sleep, the Proxy that runs on various Apple devices is supposed to seize all the IP and MAC addresses that device had registered, so it can wait for an incoming TCP SYN, (and if one's received, then signal the sleeping device to wake up and process the connection.) Bonjour and the related mDNS protocols used for AirPlay/AirPrint/etc are built on Link-Local multicast. I wonder what would happen if some random Wireless LAN controllers malfunctioned, and decided that it would like to ignore that Link-Local restriction and proxy those packets b/w subnets anyways, as if they had been unrestricted multicast or Unicast, Possibly with the source IP address on registration Mangled to or "gateway'd" from the router's IP address. (Or perhaps they wanted to have a feature to let someone AirPlay from a different VLAN than another device?) Either way.... playing around with the source IP address on the Link-local m/c packets might accidentally get them a Default Gateway IP address registered with other workstations as a mobile device that's gone to sleep and needs a proxy. -- -JH
On Thu Mar 14, 2019 at 04:19:04PM -0500, Jimmy Hess wrote:
Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy for Wake on Demand --- When a device goes to sleep, the Proxy that runs on various Apple devices is supposed to seize all the IP and MAC addresses that device had registered, so it can wait for an incoming TCP SYN, (and if one's received, then signal the sleeping device to wake up and process the connection.)
That's a very interesting observation - when we talk to the users of the Apple devices, they quite often say that the device was 'asleep' when it was sending these 'spoofed' ARP responses.
(Or perhaps they wanted to have a feature to let someone AirPlay from a different VLAN than another device?)
Cisco Wireless does claim to have some features to 'help' Bonjour / mDNS to work better. I wonder if one of those features is misbehaving. Simon
We are running 8.5 and 1815s and I don’t think we are seeing this problem. We do have a very small number of 1810s and did see some strange behavior but it doesn’t seem to match this problem description. Is proxy arp disabled on the default gateway device? That could potentially interact strangely with the features mentioned in earlier posts and mentioned below.
On Mar 14, 2019, at 4:40 PM, Simon Lockhart <simon@slimey.org> wrote:
On Thu Mar 14, 2019 at 04:19:04PM -0500, Jimmy Hess wrote:
Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy for Wake on Demand --- When a device goes to sleep, the Proxy that runs on various Apple devices is supposed to seize all the IP and MAC addresses that device had registered, so it can wait for an incoming TCP SYN, (and if one's received, then signal the sleeping device to wake up and process the connection.)
That's a very interesting observation - when we talk to the users of the Apple devices, they quite often say that the device was 'asleep' when it was sending these 'spoofed' ARP responses.
The "Information About Passive Clients” section of this document https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-5/config-guide/b... says: "Wireless LAN controllers currently act as a proxy for ARP requests. Upon receiving an ARP request, the controller responds with an ARP response instead of passing the request directly to the client. This scenario has two advantages: • The upstream device that sends out the ARP request to the client will not know where the client is located. • Power for battery-operated devices such as mobile phones and printers is preserved because they do not have to respond to every ARP requests." Perhaps that function on version 8.5 is interacting incorrectly with the Apple Sleep Proxy feature on the Apple devices. "When a sleep proxy sees an IPv4 ARP or IPv6 ND Request for one of the sleeping device's addresses, it answers on behalf of the sleeping device, without waking it up, giving its own MAC address as the current (temporary) owner of that address.” https://en.wikipedia.org/wiki/Bonjour_Sleep_Proxy https://discussions.apple.com/thread/2160614
(Or perhaps they wanted to have a feature to let someone AirPlay from a different VLAN than another device?)
Cisco Wireless does claim to have some features to 'help' Bonjour / mDNS to work better. I wonder if one of those features is misbehaving.
Simon
--- Bruce Curtis bruce.curtis@ndsu.edu Certified NetAnalyst II 701-231-8527 North Dakota State University
participants (5)
-
Curtis, Bruce -
J. Hellenthal -
Jimmy Hess -
Mel Beckman -
Simon Lockhart