Fellow Nanogers: Reports have floated across my desk in the past week, which have suggested that iPhones owned by faculty, staff and students have been flooding university campus Wi-Fi networks in parts of the country. For example, see: *"Duke Wi-Fi Crippled by Apple iPhones" *at http://www.sci-tech-today.com/story.xhtml?story_id=10200AG9NMHU Since that story first aired, and by applying a patch that was subsequently provided by Cisco, Duke has now come to see the elimination of the problem, see: "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at http://www.eweek.com/article2/0,1895,2161065,00.asp There are certain aspects of this story in which I have the most interest, and the following questions (if I may be permitted to list them) detail my concern - adequately. I wish to ask you: 1) is the iPhone an extra-ordinary device when comparing it with devices of a comparable nature, which also request ties to a Wi-Fi network, (there are many that use Wi-Fi enabled Smart-phones and PDAs on campuses -- so, why do 'they' not pose a similar problem) 2) is this problem a result of poor planning and services implementation at certain campuses, 3) is this story - a product of great exaggerations? 4) if there are technical issues indeed that permit iPhones in particular to DoS Wi-Fi nets, what can these storms be attributed to, and what can/should be done about it? If you are in a position to respond, I would like to hear from you, either publicly or privately. If there is enough group interest in the matter, I would be most happy to summarize. All the best, Robert. --
On Sat, 21 Jul 2007, Prof. Robert Mathews (OSIA) wrote:
Cisco, Duke has now come to see the elimination of the problem, see: "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at http://www.eweek.com/article2/0,1895,2161065,00.asp
Since neither Apple, Cisco nor Duke seems willing to say exactly what the problem was or what they fixed; not very surprising; it was probably a "Duh" problem unique to Duke's network. Otherwise it would be a shame for Apple, Cisco and Duke to not let other network operators that might have the same problem to know how to prevent it from recurring elsewhere.
> > Cisco, Duke has now come to see the elimination of the problem, see: > > "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at > > http://www.eweek.com/article2/0,1895,2161065,00.asp > > Since neither Apple, Cisco nor Duke seems willing to say exactly what the > problem was or what they fixed; not very surprising; it was probably a > "Duh" problem unique to Duke's network. Nope. My understanding is that it's an ARP storm, or something similar, when the iPhone roams onto a new 802.11 hotspot. Apple hasn't issued a fix yet, so Cisco had to do an emergency patch for some of their larger customers. This is just my understanding based on one conversation about it. I'd feel like an idiot saying "don't quote me" on NANOG, but... I don't have any special knowledge about it, nor personal experience of it, so... -Bill
On Sat, 2007-07-21 at 18:52 -0700, Bill Woodcock wrote:
so Cisco had to do an emergency patch for some of their larger customers.
.... or Cisco had to spend time and money getting one of their larger customers to actually apply pre-existing patches. I've see that happen all too often over the years. Never underestimate the ability of new technology to expose the weakness in older technology. -Jim P.
If that hypothesis is true, I'm surprised I haven't seen it in all the analysis I've done with it. But I don't have any Cisco AP's to play with either. On Jul 21, 2007, at 9:52 PM, Bill Woodcock wrote:
Cisco, Duke has now come to see the elimination of the problem, see: "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at http://www.eweek.com/article2/0,1895,2161065,00.asp
Since neither Apple, Cisco nor Duke seems willing to say exactly what the problem was or what they fixed; not very surprising; it was probably a "Duh" problem unique to Duke's network.
Nope. My understanding is that it's an ARP storm, or something similar, when the iPhone roams onto a new 802.11 hotspot. Apple hasn't issued a fix yet, so Cisco had to do an emergency patch for some of their larger customers. This is just my understanding based on one conversation about it. I'd feel like an idiot saying "don't quote me" on NANOG, but... I don't have any special knowledge about it, nor personal experience of it, so...
-Bill
On Jul 21, 2007, at 8:52 PM, Bill Woodcock wrote:
Cisco, Duke has now come to see the elimination of the problem, see: "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at http://www.eweek.com/article2/0,1895,2161065,00.asp
it's an ARP storm, or something similar, when the iPhone roams onto a new 802.11 hotspot. Apple hasn't issued a fix yet, so Cisco had to do an emergency patch for some of their larger customers.
As I understand, Duke is using cisco wireless controllers to run their wireless network. Apparently there is some sort of interop issue where one system was aggravating the other to cause arp floods in rfc1918 space. We've seen 116 distinct iphones so far on our campus and have had sniffers watching arps all week to look for any similar nonsense. However, we are running the AP's in autonomous (regular ios) mode without any magic central controller box. Dale -- Dale W. Carder - Network Engineer University of Wisconsin at Madison / WiscNet http://net.doit.wisc.edu/~dwcarder
Duke runs both Cisco's distributed and autonomous APs, I believe. Kevin's report on EDUCAUSE mentioned autonomous APs, but with details as hazy as they are right now, I don't dare say whether one system or another caused or received the problem. Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Dale W. Carder Sent: Sunday, July 22, 2007 2:51 PM To: Bill Woodcock Cc: Sean Donelan; North American Network Operators Group Subject: Re: iPhone and Network Disruptions ... On Jul 21, 2007, at 8:52 PM, Bill Woodcock wrote:
Cisco, Duke has now come to see the elimination of the problem, see: "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at http://www.eweek.com/article2/0,1895,2161065,00.asp
it's an ARP storm, or something similar, when the iPhone roams onto a new 802.11 hotspot. Apple hasn't issued a fix yet, so Cisco had to do an emergency patch for some of their larger customers.
As I understand, Duke is using cisco wireless controllers to run their wireless network. Apparently there is some sort of interop issue where one system was aggravating the other to cause arp floods in rfc1918 space. We've seen 116 distinct iphones so far on our campus and have had sniffers watching arps all week to look for any similar nonsense. However, we are running the AP's in autonomous (regular ios) mode without any magic central controller box. Dale -- Dale W. Carder - Network Engineer University of Wisconsin at Madison / WiscNet http://net.doit.wisc.edu/~dwcarder
On Tue, 24 Jul 2007, Frank Bulk wrote: See: http://www.cisco.com/warp/public/707/cisco-sa-20070724-arp.shtml -Hank
Duke runs both Cisco's distributed and autonomous APs, I believe. Kevin's report on EDUCAUSE mentioned autonomous APs, but with details as hazy as they are right now, I don't dare say whether one system or another caused or received the problem.
Frank
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Dale W. Carder Sent: Sunday, July 22, 2007 2:51 PM To: Bill Woodcock Cc: Sean Donelan; North American Network Operators Group Subject: Re: iPhone and Network Disruptions ...
On Jul 21, 2007, at 8:52 PM, Bill Woodcock wrote:
Cisco, Duke has now come to see the elimination of the problem, see: "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at http://www.eweek.com/article2/0,1895,2161065,00.asp
it's an ARP storm, or something similar, when the iPhone roams onto a new 802.11 hotspot. Apple hasn't issued a fix yet, so Cisco had to do an emergency patch for some of their larger customers.
As I understand, Duke is using cisco wireless controllers to run their wireless network. Apparently there is some sort of interop issue where one system was aggravating the other to cause arp floods in rfc1918 space.
We've seen 116 distinct iphones so far on our campus and have had sniffers watching arps all week to look for any similar nonsense. However, we are running the AP's in autonomous (regular ios) mode without any magic central controller box.
Dale
-- Dale W. Carder - Network Engineer University of Wisconsin at Madison / WiscNet http://net.doit.wisc.edu/~dwcarder
Adding to the random speculation pile this just arrived in my mailbox: ------------------------------------------------------------------------ -- Cisco Security Advisory: Wireless ARP Storm Vulnerabilities Advisory ID: cisco-sa-20070724-arp http://www.cisco.com/warp/public/707/cisco-sa-20070724-arp.shtml ------------------------------------------------------------------------ ---- It sounds like a badly configured pair of wireless controllers can, under fairly normal conditions, lead to an ARP storm... I have no idea if this is the actual issue that occurred at Duke, but it *is* interesting.... W On Jul 24, 2007, at 12:28 PM, Frank Bulk wrote:
Duke runs both Cisco's distributed and autonomous APs, I believe. Kevin's report on EDUCAUSE mentioned autonomous APs, but with details as hazy as they are right now, I don't dare say whether one system or another caused or received the problem.
Frank
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Dale W. Carder Sent: Sunday, July 22, 2007 2:51 PM To: Bill Woodcock Cc: Sean Donelan; North American Network Operators Group Subject: Re: iPhone and Network Disruptions ...
On Jul 21, 2007, at 8:52 PM, Bill Woodcock wrote:
Cisco, Duke has now come to see the elimination of the problem, see: "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at http://www.eweek.com/article2/0,1895,2161065,00.asp
it's an ARP storm, or something similar, when the iPhone roams onto a new 802.11 hotspot. Apple hasn't issued a fix yet, so Cisco had to do an emergency patch for some of their larger customers.
As I understand, Duke is using cisco wireless controllers to run their wireless network. Apparently there is some sort of interop issue where one system was aggravating the other to cause arp floods in rfc1918 space.
We've seen 116 distinct iphones so far on our campus and have had sniffers watching arps all week to look for any similar nonsense. However, we are running the AP's in autonomous (regular ios) mode without any magic central controller box.
Dale
-- Dale W. Carder - Network Engineer University of Wisconsin at Madison / WiscNet http://net.doit.wisc.edu/~dwcarder
-- Never criticize a man till you've walked a mile in his shoes. Then if he didn't like what you've said, he's a mile away and barefoot.
Sean Donelan wrote:
Since neither Apple, Cisco nor Duke seems willing to say exactly what the problem was or what they fixed; not very surprising; it was probably a "Duh" problem unique to Duke's network. Sean, Nanogers:
Otherwise it would be a shame for Apple, Cisco and Duke to not let other network operators that might have the same problem to know how to prevent it from recurring elsewhere. Duke CIO - Tracy Futhey's statement that "...a particular set of conditions made the Duke wireless network experience some minor and temporary disruptions in service," where the/ "deployment of a very large Cisco-based wireless network that supports multiple network
Thank you, for your responses. Given the world of NDAs and other legal instruments, it was attempting to understand if there were certain folks here in NANOG - that were aware of any particular technical shortcomings, which could have caused, or contributed to the problem. Naturally, I say this based on a personal conjecture that NANOG members may be LESS inclined to spend nearly $600 on a product they knew little about, in order to simply satisfy a "coolness factor." :-) Seriously, while I wish to not speculate, in the absence of technical details on the situation, at least on the surface, it is troubling to me that a mass marketed, personal, consumer device could have a potential such as this - to disrupt an otherwise (seemingly?) stable networked institutional environment. In a document titled: " How to Plan for User Interest in the Apple iPhone," on 27 June 2007, Gartner had issued a negative recommendation to organizations WRT to accommodating iPhone use within enterprises based on their analysis of the product lacking hooks for Outlook/Notes, and necessary security applications. Gartner also cited Apple's commitment to focus iPhone support for individual consumers rather than organizational users as a basis for issuing its negative recommendation. Gartner also went on to issue another document on 10 July 2007, titled: "iPhone First-Generation Security Is Too Weak for Enterprises," which might be of interest (at least in an informational sense) to some here as well. protocols"/ (*) seems to have been a key issue -- is frankly MORE confusing that illuminating. Is Duke, the only U.S. university campus, which has deployed a "very large Cisco-based campus wireless network" that support "multiple network protocols" ? Besides, is the 'multiple protocol' issue a 'red herring' ? By what novel/errand protocol could the iPhones flood the Duke University Wi-Fi network? NOT owning an iPhone, and lacking a technical familiarity with all of its inner workings, leaves me at a disadvantage, I am afraid. I do happen to own a nicely featured smart-phone among other Wi-Fi devices however, and remain well acquainted on just how 'that device' is likely to interfaces with Wi-Fi nets. In this respect, is the *Apple iPhone an extra-ordinary device?* I ask that question to seek clarity into the statement made by the Duke CIO, if anyone cares to comment. Quite frankly, my interest is to understand the range of *"failures in interoperability"* -- either at the device level, or at the enterprise level. Separately, I fail to see why no one is talking; particularly due to the fact that this event is effecting a first of a kind product release by Apple, and also on account of the fact that there is wide publicity now of an existing flaw in a Cisco product. I would have thought that transparently resolving this cryptogram would have built greater public confidence in those companies and respective products involved. All the best, Robert. -- * "Update on Duke's wireless network and Apple's iPhones" [see: http://www.dukenews.duke.edu/2007/07/cisco_apple.html Friday, July 20] 2007]
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Prof. Robert Mathews (OSIA) wrote:
Fellow Nanogers:
Reports have floated across my desk in the past week, which have suggested that iPhones owned by faculty, staff and students have been flooding university campus Wi-Fi networks in parts of the country. For example, see: *"Duke Wi-Fi Crippled by Apple iPhones" *at http://www.sci-tech-today.com/story.xhtml?story_id=10200AG9NMHU Since that story first aired, and by applying a patch that was subsequently provided by Cisco, Duke has now come to see the elimination of the problem, see: "*Duke Resolves iPhone, Wi-Fi Outage Problems"* at http://www.eweek.com/article2/0,1895,2161065,00.asp
There are certain aspects of this story in which I have the most interest, and the following questions (if I may be permitted to list them) detail my concern - adequately. I wish to ask you: 1) is the iPhone an extra-ordinary device when comparing it with devices of a comparable nature, which also request ties to a Wi-Fi network, (there are many that use Wi-Fi enabled Smart-phones and PDAs on campuses -- so, why do 'they' not pose a similar problem) 2) is this problem a result of poor planning and services implementation at certain campuses, 3) is this story - a product of great exaggerations? 4) if there are technical issues indeed that permit iPhones in particular to DoS Wi-Fi nets, what can these storms be attributed to, and what can/should be done about it?
Hi Robert, While I am not at liberty to discuss specifics of customer cases, I think that you will find some of the answers to your questions in a Cisco Security Advisory which was released today: http://www.cisco.com/warp/public/707/cisco-sa-20070724-arp.shtml - -Mike-
If you are in a position to respond, I would like to hear from you, either publicly or privately. If there is enough group interest in the matter, I would be most happy to summarize.
All the best, Robert. --
- -- Mike Caudill <mcaudill@cisco.com> PSIRT Incident Manager DSS PGP: 0xEBBD5271 +1.919.392.2855 / +1.919.522.4931 (cell) http://www.cisco.com/go/psirt -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGpkkoimPJSeu9UnERApprAJ9KYWlGBiSsjDUeBhtdBusbmO8BIwCfbIjs C2UXzGWZ3biS7EmZSf6hzz4= =1Ne9 -----END PGP SIGNATURE-----
participants (10)
-
Bill Woodcock -
Christian Kuhtz -
Dale W. Carder -
Frank Bulk -
Hank Nussbacher -
Jim Popovitch -
Mike Caudill -
Prof. Robert Mathews (OSIA) -
Sean Donelan -
Warren Kumari