i have no details regarding the ios vulnerability other than what has already been stated on-list, but the IOS matrix obtained this evening and listed at http://www.0ptical.net/cisco.html shows what versions are affected, and what to upgrade to resolve the mystery issue. not sure why psirt is keeping this under wraps, since most NSPs are publicly scheduling "emergency upgrades" to fix "network problems" that arent being detailed to customers, and those same customers can and will be affected by the same problem. thx, JT _______________________________________________ Join Excite! - http://www.excite.com The most personalized portal on the Web!
Cisco has posted information regarding this issue and work arounds. 12.3 based code does not exhibit this problem. Cisco Security Advisory: Cisco IOS Interface Blocked by IPv4 Packet http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml - Darrell -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of John Timmons Sent: Wednesday, July 16, 2003 9:20 PM To: nanog@merit.edu Subject: Cisco IOS Vulnerability i have no details regarding the ios vulnerability other than what has already been stated on-list, but the IOS matrix obtained this evening and listed at http://www.0ptical.net/cisco.html shows what versions are affected, and what to upgrade to resolve the mystery issue. not sure why psirt is keeping this under wraps, since most NSPs are publicly scheduling "emergency upgrades" to fix "network problems" that arent being detailed to customers, and those same customers can and will be affected by the same problem. thx, JT _______________________________________________ Join Excite! - http://www.excite.com The most personalized portal on the Web!
On Wed, Jul 16, 2003 at 10:11:49PM -0500, Darrell Kristof wrote:
Cisco has posted information regarding this issue and work arounds. 12.3 based code does not exhibit this problem.
Cisco Security Advisory: Cisco IOS Interface Blocked by IPv4 Packet http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml
I'm not sure how many of you have seen cases of a stuck input or output queue on an interface in the past as well, seems like cisco needs a "clear queue" command. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Wednesday 16 July 2003 23:18, Jared Mauch wrote:
On Wed, Jul 16, 2003 at 10:11:49PM -0500, Darrell Kristof wrote:
Cisco has posted information regarding this issue and work arounds. 12.3 based code does not exhibit this problem.
Cisco Security Advisory: Cisco IOS Interface Blocked by IPv4 Packet http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.sh tml
I'm not sure how many of you have seen cases of a stuck input or output queue on an interface in the past as well, seems like cisco needs a "clear queue" command.
- Jared
there is one - 'reload' ;) the disturbing part of this advisory is that i can do something very similar to one of my routers....and heretofore cisco was unable to tell me what was wrong anyone have the 'scheduled maintenance" mp3 lying around? i have a feeling i am going to need it /joshua -- What difference does it make to the dead, the orphans, and the homeless, whether the mad destruction is brought under the name of totalitarianism or the holy name of liberty and democracy? - Gandhi -
On Wednesday, July 16, 2003, at 11:34 PM, joshua sahala wrote:
anyone have the 'scheduled maintenance" mp3 lying around? i have a feeling i am going to need it
This wouldn't be the "My gig port's down, and now it's up again..." song would it? :) If not, pass along the right one when you find it, will ya?
On Thu, Jul 17, 2003 at 01:02:42AM -0400, Jason Lixfeld wrote:
On Wednesday, July 16, 2003, at 11:34 PM, joshua sahala wrote:
anyone have the 'scheduled maintenance" mp3 lying around? i have a feeling i am going to need it
This wouldn't be the "My gig port's down, and now it's up again..." song would it? :)
If not, pass along the right one when you find it, will ya?
1) I didn't make this 2) I cna't remmber where i got it from 3) please don't abuse my connection too much tonight http://puck.nether.net/~jared/gigflapping.mp3 - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Thu, 17 Jul 2003, Jared Mauch wrote:
On Thu, Jul 17, 2003 at 01:02:42AM -0400, Jason Lixfeld wrote:
On Wednesday, July 16, 2003, at 11:34 PM, joshua sahala wrote:
anyone have the 'scheduled maintenance" mp3 lying around? i have a feeling i am going to need it
This wouldn't be the "My gig port's down, and now it's up again..." song would it? :)
If not, pass along the right one when you find it, will ya?
1) I didn't make this 2) I cna't remmber where i got it from 3) please don't abuse my connection too much tonight
don't abuse Jared, abuse me: ftp://mirrors.secsup.org/tmp/gigflapping.mp3 it should be completely there in a few minutes.
So that was the one... On Thursday, July 17, 2003, at 1:09 AM, Jared Mauch wrote:
On Thu, Jul 17, 2003 at 01:02:42AM -0400, Jason Lixfeld wrote:
On Wednesday, July 16, 2003, at 11:34 PM, joshua sahala wrote:
anyone have the 'scheduled maintenance" mp3 lying around? i have a feeling i am going to need it
This wouldn't be the "My gig port's down, and now it's up again..." song would it? :)
If not, pass along the right one when you find it, will ya?
1) I didn't make this 2) I cna't remmber where i got it from 3) please don't abuse my connection too much tonight
http://puck.nether.net/~jared/gigflapping.mp3
- jared
-- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
| -----Original Message----- | From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of | Jared Mauch | Sent: Thursday, July 17, 2003 1:10 AM | To: Jason Lixfeld | Cc: joshua sahala; 'nanog@merit.edu' | Subject: Re: Cisco IOS Vulnerability | | | On Thu, Jul 17, 2003 at 01:02:42AM -0400, Jason Lixfeld wrote: | > | > | > On Wednesday, July 16, 2003, at 11:34 PM, joshua sahala wrote: | > | > >anyone have the 'scheduled maintenance" mp3 lying around? i have a | > >feeling i am going to need it | > | > This wouldn't be the "My gig port's down, and now it's up again..." | > song would it? :) | > | > If not, pass along the right one when you find it, will ya? | | 1) I didn't make this | 2) I cna't remmber where i got it from | 3) please don't abuse my connection too much tonight | | http://puck.nether.net/~jared/gigflapping.mp3 That link is returning a 403. Here's a copy on one of my boxes: http://www.ciphin.com/nanog/gigflapping.mp3 Todd -- | | - jared | | -- | Jared Mauch | pgp key available via finger from jared@puck.nether.net | clue++; | http://puck.nether.net/~jared/ My statements are only | mine.
On Thu, 17 Jul 2003 01:09:36 -0400, Jared Mauch <jared@puck.Nether.net> wrote:
Mirrored at http://www.netacc.net/~rtucker/gigflapping.mp3 ... same disclaimers as Jared gives, but I have more bandwidth. :-) -rt (what do you mean I need a new chassis?) -- Ryan Tucker <rtucker@netacc.net>
1) I didn't make this 2) I cna't remmber where i got it from 3) please don't abuse my connection too much tonight
There is another thing to play when reloading boxes, above disclaimers 1 and 2 apply. http://www.he.iki.fi/favorites.mpeg Pete
On Thu, 17 Jul 2003, Jason Lixfeld wrote:
This wouldn't be the "My gig port's down, and now it's up again..." song would it? :)
Folks may remember when ISPs were responding to the SNMP vulnerability many backbones were rebooting their routers during maintenance windows. At the time, some people monitoring BGP and other things thought the Internet was under attack because a huge portion of the net bounced early in the morning. In reality it was just one backbone during a global router reboot. Don't panic if you see BGP flaps from backbones during the next few weeks.
On Wed, 16 Jul 2003, Darrell Kristof wrote:
Cisco Security Advisory: Cisco IOS Interface Blocked by IPv4 Packet http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml
IS anyone seeing this exploited in the wild? It'd be good to know if we need to do panic upgrade or can schedule it for our next maintenance window (which is during the weekend). -- Mikael Abrahamsson email: swmike@swm.pp.se
On Thu, Jul 17, 2003 at 07:48:24AM +0200, Mikael Abrahamsson wrote:
On Wed, 16 Jul 2003, Darrell Kristof wrote:
Cisco Security Advisory: Cisco IOS Interface Blocked by IPv4 Packet http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml
IS anyone seeing this exploited in the wild? It'd be good to know if we need to do panic upgrade or can schedule it for our next maintenance window (which is during the weekend).
I've been keeping my ear close to the ground. A number of people have been attempting to find the packet to better place ACLs in the internet community, but i've also heard of people seeing more series of "unusual" packets on their network in the past few days as well. Nobody has found it yet that i'm aware of and Cisco found this in internal testing so I expect you will be safe for a period of time sufficent to do weekend upgrades. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Thu, 17 Jul 2003, Mikael Abrahamsson wrote:
On Wed, 16 Jul 2003, Darrell Kristof wrote:
Cisco Security Advisory: Cisco IOS Interface Blocked by IPv4 Packet http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml
IS anyone seeing this exploited in the wild? It'd be good to know if we need to do panic upgrade or can schedule it for our next maintenance window (which is during the weekend).
According to the cisco advisory, there are no reports of public knowledge of the exploit nor has anyone been detected using the exploit. Since Cisco is keeping the packet information confidential, you can't program an IDS to detect it (i.e. no signature is available). But if your router does hang up, the cisco advisory includes information about checking if you've been hit by this bug; versus the numerous other bugs :-( Cisco stated if they receive any reports of the exploit in the wild, they will re-issue the advisory with the updated information.
The workaround for transit suggests permitting only tcp, udp, icmp, gre, esp, and ah protocols. Is this sufficient to protect the router itself, or do you have to get hard-nosed with specific ACLs (restricting access to all your possible interface addresses)? Jeff
Sean Donelan wrote:
Cisco stated if they receive any reports of the exploit in the wild, they will re-issue the advisory with the updated information.
Sendmail root exploit took less than 24 hours to craft. I suspect that this exploit will be found within 48 hours. Enough information was provided to quickly guess where the problem lies with IPv4 processing. -Jack
On Thu, 17 Jul 2003, Jack Bates wrote:
Sean Donelan wrote:
Cisco stated if they receive any reports of the exploit in the wild, they will re-issue the advisory with the updated information.
Sendmail root exploit took less than 24 hours to craft. I suspect that this exploit will be found within 48 hours. Enough information was provided to quickly guess where the problem lies with IPv4 processing.
Sendmail is open source, IOS is not. Knowing where the problem is and knowing how to exploit it are two entirely different situations. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
On Thursday, Jul 17, 2003, at 15:59 Canada/Eastern, Andy Dills wrote:
On Thu, 17 Jul 2003, Jack Bates wrote:
Sendmail root exploit took less than 24 hours to craft. I suspect that this exploit will be found within 48 hours. Enough information was provided to quickly guess where the problem lies with IPv4 processing.
Sendmail is open source, IOS is not.
Knowing where the problem is and knowing how to exploit it are two entirely different situations.
If any IOS source code has ever found its way out of cisco since IOS 10.3 (and surely, that must have happened), then it seems reasonable to assume that there are people in the world currently comparing the advisory to the source. Joe
11.x IOS source was floating around a few years ago. I wouldn't be surprised if more recent versions were being distributed within the underground community. /m ----- Original Message ----- From: "Joe Abley" <jabley@isc.org> To: "Andy Dills" <andy@xecu.net> Cc: "Jack Bates" <jbates@brightok.net>; "Sean Donelan" <sean@donelan.com>; "Mikael Abrahamsson" <swmike@swm.pp.se>; <nanog@merit.edu> Sent: Thursday, July 17, 2003 1:11 PM Subject: Re: Cisco IOS Vulnerability
On Thursday, Jul 17, 2003, at 15:59 Canada/Eastern, Andy Dills wrote:
On Thu, 17 Jul 2003, Jack Bates wrote:
Sendmail root exploit took less than 24 hours to craft. I suspect that this exploit will be found within 48 hours. Enough information was provided to quickly guess where the problem lies with IPv4 processing.
Sendmail is open source, IOS is not.
Knowing where the problem is and knowing how to exploit it are two entirely different situations.
If any IOS source code has ever found its way out of cisco since IOS 10.3 (and surely, that must have happened), then it seems reasonable to assume that there are people in the world currently comparing the advisory to the source.
Joe
Foundstone Security Briefings: Cisco IPv4 Remote Denial of Service Vulnerability Date: Today, Thursday, July 17, 2003 Time: 5:30 PM Eastern, 2:30 PM Pacific Date: Tomorrow, Friday, July 18, 2003 Time: 11:00 AM Eastern, 8:00 AM Pacific You're invited to a Special Web Seminar today covering this critical vulnerability. If you cannot attend today's briefing please see instructions below to register for a follow up Web Seminar tomorrow. Cisco today announced a serious vulnerability for all Cisco devices that implement and are configured to process Internet Protocol version 4 (IPv4) packets. Foundstone Labs, first to respond to this serious risk, is offering this Security Briefing as part of a coordinated effort designed to protect current customers and other organizations. This vulnerability should be considered extremely critical due to the impact and ease-of-exploitation. Devices are vulnerable to a Denial of Service (DoS) attack and although no known exploit has been yet identified, a complex purposely malicious sequence of IPv4 packets targeted to a vulnerable Cisco switch or router can cause the processing interface to stop processing traffic. This vulnerability can be executed by remote unauthenticated users with mere knowledge of at least one interface IP address. Web Seminar Outline Introduction Overview of Cisco IOS Issues Analysis of the Cisco IOS Vulnerability Understanding the Impact Protection Mechanisms Questions and Answers Presenters Matt Ploessel - Foundstone Labs Tony Change - VP Engineering Brian Kenyon - Director of Product Services TO ATTEND TODAY'S WEB SEMINAR 1. Click the following Meeting URL or enter it in your browser: http://www.placeware.com/cc/encounter/A?id=07172003&pw=798380 Or alternatively, use the following URL: http://www.placeware.com/cc/encounter 2. On the "Enter Meeting" page that appears, supply this information if requested: Your Name: (enter your name) Meeting ID: 07172003 Meeting Key: 798380 Conference Center Name: encounter and then click the ENTER button at the bottom of the page. 3. Access audio for the meeting based on the following: If inside the US or Canada dial 1-800-223-9488 If outside the US or Canada dial 1-785-832-1508 Conference ID: Foundstone TO REGISTER FOR TOMORROW'S WEB SEMINAR Click the following URL or enter it in your browser: http://www.globalknowledge.com/training/course.asp?pageid=10&courseid=8157&catid=248 ++++++++++++++ Foundstone® Inc., experts in strategic security, offers a unique combination of software, services, and education to help organizations continuously and measurably protect the most important assets from the most critical threats. Through a strategic approach to security, Foundstone identifies and implements the right balance of technology, people, and process to manage digital risk and leverage security investments more effectively. ++++++++++++++ If you wish to be excluded from future announcements, simply reply to this e-mail with the single word REMOVE in the SUBJECT LINE. © 2003 Foundstone, Inc.
On 17.07 15:59, Andy Dills wrote:
Sendmail is open source, IOS is not.
Knowing where the problem is and knowing how to exploit it are two entirely different situations.
You are naive: Security through obscurity has never worked. You need secrecy if you go down this road; and that is hard to do. We are extremely lucky that Cisco managed to keep this under wraps for more than two months. The luck will not stretch to noone having the source code to a version of IOS with the probelm or the imagination necessary to find it without source. Daniel
On Friday 18 July 2003 03:04, Daniel Karrenberg wrote: [cut]
The luck will not stretch to noone having the source code to a version of IOS with the probelm or the imagination necessary to find it without source.
Daniel
cisco posted what the four 'bad' protocol types were in rev 1.3 of the online doc - now it is just an academic exercise to get them crafted correctly........no imagination necessary, only a router, a cco login, and a traffic generator needed /joshua -- What difference does it make to the dead, the orphans, and the homeless, whether the mad destruction is brought under the name of totalitarianism or the holy name of liberty and democracy? - Gandhi -
cisco posted what the four 'bad' protocol types were in rev 1.3 of the online doc - now it is just an academic exercise to get them crafted correctly........no imagination necessary, only a router, a cco login, and a traffic generator needed
With rev 1.0 it took me two hours. IP protocol space is only eight bits anyway. 1.3 gives exploit options for less networking oriented people too. Pete
If Cisco made THIS big a deal of this to not release info to the public, I wouldn't wait. There must be a reason. I had to push and push to get any info and I think they finally gave up because too many people knew. If you notice http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml For Public Release 2003 July 17 at 0:00 UTC (GMT) But at the bottom is says: Distribution This notice will be posted on the Cisco worldwide website at http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml at 21:00 GMT on July 17th, 2003. Hmmm... I think that means 4PM CT TOMORROW! From what I understand they didn't want this to be public until tomorrow afternoon. - D -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Mikael Abrahamsson Sent: Thursday, July 17, 2003 12:48 AM To: nanog@merit.edu Subject: RE: Cisco IOS Vulnerability On Wed, 16 Jul 2003, Darrell Kristof wrote:
Cisco Security Advisory: Cisco IOS Interface Blocked by IPv4 Packet http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml
IS anyone seeing this exploited in the wild? It'd be good to know if we need to do panic upgrade or can schedule it for our next maintenance window (which is during the weekend). -- Mikael Abrahamsson email: swmike@swm.pp.se
On Thu, 17 Jul 2003 01:05:46 CDT, Darrell Kristof <darrell.kristof@wholefoods.com> said:
If Cisco made THIS big a deal of this to not release info to the public, I wouldn't wait. There must be a reason. I had to push and push to get any info and I think they finally gave up because too many people knew.
http://www.cisco.com/warp/public/707/cisco-sa-20030717-blocked.shtml
which says... "Customers with contracts should obtain upgraded software free of charge through their regular update channels. For most customers, this means that upgrades should be obtained through the Software Center on the Cisco worldwide website at http://www.cisco.com/tacpage/sw-center/sw-ios.html." I may have been a few off, but I counted *139* different trains on that page as being affected. The 12.0S train alone has *13* different rebuilds. And there's *gotta* be at least 3-4 trains that suffer from bad karma and refuse to rebuild unless the Rebuild Wizard comes by and sprinkles Magic Rebuild Dust all over the place, and then there's the special procedure put in place after last year's debacle when the Magic Rebuild Dust got on that llama... ;) In other words - yeah, it's probably important to get this update deployed. But unless somebody has hard evidence to the contrary, I'm betting on it just being an attempt to not let things leak out till they're ready to ship across the board. That's a LOT of trains and rebuilds that all need to be ready at the same time, and Fred Brooks taught us all 30 years ago what happens when you try something like that. :)
On Thu, 17 Jul 2003 Valdis.Kletnieks@vt.edu wrote: :should be obtained through the Software Center on the Cisco worldwide website :at http://www.cisco.com/tacpage/sw-center/sw-ios.html I'm getting a 404 "not found" for that URL, while logged into CCO.
It should be: http://www.cisco.com/tacpage/sw-center/sw-ios.shtml The Advisory is being updated. It might even be out there.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Brian Wallingford Sent: Thursday, July 17, 2003 12:18 AM To: Valdis.Kletnieks@vt.edu Cc: Darrell Kristof; nanog@merit.edu Subject: Re: Cisco IOS Vulnerability
On Thu, 17 Jul 2003 Valdis.Kletnieks@vt.edu wrote:
:should be obtained through the Software Center on the Cisco worldwide website :at http://www.cisco.com/tacpage/sw-center/sw-ios.html
I'm getting a 404 "not found" for that URL, while logged into CCO.
It should be:
http://www.cisco.com/tacpage/sw-center/sw-ios.shtml
The Advisory is being updated. It might even be out there.
Do you know if they are going to update the advisory with more detail? At least I´m able to generate packets which get stuck in the input queue on the vulnerable releases but get properly discarded when sent to a box running a "Rebuild" release downloaded from CCO today. Design of the packet does not go past your average firewall configuration, not sure if there is one which would. So most people should be safe and the workarounds in the advisory do fend this one off also. Pete
Does anyone really use the RADb ? Worth the $250 per year? Just wondering if it worth renewing..
On Thu, 17 Jul 2003 03:17:32 EDT, Brian Wallingford said:
:at http://www.cisco.com/tacpage/sw-center/sw-ios.html
I'm getting a 404 "not found" for that URL, while logged into CCO.
Hmm.. you mean Magic Rebuild Dust doesn't work on webpages? ;) But yeah, it's *that* sort of thing that you want to try to iron out before the news gets out - having 139 trains all ready to go at the same time and making sure that TAC doesn't get slashdotted as a result is quite the intricate problem, and the last thing you need is complaints about 404's on webpages that weren't supposed to go live till tomorrow. ;)
Valdis.Kletnieks@vt.edu wrote:
In other words - yeah, it's probably important to get this update deployed. But unless somebody has hard evidence to the contrary, I'm betting on it just being an attempt to not let things leak out till they're ready to ship across the board. That's a LOT of trains and rebuilds that all need to be ready at the same time, and Fred Brooks taught us all 30 years ago what happens when you try something like that. :)
One of the 12.2 lines I have to use shows a post of June, 25. My guess is that they started rebuilding some of the later IOS versions and worked their way back. My 12.0S line didn't post until today. -Jack
On Thu, 17 Jul 2003, Mikael Abrahamsson wrote:
IS anyone seeing this exploited in the wild? It'd be good to know if we need to do panic upgrade or can schedule it for our next maintenance window (which is during the weekend).
Well, there's this from Wednesday afternoon... - > Dear AT&T IP Services Customer: - > - > Please be advised of the following: - > - > This is a preliminary notification to inform you that AT&T IP Services - > experienced an impairment that may have affected some customer traffic - > on the West Coast. [The above is is a mild understatement...] - > Our Network Engineers have resolved the issue and are currently - > investigating the root cause. A follow-up email will be sent at - > the conclusion of the investigation with more information. [Nothing received yet...] This was rumored to be a backhoe fade but the advisory refers only to IP services and there was nothing in the popular press about any major phone outage, so I have my suspicions. Usually if there's a fiber cut they say so. About this time is when all of the major backbones began flooding the net with their notices of panic upgrades. (This is being typed while watching rows and rows of "!!!!!!!"). -- Jay Hennigan - CCIE #7880 - Network Administration - jay@west.net WestNet: Connecting you to the planet. 805 884-6323 WB6RDV NetLojix Communications, Inc. - http://www.netlojix.com/
participants (23)
-
Andy Dills -
Barry Raveendran Greene -
Brian Wallingford -
Christopher L. Morrow -
Daniel Karrenberg -
Darrell Kristof -
Jack Bates -
Jared Mauch -
Jason Lixfeld -
Jay Hennigan -
Jeff Kell -
Joe Abley -
John Timmons -
joshua sahala -
micah mcnelly -
Michael Painter -
Mikael Abrahamsson -
mike harrison -
Petri Helenius -
Ryan Tucker -
Sean Donelan -
Todd Mitchell - lists -
Valdis.Kletnieks＠vt.edu