RE: Cisco IOS Exploit Cover Up
For what ot's worth, this story is running in the popular trade press: "Cisco nixes conference session on hacking IOS router code" http://www.networkworld.com/news/2005/072705-cisco-ios.html - ferg -- "Hannigan, Martin" <hannigan@verisign.com> wrote:
For those who like to keep abreast of security issues, there are interesting developments happening at BlackHat with regards to Cisco IOS and its vulnerability to arbitrary code executions.
I apologize for the article itself being brief and lean on technical details, but allow me to say that it does represent a real problem (as in practical and confirmed):
http://blogs.washingtonpost.com/securityfix/2005/07/mending_a_ hole_.html
Yes, practical _and_ confirmed, but you'll never get $vendor to admit it, which is the problem to begin with. -M< -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg@netzero.net or fergdawg@sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/
Damn he sure did cause a shit storm AGAIN.. from the crn article it looks like they might have him pinned on an NDA violation.. (taking a shot in the dark) quote below. "Cisco respects and encourages the work of independent research scientists; however, we follow an industry established disclosure process for communicating to our customers and partners," the company said in a statement released Wednesday. "It is especially regretful, and indefensible, that the Black Hat Conference organizers have given Mr. Lynn a platform to publicly disseminate the information he illegally obtained." Which i find is funny because i know that for years people have been beating up on him for more info into the cisco wireless cards that he had access to under NDA. He never once budged from what i know of and heard. Damn guess we will have to wait and see what happens, to bad i missed the talk. On 7/27/05, Fergie (Paul Ferguson) <fergdawg@netzero.net> wrote:
For what ot's worth, this story is running in the popular trade press:
"Cisco nixes conference session on hacking IOS router code" http://www.networkworld.com/news/2005/072705-cisco-ios.html
- ferg
-- "Hannigan, Martin" <hannigan@verisign.com> wrote:
For those who like to keep abreast of security issues, there are interesting developments happening at BlackHat with regards to Cisco IOS and its vulnerability to arbitrary code executions.
I apologize for the article itself being brief and lean on technical details, but allow me to say that it does represent a real problem (as in practical and confirmed):
http://blogs.washingtonpost.com/securityfix/2005/07/mending_a_ hole_.html
Yes, practical _and_ confirmed, but you'll never get $vendor to admit it, which is the problem to begin with.
-M<
-- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg@netzero.net or fergdawg@sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/
On Wed, 27 Jul 2005, Fergie (Paul Ferguson) wrote:
For what ot's worth, this story is running in the popular trade press:
"Cisco nixes conference session on hacking IOS router code" http://www.networkworld.com/news/2005/072705-cisco-ios.html
This is looking like a complete PR disaster for cisco. They would have been better off allowing the talk to take place, and actually fixing the holes rather than wasting money on a small army of razorblade-equipped censors. -Dan
This is looking like a complete PR disaster for cisco. They would have been better off allowing the talk to take place, and actually fixing the holes rather than wasting money on a small army of razorblade-equipped censors.
I couldn't disagree more. Cisco are trying to control the situation as best they can so that they can deploy the needed fixes before the $scriptkiddies start having their fun. Its no different to how any other vendor handles a exploit and I'm surprised to see network operators having such an attitude. Regards. Neil.
* Neil J. McRae:
I couldn't disagree more. Cisco are trying to control the situation as best they can so that they can deploy the needed fixes before the $scriptkiddies start having their fun. Its no different to how any other vendor handles a exploit and I'm surprised to see network operators having such an attitude.
Cisco is different in at least one regard: they only list confirmed impact, not potential impact. Thus many bugs get labeled as DoS issues, which other vendors would have described as a vulnerability which potentially enables remote code injection exploits.
In a message written on Thu, Jul 28, 2005 at 08:29:22AM +0100, Neil J. McRae wrote:
I couldn't disagree more. Cisco are trying to control the situation as best they can so that they can deploy the needed fixes before the $scriptkiddies start having their fun. Its no different to how any other vendor handles a exploit and I'm surprised to see network operators having such an attitude.
This is not a Cisco specific comment, but it is a network operator comment. You change your mind when you get hit by a network wide bug taking out all your customers, and then spend six months beating up the gear in your own lab to reproduce the problem, and when you do the vendor finally admits "well, we've known about the bug for 4 years, but we were pretty sure it couldn't happen in your network so we didn't tell you." I'm sure the vendors find bugs, quietly fix them, the code is naturally upgraded and nothing ever happens. Which is a good thing. The problem is, most of the major operators have been hit by a bug where the vendor knew, did nothing, or at least not enough, the operator was hit and then the vendor continued to not want to admit the problem because of course now they look even worse for sitting on it. For better or for worse, right now the only check and balance to the vendors is conferences like the Black Hat forum. For Cisco to send an army of razor blade toting employees to such a conference is chilling. I can see them working with the parties before hand, but to make that kind of show in public? What is the motovation? If this bug is, as Cisco puts it, "not serious" then they just spent a lot of money on people to go do all of that for nothing. Doesn't seem likely. So what everyone's spidy sense is now telling them is Cisco wouldn't spend thousands of dollars on legal injunctions and armys of razor blade toters for nothing, so there must be something to this paper. Which makes their denial all the more hollow. This isn't an endorsement of the pro-disclosure crowd. Telling these things to the world at large in a forum like this gives the script kiddies a leg up, as they are almost always faster than the vendors. These things should happen at a more measured pace, inside normal support channels. That said, no one likes a coverup. Once it's public in any form, don't try to sweep it under the rug. Doesn't work in politics, doesn't work for vendors. Sometimes you can get away with it once or twice, but in the end it costs credibility, which is something that is extremely hard and costly to earn back. If Cisco wanted to make me feel better right now they could contact my company via normal support channels and have a frank and open discussion about what this paper/presentation means, and what action if any they are taking as a result. Somehow for what the boxes and support costs that doesn't seem like too much to ask. The presentation is out there, we will get it and read it, don't pretend like we won't. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On Thu, 28 Jul 2005, Leo Bicknell wrote:
In a message written on Thu, Jul 28, 2005 at 08:29:22AM +0100, Neil J. McRae wrote:
I couldn't disagree more. Cisco are trying to control the situation as best they can so that they can deploy the needed fixes before the $scriptkiddies start having their fun. Its no different to how any other vendor handles a exploit and I'm surprised to see network operators having such an attitude.
This is not a Cisco specific comment, but it is a network operator comment. --snip--- but to make that kind of show in public? What is the motovation? If this bug is, as Cisco puts it, "not serious" then they just spent a lot of money on people to go do all of that for nothing. Doesn't seem likely. So what everyone's spidy sense is now telling them is Cisco wouldn't spend thousands of dollars on legal injunctions and armys of razor blade toters for nothing, so there must be something to this paper. Which makes their denial all the more hollow.
There is the possiblity that cisco, in this case, knows that they have a significant base of folks that 'never upgrade' devices. I know of several thousand 2500's with 11.x code on them, which will NEVER be upgraded... So, the potential for Neil's network or Leo's or Martin's to be vulnerable to something patched in 12.0.x.y.z code train 9 months ago isn't there. That's a good thing for them, it doesn't address the thousands, or hundreds of thousands of devices which never get upgraded and still connect to Neil/Martin/Leo's networks as CPE or cpe to cpe... These devices could still cause some pain to the networks in question. (all this without seeing the talk of course... perhaps he said: push button yellow and router go boom. I don't know.)
On Jul 28, 2005, at 3:29 AM, Neil J. McRae wrote:
I couldn't disagree more. Cisco are trying to control the situation as best they can so that they can deploy the needed fixes before the $scriptkiddies start having their fun. Its no different to how any other vendor handles a exploit and I'm surprised to see network operators having such an attitude.
That's part of the issue: this wasn't an exploit in the sense of something a $scriptkiddie could exploit. The sheer technical requirements of the exploit itself ensure that it will only be reproduced by a small number of people across the globe. There was no source or proof of concept code released and duplicating the information would only provide you a method to increase the severity of other potential exploits. It does not create any new exploits. Moreover, the fix for this was already released and you have not been able to download a vulnerable version of the software for months however there was no indication from Cisco regarding the severity of the required upgrade. That is to say, they knew in April that arbitrary code execution was possible on routers, they had it fixed by May, and we're hearing about it now and if Cisco had its way we might still not be hearing about it. How many network engineers knew there was a potential problem of this magnitude at the beginning of May? If, knock on wood, someone had released this code into the wild then how many networks who have been vulnerable despite the availability of a fix? Considering that Mr. Lynn's presentation was flawless, it is interesting to note that Cisco and ISS considered the information to be "not quite complete." This is especially interesting since the research was done weeks ago according the researcher. Its surprising that such a decision as to the incompleteness of the presentation and the retraction of Cisco's support for the presentation were withdrawn only several days before the talk. It would lead me to believe that both companies had less interest in a "process of disclosure and communication" and more with burying this information for a year or more. I agree with everyone that making attack tools and exploit information available to the public prior to a fix being generated with the vendor is a poor method of encouraging good security, however that is far from the case in this matter. A fix had been generated with the vendor and it was time that the information to become public so network operators understood that the remote execution empty world we had lived in until now was over. More links: http://www.wired.com/news/privacy/0,1848,68328,00.html? tw=wn_story_page_prev2 http://securityfocus.com/news/11259
* James Baldwin:
A fix had been generated with the vendor and it was time that the information to become public so network operators understood that the remote execution empty world we had lived in until now was over.
Huh? Remote code injection exploits on Cisco routers have been demonstrated before, haven't they? Previous ones were rather fragile, and the amount of knowledge and experimentation needed was rather high. Actually, this is the type of exploit I would expect to be unavailable to the general public (read: network operators) for a long, long time. If there was a perception in the community that remote code injection exploits were a non-issue on routers, then this incident was long overdue, and Cisco should be thankful because their customers can assess risks in a more realistic way. ISS is probably the real loser here because these days, their business is based to a large extent on selling access to relevant strategic information, and dissemination of any background information reduces the value of their service (or the exclusiveness of the offerrings, at the least).
James Baldwin <jbaldwin@antinode.net> writes:
On Jul 28, 2005, at 3:29 AM, Neil J. McRae wrote:
I couldn't disagree more. Cisco are trying to control the situation as best they can so that they can deploy the needed fixes before the $scriptkiddies start having their fun. Its no different to how any other vendor handles a exploit and I'm surprised to see network operators having such an attitude.
That's part of the issue: this wasn't an exploit in the sense of something a $scriptkiddie could exploit. The sheer technical requirements of the exploit itself ensure that it will only be reproduced by a small number of people across the globe. There was no source or proof of concept code released and duplicating the information would only provide you a method to increase the severity of other potential exploits. It does not create any new exploits. Moreover, the fix for this was already released and you have not been able to download a vulnerable version of the software for months however there was no indication from Cisco regarding the severity of the required upgrade. That is to say, they knew in April that arbitrary code execution was possible on routers, they had it fixed by May, and we're hearing about it now and if Cisco had its way we might still not be hearing about it.
Can you or someone else who was there or has some details describe what the actual result is and what the fix was? Based on what I've been reading, it sounds like Lynn's result was a method for exploiting arbitrary new vulnerabilities. Are you saying that this method can't be used in future IOS revs? Thanks, -Ekr [Eric Rescorla RTFM, Inc.]
On Thu, Jul 28, 2005 at 07:03:31AM -0700, Eric Rescorla wrote:
Can you or someone else who was there or has some details describe what the actual result is and what the fix was? Based on what I've been reading, it sounds like Lynn's result was a method for exploiting arbitrary new vulnerabilities. Are you saying that this method can't be used in future IOS revs?
As nearly as I can tell from reports (I wasn't there), he (1) talked about a general way to exploit a buffer overflow to cause arbitrary code execution (this would apply to buffer overflows generally, but would be completely useless if you didn't know of a buffer overflow to exploit), and (2) demonstrated his technique using a previosuly known buffer overflow vulnerability which Cisco has already patched. So Cisco is correct in saying that he didn't identifiy any new vulnerabilities, and Cisco is also correct in saying that the vulnerability he used in his presentation to demonstrate his technique has been patched. However, the same technique will be useful on the next buffer overflow vulnerability to be discovered. -- Brett
Bear in mind though that when the M$ SQL Slammer worm hit everyone, the same attitude existed. The patch had been available for months. People knew about the vulnerability and it wasn't anything "new". And yet, look how much havoc was created there. It's always the "potential" stuff that scares people more. While I do think it's obnoxious to try to censor someone, on the other hand if they have proprietary internal information somehow that they aren't supposed to have to begin with, I don't think it is in security's best interested to commit a crime in order to get tighter security. Is this the technical version of civil disobedience? Scott -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of James Baldwin Sent: Thursday, July 28, 2005 9:24 AM To: Neil J.McRae Cc: nanog@merit.edu Subject: Re: Cisco IOS Exploit Cover Up On Jul 28, 2005, at 3:29 AM, Neil J. McRae wrote:
I couldn't disagree more. Cisco are trying to control the situation as best they can so that they can deploy the needed fixes before the $scriptkiddies start having their fun. Its no different to how any other vendor handles a exploit and I'm surprised to see network operators having such an attitude.
That's part of the issue: this wasn't an exploit in the sense of something a $scriptkiddie could exploit. The sheer technical requirements of the exploit itself ensure that it will only be reproduced by a small number of people across the globe. There was no source or proof of concept code released and duplicating the information would only provide you a method to increase the severity of other potential exploits. It does not create any new exploits. Moreover, the fix for this was already released and you have not been able to download a vulnerable version of the software for months however there was no indication from Cisco regarding the severity of the required upgrade. That is to say, they knew in April that arbitrary code execution was possible on routers, they had it fixed by May, and we're hearing about it now and if Cisco had its way we might still not be hearing about it. How many network engineers knew there was a potential problem of this magnitude at the beginning of May? If, knock on wood, someone had released this code into the wild then how many networks who have been vulnerable despite the availability of a fix? Considering that Mr. Lynn's presentation was flawless, it is interesting to note that Cisco and ISS considered the information to be "not quite complete." This is especially interesting since the research was done weeks ago according the researcher. Its surprising that such a decision as to the incompleteness of the presentation and the retraction of Cisco's support for the presentation were withdrawn only several days before the talk. It would lead me to believe that both companies had less interest in a "process of disclosure and communication" and more with burying this information for a year or more. I agree with everyone that making attack tools and exploit information available to the public prior to a fix being generated with the vendor is a poor method of encouraging good security, however that is far from the case in this matter. A fix had been generated with the vendor and it was time that the information to become public so network operators understood that the remote execution empty world we had lived in until now was over. More links: http://www.wired.com/news/privacy/0,1848,68328,00.html? tw=wn_story_page_prev2 http://securityfocus.com/news/11259
In a message written on Thu, Jul 28, 2005 at 10:14:42AM -0400, Scott Morris wrote:
And yet, look how much havoc was created there. It's always the "potential" stuff that scares people more. While I do think it's obnoxious to try to censor someone, on the other hand if they have proprietary internal information somehow that they aren't supposed to have to begin with, I don't think it is in security's best interested to commit a crime in order to get tighter security.
We don't have all the details, so I don't know what he's accused of doing which is illegal, however, from http://news.zdnet.co.uk/internet/security/0,39020375,39211011,00.htm I quote: ] The filing in US District Court for the Northern District of California ] asks the court to prevent Lynn and Black Hat from "further disclosing ] proprietary information belonging to Cisco and ISS," said John Noh, a ] Cisco spokesman. ] ] "It is our belief that the information that Lynn presented at Black Hat ] this morning is information that was illegally obtained and violated our ] intellectual-property rights," Noh added. ] ] Lynn decompiled Cisco's software for his research and by doing so ] violated the company's rights, Noh said. I am not a lawyer, and so under the current DMCA and other laws it may well be illegal to "decompile" code. That said, it sounds rather like the technical equivilant to Ralph Nader "disassembling" the Corvair to prove the suspension design was flawed. GM sure didn't like that any more than Cisco likes this incident. I don't know when we decided a program should be a black box welded shut kept from all prying eyes, and that anyone who could run a decompiler was instantly a crimimal. It probably all came about from the crazy decision that software should be licensed, not sold. We'd be in a world of hurt if anyone who figured out how to put a lift kit on his pickup was sued by ford for "disassembling" the truck and figuring out their "propretary internal designs". Why is software special? -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On 7/28/05, Leo Bicknell <bicknell@ufp.org> wrote:
I am not a lawyer, and so under the current DMCA and other laws it may well be illegal to "decompile" code.
I'm sure all the script kiddies and real hackers out there will be sure to obey the law.. This is the bit of the DMCA I have a huge issue with.. Hackers and others engaging in illegal activities will have no trouble breaking the law and decompiling code looking for exploits. But, if a researcher does it, they get slapped with a lawsuit.. The difference being, the researcher is (usually) doing it to help identify problems and increase security.. There should be some safe harbor here..
That said, it sounds rather like the technical equivilant to Ralph Nader "disassembling" the Corvair to prove the suspension design was flawed. GM sure didn't like that any more than Cisco likes this incident.
To prove a flaw.. This is a great example. Nader wasn't stealing technology, nor was he interested in exploitinig the flaw.. He was proving that it was unsafe, thus providing the vendor with vital information on how it was flawed.. Hopefully the vendor takes that information and fixes the flaw..
I don't know when we decided a program should be a black box welded shut kept from all prying eyes, and that anyone who could run a decompiler was instantly a crimimal. It probably all came about from the crazy decision that software should be licensed, not sold. We'd be in a world of hurt if anyone who figured out how to put a lift kit on his pickup was sued by ford for "disassembling" the truck and figuring out their "propretary internal designs". Why is software special?
Good point.. :) What about my house? Can I no longer modify my kitchen at the whim of my wife because I didn't build the house, someone else did? I purchased the home, although it's still mortgaged... So that's even worse.. I don't even really own it.. :) Crap.. anyone know a good lawyer? :)
-- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
-- Jason 'XenoPhage' Frisvold XenoPhage0@gmail.com
On Jul 28, 2005, at 10:14 AM, Scott Morris wrote:
While I do think it's obnoxious to try to censor someone, on the other hand if they have proprietary internal information somehow that they aren't supposed to have to begin with, I don't think it is in security's best interested to commit a crime in order to get tighter security.
Lynn developed this information based on publicly available IOS images. There were no illegal acts committed in gaining this information nor was any proprietary information provided for its development. Reverse engineering, specifically for security testing has an exemption from the DMCA (http://cyber.law.harvard.edu/openlaw/ DVD/1201.html). That being said, what information is he not supposed to have? All the information he had is available to anyone with a disassembler, an IOS image, and an understanding of PPC assembly. If anything, the only "crime" he may or may not have committed is violation of an NDA with ISS, which should a contractual, civil issue not a criminal one.
On Thu, Jul 28, 2005 at 01:36:01PM -0400, James Baldwin wrote:
On Jul 28, 2005, at 10:14 AM, Scott Morris wrote:
While I do think it's obnoxious to try to censor someone, on the other hand if they have proprietary internal information somehow that they aren't supposed to have to begin with, I don't think it is in security's best interested to commit a crime in order to get tighter security.
Lynn developed this information based on publicly available IOS images. There were no illegal acts committed in gaining this information nor was any proprietary information provided for its development. Reverse engineering, specifically for security testing has an exemption from the DMCA (http://cyber.law.harvard.edu/openlaw/ DVD/1201.html).
That being said, what information is he not supposed to have? All the information he had is available to anyone with a disassembler, an IOS image, and an understanding of PPC assembly.
If anything, the only "crime" he may or may not have committed is violation of an NDA with ISS, which should a contractual, civil issue not a criminal one.
I think that's why it was a restraining order and not damanges in the amounts of billions, but IANAL. Same way people were asked to not disclose who the half-blooded prince was. I'm not saying it's right, but that's up for the judge(s) involved to decide. As far as Cisco goes, I know it takes them some time to fix bugs, but generally speaking they need to "fix them faster". But this can be said for most vendors. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Thus spake "James Baldwin" <jbaldwin@antinode.net>
Moreover, the fix for this was already released and you have not been able to download a vulnerable version of the software for months however there was no indication from Cisco regarding the severity of the required upgrade. That is to say, they knew in April that arbitrary code execution was possible on routers, they had it fixed by May, and we're hearing about it now and if Cisco had its way we might still not be hearing about it.
Cisco's policy, as best I can tell, is that they patch security holes immediately but delay notification until either (a) six months pass, or (b) an exploit is seen in the wild. The former is intended to give customers ample time to upgrade to patched versions (often without their knowledge) without tipping their hand to the "bad guys". However, a CERT advisory is prepared and ready for immediate distribution if the latter occurs.
How many network engineers knew there was a potential problem of this magnitude at the beginning of May? If, knock on wood, someone had released this code into the wild then how many networks who have been vulnerable despite the availability of a fix?
There are network engineers that knew, but they couldn't admit it due to NDAs. This is one of the benefits of buying "high touch" support contracts -- and Cisco is not alone in that model. S Stephen Sprunk "Those people who think they know everything CCIE #3723 are a great annoyance to those of us who do." K5SSS --Isaac Asimov
participants (14)
-
Andre Ludwig -
Brett Frankenberger -
Christopher L. Morrow -
Dan Hollis -
Eric Rescorla -
Fergie (Paul Ferguson) -
Florian Weimer -
James Baldwin -
Jared Mauch -
Jason Frisvold -
Leo Bicknell -
Neil J. McRae -
Scott Morris -
Stephen Sprunk