Re: [NANOG] IOS rootkits
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -- Dragos Ruiu <dr@kyx.net> wrote:
The question this presentation begs for me... is how many of the folks on this list do integrity checking on their routers?
You can no longer say this isn't necessary :-).
I know FX and a few others are working on toolsets for this...
I'll probably have other comments after I see the presentation. This development has all sort of implications for binary signing requirements, etc...
Yep -- I'd say just wait for the presentation (assuming Cisco doesn't go after this guy like they did Mike Lynn) and then determine the level of seriousness. It would appear to have people very nervous, however. Including Cisco. It will be interesting to see what develops. - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFILlgzq1pz9mNUZTMRAtmoAKC3bQLSqJzFDZklPMfdnkBX7fyccwCeN5mc K1QQ9JnTqLmSfcNuj5JZ6Z8= =W5F0 -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
What if some good comes from this "root kit"? For instance, what if it lets us fix things like DOM on non-Cisco XENPAKs and SFPs? Or lets us un-cripple our 6500 chassis to run the code we want? Of course, given the messenger, I'm sure it's just hype to help bolster Gadi's security practice, and will prove to be no big deal. Paul
Paul Wall wrote:
What if some good comes from this "root kit"?
I'm sure it'll be good for a number of security providers to hawk their wares. If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. MMC
On Sat May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote:
Paul Wall wrote:
What if some good comes from this "root kit"?
I'm sure it'll be good for a number of security providers to hawk their wares.
How long before we need to install Anti-virus / Anti-root-kit software on our routers? Simon -- Simon Lockhart | * Sun Server Colocation * ADSL * Domain Registration * Director | * Domain & Web Hosting * Internet Consultancy * Bogons Ltd | * http://www.bogons.net/ * Email: info@bogons.net *
On Sat, 17 May 2008, Simon Lockhart wrote:
On Sat May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote:
Paul Wall wrote:
What if some good comes from this "root kit"?
I'm sure it'll be good for a number of security providers to hawk their wares.
How long before we need to install Anti-virus / Anti-root-kit software on our routers?
Very astute. Sadly, this is already being done by a few people I know. No AV vendor has such a tool to offer you, so don't bother asking them. The question is, can you afford not to? The answer may be yes, you can afford for your router to be a spying machine for the enemy/competitor, and you can afford for it to be a bot participating in DDoS (as currently, for example, many *nix routers are known to be). The question is who can't afford for these things to happen... Gadi.
Simon -- Simon Lockhart | * Sun Server Colocation * ADSL * Domain Registration * Director | * Domain & Web Hosting * Internet Consultancy * Bogons Ltd | * http://www.bogons.net/ * Email: info@bogons.net *
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
The question is who can't afford for these things to happen...
Gadi.
I can't help but feel you're pushing fear to further some other interest here Gadi. Do you actually have live examples of this or able to demonstrate it or are you just theorising about it all? MMC
On Sat, 17 May 2008, Matthew Moyle-Croft wrote:
It is alright to have feelings.
Gadi.
So I ask again, expecting nothing but another flippant answer:
I will honour you flame-bait, but only once.
Do you actually have live examples of this or able to demonstrate it or are you just theorising about it all?
Your question is irrelevant to our discussion, as I obviously base myself on the first email in this thread discussing the poc (?) about to be released, and my own statements from that first email in which I mention I will not discuss my own experience on the subject of rootkit risks and solutions until said poc (?) is released due to matters of honour.
MMC
On Sat, 17 May 2008 07:03:58 -0500 (CDT) Gadi Evron <ge@linuxbox.org> wrote:
On Sat, 17 May 2008, Matthew Moyle-Croft wrote:
The question is who can't afford for these things to happen...
Gadi.
I can't help but feel you're pushing fear to further some other interest here Gadi.
It is alright to have feelings.
The rational thing to do is to move beyond fear. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft <mmc@internode.com.au> wrote:
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch. --srs -- Suresh Ramasubramanian (ops.lists@gmail.com)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Suresh Ramasubramanian wrote:
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft <mmc@internode.com.au> wrote:
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch.
--srs
According to Cisco, there is nothing to patch: http://www.cisco.com/warp/public/707/cisco-sr-20080516-rootkits.shtml Jon Kibler - -- Jon R. Kibler Chief Technical Officer Advanced Systems Engineering Technology, Inc. Charleston, SC USA o: 843-849-8214 c: 843-224-2494 s: 843-564-4224 My PGP Fingerprint is: BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkgusjEACgkQUVxQRc85QlO5kACfaZtij86HqIH540xeH+Uh/NyI ccQAnjiRCMFnLxk/Ew9EuUKDzdLN6HQZ =BCdw -----END PGP SIGNATURE----- ================================================== Filtered by: TRUSTEM.COM's Email Filtering Service http://www.trustem.com/ No Spam. No Viruses. Just Good Clean Email.
On Sat, May 17, 2008 at 11:12 AM, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft <mmc@internode.com.au> wrote:
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch.
Agreed, You've got to remember though that a security conference is a commercial venture, it makes business sense for this to be publically announced at this security conference. I think security conferences have become something that sucks as its all become money making oriented and the people who run these things don't really have security in mind, just the Ā£ signs reflecting on their eye balls.
--srs -- Suresh Ramasubramanian (ops.lists@gmail.com)
All the best, n3td3v
On Sat, 17 May 2008, Suresh Ramasubramanian wrote:
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft <mmc@internode.com.au> wrote:
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch.
I'd like to discuss: 1. What is it we are talking about. 2. Why it is serious. 3. What we can do to defend ourselves. I'll be brief as this is not a briefing. You are absolutely right on the sentiment, but miss the point on this particular issue. I agree with you that in most cases, software vulnerability issues should be resolved with the vendor first, especially where critical infrastructure is involved. This is not only about exploiting a vulnerability. In this case it the the very realization that these issues exist (namely being able to run Trojan horses on IOS systems AND/or hiding their presense) is what we are discussing. Router security as far as most operators are concerned includes the following issues: software version (now update), configuration, ACL and authentication (password) security. I include subjects such as BGP MD5 in configuration. These issues are indeed important and very neglected, after all, how many "0wned" routers can be found that respond to cisco/cisco? The main difference here is that we are now at a cross-roads where the face of router security changes, It is that the realization that: 1. A router is not an hardware device, it is an embedded device with a software operating system. As such it is as vulnerable to malware (wide-spreading--worm, or targeted--Trojan horse) as a Windows machine is.) 2. There are no real tools today for us to be able to detect such malicious activity on a router, listing processes doesn't cut it. 3. What tools exist, which I hope to secure permission to discuss later on, are only from third parties. This is not about fear mongering, it's about facing reality how about how Cisco handles security threats to their customer base before such an issue becomes a public concern--namely, ignoring its very existence, at least as far as the public can see. The point is, I don't want to rely on third parties for my router's security, even if I trust the said third party. Gadi.
On 17-May-08, at 3:12 AM, Suresh Ramasubramanian wrote:
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft <mmc@internode.com.au> wrote:
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch.
Bullshit. There is nothing to patch. It needs to be presented at conferences, exactly because people will play ostrich and stick their heads in the sand and pretend it can't happen to them, and do nothing about it until someone shows them, "yes it can happen" and here is how.... Which is exactly why we've accepted this talk. We've all known this is a possibility for years, but I haven't seen significant motion forward on this until we announced this talk. So in a fashion, this has already helped make people more realistic about their infrastructure devices. And the discussions, and idea interchange that will happen between the smart folks at the conference will undoubtedly usher forth other related issues and creative solutions. Problems don't get fixed until you talk about them. cheers, --dr -- World Security Pros. Cutting Edge Training, Tools, and Techniques London, U.K. May 21/22 - 2008 http://cansecwest.com pgpkey http://dragos.com/ kyxpgp
Let's put it this way. 1. Yes there's nothing to patch, as such 2. It can be prevented by what's widely regarded as BCP on router security, and has been covered at *nog, in cisco training material, etc etc for quite some time now. I am much less concerned about security conferences discussing this than about the (highly uninformed) publicity that accompanies these conferences. Yes, this sounds a lot more like the bugtraq v/s full disclosure discussion than I'm comfortable with, but I still think this could have been handled a lot better. --srs On Sun, May 18, 2008 at 7:27 PM, Dragos Ruiu <dr@kyx.net> wrote:
Bullshit. There is nothing to patch. It needs to be presented at conferences, exactly because people will play ostrich and stick their heads in the sand and pretend it can't happen to them, and do nothing about it until someone shows them, "yes it can happen" and here is how.... Which is exactly why we've accepted this talk. We've all known this is a possibility for years, but I haven't seen significant motion forward on this until we announced this talk. So in a fashion, this has already helped make people more realistic about their infrastructure devices. And the discussions, and idea interchange that will happen between the smart folks at the conference will undoubtedly usher forth other related issues and creative solutions. Problems don't get fixed until you talk about them. cheers, --dr
On Sun, 18 May 2008, Suresh Ramasubramanian wrote:
Let's put it this way.
1. Yes there's nothing to patch, as such
2. It can be prevented by what's widely regarded as BCP on router security, and has been covered at *nog, in cisco training material, etc etc for quite some time now.
I am much less concerned about security conferences discussing this than about the (highly uninformed) publicity that accompanies these conferences.
Yes, this sounds a lot more like the bugtraq v/s full disclosure discussion than I'm comfortable with, but I still think this could have been handled a lot better.
It's easy to blame researchers for doing their studies, but the fact is, if one whitehat researcher has done work on it, it is already exploited in the wild. Gadi.
--srs
On Sun, May 18, 2008 at 7:27 PM, Dragos Ruiu <dr@kyx.net> wrote:
Bullshit. There is nothing to patch. It needs to be presented at conferences, exactly because people will play ostrich and stick their heads in the sand and pretend it can't happen to them, and do nothing about it until someone shows them, "yes it can happen" and here is how.... Which is exactly why we've accepted this talk. We've all known this is a possibility for years, but I haven't seen significant motion forward on this until we announced this talk. So in a fashion, this has already helped make people more realistic about their infrastructure devices. And the discussions, and idea interchange that will happen between the smart folks at the conference will undoubtedly usher forth other related issues and creative solutions. Problems don't get fixed until you talk about them. cheers, --dr
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
On 18-May-08, at 7:11 AM, Suresh Ramasubramanian wrote:
2. It can be prevented by what's widely regarded as BCP on router security, and has been covered at *nog, in cisco training material, etc etc for quite some time now.
I am much less concerned about security conferences discussing this than about the (highly uninformed) publicity that accompanies these conferences.
I'm not going to touch the disclosure or not debate... it's been done. But I will agree to disagree with you about the above two points. First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools. Regarding the second point, I also lament the often liberal doses of alarmism/FUD that get plastered over the popular media whenever complicated technical issues are discussed - but unless we have some have the discussions, and information dispersal, then the misconceptions have no chance of being dispelled. The threat of misinformed press does not seem to be sufficient to justify censuring open discussion of the issues imho. One of the thing I truly enjoy about the conferences we organize, is seeing the synergism that occurs when multiple minds focus on these security issues at the conferences. When the analysis is parallelized over multiple brains, inevitably the creative solutions that occur from the congregation of different viewpoints and ideas is pleasantly surprising, and powerful. I've seen numerous examples of this: even just last April I had a chance to be a fly on the wall at a discussion between Jacob Appelbaum and Theo DeRaadt talking about the cold memory attacks research Jacob started - the result of which was that during the discussion it was realized that with the addition of about 30 lines of code in the power fail interrupt handler a large segment of those attacks could be nullified, as they are now on OpenBSD. If the discussion hadn't happened, the creative solution to it would have never arisen. These kinds of "out of the box" solutions frequently arise out of multi-person debate and free association that follows discussions of serious issues - no-one has the whole picture and adding other's viewpoints often brings superior solutions to problems up. So in my opinion the benefits of discussing serious issues at conferences far outweigh the potential drawbacks of misguided media coverage of them. What I infer from your post is that you are of the opinion that issues such as this rootkit prototype should be reported to CSIRT and then shuffled under a carpet. To which I respond that that kind of attitude has led to what I currently consider to be an inappropriate level of concern and awareness amongst service providers of the seriousness of this threat. Cisco has some great guys, but surely discussion of this threat amongst the wider security community will lead to more and better solutions than Cisco operating in a vacuum. And more importantly this issue is not a Cisco issue - the basic threat vector should be a concern to other infrastructure equipment manufacturers too. Until we talk about it, we cannot find the right responses to the problem, and experts talking about it usually leads to better and more comprehensive solutions than single persons or smaller groups working in isolation. cheers, --dr -- World Security Pros. Cutting Edge Training, Tools, and Techniques London, U.K. May 21/22 - 2008 http://eusecwest.com pgpkey http://dragos.com/ kyxpgp
Dragos Ruiu wrote:
First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools.
I have 6 years worth of rancid logs for every time the reported number of blocks in use on my flash changes, I imagine others do as well. That's hardly the silver bullet however. We as I imagine others do expended a fair amount of cycles monitoring who it is that our routers are talking to and protecting the integrity of the communications channels that they use (bgp, ospf, ssh, tftp etc), If a router has a tcp connection to someplace it shouldn't we'll probably know about it. If it's announcing a prefix it shouldn't be, we'll probably know about it, those are the easy ones though. There are some things one might consider adding in terms of auditing, comparing the running image more closely to the one in flash for example, peroidic checksum of the on onflash image, after downloading to another host would be another. I'm not sure that I'd trust the later given the rooted box can I suppose hand you an unmodified version of the subverted image. In the end if you subvert a router, presumably you're doing it for a purpose and given what the device does, that purpose is probably detectable in a well instrumented network. It is desirable I expect to insure that any locally stored security credentials that might be subverted not be usable when connecting to another router, that applies in a absence of root kits however.
Regarding the second point, I also lament the often liberal doses of alarmism/FUD that get plastered over the popular media whenever complicated technical issues are discussed - but unless we have some have the discussions, and information dispersal, then the misconceptions have no chance of being dispelled. The threat of misinformed press does not seem to be sufficient to justify censuring open discussion of the issues imho.
On Sun, 18 May 2008, Joel Jaeggli wrote:
Dragos Ruiu wrote:
First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools.
I have 6 years worth of rancid logs for every time the reported number of blocks in use on my flash changes, I imagine others do as well. That's hardly the silver bullet however.
We as I imagine others do expended a fair amount of cycles monitoring who it is that our routers are talking to and protecting the integrity of the communications channels that they use (bgp, ospf, ssh, tftp etc), If a router has a tcp connection to someplace it shouldn't we'll probably know about it. If it's announcing a prefix it shouldn't be, we'll probably know about it, those are the easy ones though.
I am very happy to hear you do these... very useful and will catch quite a bit.
There are some things one might consider adding in terms of auditing, comparing the running image more closely to the one in flash for example, peroidic checksum of the on onflash image, after downloading to another host would be another. I'm not sure that I'd trust the later given the rooted box can I suppose hand you an unmodified version of the subverted image.
The result from your check can easily be modified, first thing I would have changed is the checker. Say you did this from a usb stick--I'd just hide the rootkit in memory.
In the end if you subvert a router, presumably you're doing it for a purpose and given what the device does, that purpose is probably detectable in a well instrumented network.
Subversion may not be the goal. A router is perfect for faking outgoing traffic. This traffic can contain stolen sniffed or relayed data.
It is desirable I expect to insure that any locally stored security credentials that might be subverted not be usable when connecting to another router, that applies in a absence of root kits however.
Gadi Evron wrote:
On Sun, 18 May 2008, Joel Jaeggli wrote:
Dragos Ruiu wrote:
First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools.
I have 6 years worth of rancid logs for every time the reported number of blocks in use on my flash changes, I imagine others do as well. That's hardly the silver bullet however.
We as I imagine others do expended a fair amount of cycles monitoring who it is that our routers are talking to and protecting the integrity of the communications channels that they use (bgp, ospf, ssh, tftp etc), If a router has a tcp connection to someplace it shouldn't we'll probably know about it. If it's announcing a prefix it shouldn't be, we'll probably know about it, those are the easy ones though.
I am very happy to hear you do these... very useful and will catch quite a bit.
There are some things one might consider adding in terms of auditing, comparing the running image more closely to the one in flash for example, peroidic checksum of the on onflash image, after downloading to another host would be another. I'm not sure that I'd trust the later given the rooted box can I suppose hand you an unmodified version of the subverted image.
The result from your check can easily be modified, first thing I would have changed is the checker.
That is a normal thing to do with rootkits (return bogus results). Which is part of the reason I suggested that method I did. Short of pulling the flash you're not going to get a fully unbiased view of what's it on it thusly the audit process has some limitations. A TCPA style boot process would be a better approach. It's certainly not a quick fix since it in general can't be retrofited to existing products.
Say you did this from a usb stick--I'd just hide the rootkit in memory.
In the end if you subvert a router, presumably you're doing it for a purpose and given what the device does, that purpose is probably detectable in a well instrumented network.
Subversion may not be the goal. A router is perfect for faking outgoing traffic. This traffic can contain stolen sniffed or relayed data.
If my device is now taking marching orders from a third party then by definition it is subverted, regardless of agency or activity. sub verte - turn from under
On Sun, 18 May 2008, Joel Jaeggli wrote:
The result from your check can easily be modified, first thing I would have changed is the checker.
That is a normal thing to do with rootkits (return bogus results). Which is part of the reason I suggested that method I did. Short of pulling the flash you're not going to get a fully unbiased view of what's it on it thusly the audit process has some limitations.
A TCPA style boot process would be a better approach. It's certainly not a quick fix since it in general can't be retrofited to existing products.
EuSecWest released this interview about the rootkit with its creator, Sebastian Muniz of Core Security, it also mentions a third party product to detect some of these issues. Thank whatever diety we like for FX's work, as obviously Cisco isn't there yet. http://eusecwest.com/sebastian-muniz-da-ios-rootkit.html
Say you did this from a usb stick--I'd just hide the rootkit in memory.
In the end if you subvert a router, presumably you're doing it for a purpose and given what the device does, that purpose is probably detectable in a well instrumented network.
Subversion may not be the goal. A router is perfect for faking outgoing traffic. This traffic can contain stolen sniffed or relayed data.
If my device is now taking marching orders from a third party then by definition it is subverted, regardless of agency or activity.
sub verte - turn from under
its worth a digg... <http://digg.com/security/Da_IOS_Rootkit> regards -- "Use your imagination not to scare yourself to death but to inspire yourself to life." Les enfants teribbles - research and deployment Marc Manthey - head of research and innovation Hildeboldplatz 1a D - 50672 Kƶln - Germany Tel.:0049-221-3558032 Mobil:0049-1577-3329231 jabber :marc@kgraff.net blog : http://www.let.de ipv6 http://www.ipsix.org xing : https://www.xing.com/profile/Marc_Manthey
On Sun, 18 May 2008, Joel Jaeggli wrote:
Dragos Ruiu wrote:
First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools.
I have 6 years worth of rancid logs for every time the reported number of blocks in use on my flash changes, I imagine others do as well. That's hardly the silver bullet however.
Cisco considerably updated its rootkits page (which was 3 lines, yes, just 3 lines, last week, you might think it was a previously unknown threat). Last Updated 2008 May 22 1600 UTC (GMT) For Public Release 2008 May 16 0400 UTC (GMT) Some update! The new page gives a lot of information on best practices, MD5 verifications, etc. Very good as a security best practices page but still not much of an "anti rootkit" page. Well worth taking a look: http://www.cisco.com/warp/public/707/cisco-sr-20080516-rootkits.shtml Again, very good page even if it in no way addresses the threat. Last week my opinions were well-formed after a few years of thinking on the subject. I decided to re-examine my take as I may have just stagnated on the issue and the landscape changed. I reached the same conclusions. Still no decent response on why they never spoke to their clients on Trojan horses on IOS, rootkits on IOS.. or practically, what tools they provide to deal with them or what their plans are to help us protect ourselves and our infrastructure. One could guess they have non. As someone recently mentioned to me, after the Michael Lynn talk they started admitting to remote code execution vulnerabilities being more than just DoS in their announcements. Maybe that is a trend and we will get more information from them in the future, now that rootkits as a threat to IOS is a publis issue. Cisco's "threats don't exist until our clients already know of them" strategy is running out of steam, and will soon outlive its usefulness. Cisco is acting pretty much like Microsoft did 10 years ago, they shouldn't be surprised if security research treats them the same way as it treated Microsoft. I know what their treatment made _me_ do psychologically, it made me not want to reach out to them. It seems like the Michael Lynn way is the only way to go with their current attitude--full disclosure. As to the risk itself, it is my personal belief IOS rootkits are currently a threat as a targeted attack. Therefore, although of serious concern it is not yet something I fear on the Internet scale. Pure FUD, Cisco provided us with no real data: I do however dread the day XR gains some popularity, then it is as bad as Windows XP exploitability-wise. 2003, year of the worm. 2013, year of the Cisco worms? Gadi.
any news of the presentation surfacing anywhere? interested to details of what was discussed On Sun, May 25, 2008 at 6:27 AM, Gadi Evron <ge@linuxbox.org> wrote:
On Sun, 18 May 2008, Joel Jaeggli wrote:
Dragos Ruiu wrote:
First of all about prevention, I'm not at all sure about this being
covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools.
I have 6 years worth of rancid logs for every time the reported number of blocks in use on my flash changes, I imagine others do as well. That's hardly the silver bullet however.
Cisco considerably updated its rootkits page (which was 3 lines, yes, just 3 lines, last week, you might think it was a previously unknown threat).
Last Updated 2008 May 22 1600 UTC (GMT) For Public Release 2008 May 16 0400 UTC (GMT) Some update!
The new page gives a lot of information on best practices, MD5 verifications, etc. Very good as a security best practices page but still not much of an "anti rootkit" page. Well worth taking a look:
http://www.cisco.com/warp/public/707/cisco-sr-20080516-rootkits.shtml
Again, very good page even if it in no way addresses the threat.
Last week my opinions were well-formed after a few years of thinking on the subject. I decided to re-examine my take as I may have just stagnated on the issue and the landscape changed. I reached the same conclusions.
Still no decent response on why they never spoke to their clients on Trojan horses on IOS, rootkits on IOS.. or practically, what tools they provide to deal with them or what their plans are to help us protect ourselves and our infrastructure. One could guess they have non.
As someone recently mentioned to me, after the Michael Lynn talk they started admitting to remote code execution vulnerabilities being more than just DoS in their announcements. Maybe that is a trend and we will get more information from them in the future, now that rootkits as a threat to IOS is a publis issue.
Cisco's "threats don't exist until our clients already know of them" strategy is running out of steam, and will soon outlive its usefulness. Cisco is acting pretty much like Microsoft did 10 years ago, they shouldn't be surprised if security research treats them the same way as it treated Microsoft.
I know what their treatment made _me_ do psychologically, it made me not want to reach out to them. It seems like the Michael Lynn way is the only way to go with their current attitude--full disclosure.
As to the risk itself, it is my personal belief IOS rootkits are currently a threat as a targeted attack. Therefore, although of serious concern it is not yet something I fear on the Internet scale.
Pure FUD, Cisco provided us with no real data: I do however dread the day XR gains some popularity, then it is as bad as Windows XP exploitability-wise. 2003, year of the worm. 2013, year of the Cisco worms?
Gadi.
On Sun, 18 May 2008 13:33:53 -0700 Dragos Ruiu <dr@kyx.net> wrote:
On 18-May-08, at 7:11 AM, Suresh Ramasubramanian wrote:
2. It can be prevented by what's widely regarded as BCP on router security, and has been covered at *nog, in cisco training material, etc etc for quite some time now.
I am much less concerned about security conferences discussing this than about the (highly uninformed) publicity that accompanies these conferences.
I'm not going to touch the disclosure or not debate... it's been done.
But I will agree to disagree with you about the above two points.
First of all about prevention, I'm not at all sure about this being covered by existing router security planning / BCP. I don't believe most operators reflash their routers periodically, nor check existing images (particularly because the tools for this integrity verification don't even exist). If I'm wrong about this I would love to be corrected with pointers to the tools.
<snip> Cisco publish an MD5 sum (and BSD 'sum' IIRC) for the IOS image just before you hit the download page, which you can record and then verify after downloading (although they could make this a *lot* easier to do by posting it after, or before and after the download). There is also a /md5 option for the IOS verify command which can be used to generate an MD5 sum for an IOS image stored on a router. Regards, Mark. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
On Mon, May 19, 2008 at 2:03 AM, Dragos Ruiu <dr@kyx.net> wrote:
So in my opinion the benefits of discussing serious issues at conferences far outweigh the potential drawbacks of misguided media coverage of them. What I infer from your post is that you are of the opinion that issues such
Well, there are any number of closed, no media, relevant people only conferences, or communities like nsp-sec, that come in useful Report to CSIRT by all means but that doesnt imply "brush it under the carpet". Getting releases out and fixes (if only router management bcp like in Joel Jaeggli's post) without various people spreading FUD about it should certainly be an achievable goal? srs
On Sun, 18 May 2008, Dragos Ruiu wrote:
On 17-May-08, at 3:12 AM, Suresh Ramasubramanian wrote:
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft <mmc@internode.com.au> wrote:
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
This needs fixing. It doesnt need publicity at security conferences till after cisco gets presented this stuff first and asked to release an emergency patch.
Bullshit.
There is nothing to patch.
It needs to be presented at conferences, exactly because people will play ostrich and stick their heads in the sand and pretend it can't happen to them, and do nothing about it until someone shows them, "yes it can happen" and here is how....
Which is exactly why we've accepted this talk. We've all known this is a possibility for years, but I haven't seen significant motion forward on this until we announced this talk. So in a fashion, this has already helped make people more realistic about their infrastructure devices. And the discussions, and idea interchange that will happen between the smart folks at the conference will undoubtedly usher forth other related issues and creative solutions. Problems don't get fixed until you talk about them.
Dragus, while I hold full disclosure very close and it is dear to my heart, I admit the fact that it can be harmful. Let me link that to network operations. People forget history. A few years back I had a chat with Aleph1 on the first days of bugtraq. He reminded me how things are not always black and white. Full disclosure, while preferable in my ideology, is not the best solution for all. One of the reasons bugtraq was created is because vendors did not care about security, not to mention have a capability to handle security issues, or avoid them to begin with. Full disclosure made a lot of progress for us, and while still a useful tool, with some vendors it has become far more useful to report to them and let them provide with a solution first. In the case of routers which are used for infrastructure as well as critical infrastructure, it is my strong belief that full disclosure is, at least at face value, a bad idea. I'd like to think Cisco, which has shown capability in the past, is as responsible as it should be on these issues. Experience tells me they have a ways to go yet even if they do have good processes in place with good people to employ them. I'd also like to think tier-1 and tier-2 providers get patches first before such releases. This used to somewhat be the case, last I checked it no longer is -- for legitimate concerns by Cisco. has this changed? So, if we don't patch the infrastructure up first, and clients don't know of problems until they are public "for their own security" (an argument that holds water only so much) perhaps it is the time for full disclosure to be considered a viable alternative. All that aside, this is a rootkit, not a vulnerability. There is no inherent vulnerability to patch (unless it is very local). There is the vulnerability of operators who don't so far even consider trojan horses as a threat, and the fact tools don't exist for them to do something once they do. Gadi.
cheers, --dr
-- World Security Pros. Cutting Edge Training, Tools, and Techniques London, U.K. May 21/22 - 2008 http://cansecwest.com pgpkey http://dragos.com/ kyxpgp
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
On Sat, May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote:
I'm sure it'll be good for a number of security providers to hawk their wares.
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
I personally like Gadi's work, but not as much as I like getting my packets to their destination. I personally don't quite understand why netops keep buying proprietary, closed technology for routers, but I'm not and have never been a netop so I'm sure there's good reasons. To me it seems that if you need reliable router hardware, you can buy that from a vendor, but in theory I don't see why the software for routers couldn't be much more open. When I can, I reflash my WAPs with DD-WRT, because at least then I understand the system (and you can't secure what you don't understand), but I am not saying that's much of a comparison. So, speaking of hawking wares... ;-) Since I see some disclosure discussions brewing here, so I thought I'd mention that I have a free online book on security, and I'm trying to capture all the arguments about disclosure policies so that they don't ever have to be rehashed. Instead, we can just point someone to it, and move on. Here's the section on disclosure: http://www.subspacefield.org/security/security_concepts.html#tth_sEc25.1 I'm numbering them for your convenience, so that if for some reason you want to state a particular argument, you can compress the conversation by simply giving its index. ;-) HHOS, Travis -- Crypto ergo sum. https://www.subspacefield.org/~travis/ If you are a spammer, please email john@subspacefield.org to get blacklisted.
On Sat, 17 May 2008 09:34:19 -0500 travis+ml-nanog@subspacefield.org wrote:
On Sat, May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote:
I'm sure it'll be good for a number of security providers to hawk their wares.
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
I personally like Gadi's work, but not as much as I like getting my packets to their destination. I personally don't quite understand why netops keep buying proprietary, closed technology for routers, but I'm not and have never been a netop so I'm sure there's good reasons. To me it seems that if you need reliable router hardware, you can buy that from a vendor, but in theory I don't see why the software for routers couldn't be much more open. When I can, I reflash my WAPs with DD-WRT, because at least then I understand the system (and you can't secure what you don't understand), but I am not saying that's much of a comparison.
Have you read and security validated every line of open code you're running? Even if you've only read and security validated 99% of it, you're still trusting that the other 1% doesn't have any vulnerabilities in it. Then again, even if you have audited every line of code, and it is 100% "secure", who's to say the compiler used to compile it is ... so you'll have to audit that too. "Reflections on Trusting Trust" http://cm.bell-labs.com/who/ken/trust.html And then, even if you trust your build environment because you've audited it, who's to say that the CPU itself doesn't have a vulnerability in its microcode (accidental or intentionally), or the harddrive, or the TOE network card (that can probably establish TCP connections completely transparent to the OS it's working fo) or the keyboard doesn't have a built in key sniffer, inserted by the chip manufacturer when the designers outsourced it. (The TOE card is an interesting one - they'd commonly be used on high performance servers, and typically those are holding or have access to the organisation's most sensitive data ...) So the only way you can be absolutely sure is to build everything or audit everything yourself, from the transistor level up through the layers OSI RM. And even then, you're battling your own human frailty - how can you be sure you haven't missed something? As the cliche goes, "If you want something done properly, you have to do it yourself." If you can't do it (all) yourself, because you don't have the time and the expertise, then inherently you have to place a level of trust in other people. Regards, Mark. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
On Sun, 18 May 2008, Mark Smith wrote:
"Reflections on Trusting Trust" http://cm.bell-labs.com/who/ken/trust.html
That is the #1 paper on security anyone can read, and reading your email I was about to ask if you ever read it. It certainly is my fav. Thanks for reminding us all of the url. Gadi.
On Sun, 18 May 2008 09:29:47 +0930 Mark Smith <nanog@85d5b20a518b8f6864949bd940457dc124746ddc.nosense.org> wrote:
On Sat, 17 May 2008 09:34:19 -0500 travis+ml-nanog@subspacefield.org wrote:
On Sat, May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote:
I'm sure it'll be good for a number of security providers to hawk their wares.
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again.
I personally like Gadi's work, but not as much as I like getting my packets to their destination. I personally don't quite understand why netops keep buying proprietary, closed technology for routers, but I'm not and have never been a netop so I'm sure there's good reasons. To me it seems that if you need reliable router hardware, you can buy that from a vendor, but in theory I don't see why the software for routers couldn't be much more open. When I can, I reflash my WAPs with DD-WRT, because at least then I understand the system (and you can't secure what you don't understand), but I am not saying that's much of a comparison.
<snip>
As the cliche goes, "If you want something done properly, you have to do it yourself." If you can't do it (all) yourself, because you don't have the time and the expertise, then inherently you have to place a
should have been "have the time and/or the expertise"
level of trust in other people.
Regards, Mark.
--
"Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
-- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
Mark Smith wrote:
On Sat, 17 May 2008 09:34:19 -0500 travis+ml-nanog@subspacefield.org wrote:
I'm sure it'll be good for a number of security providers to hawk their wares.
If the way of running this isn't out in the wild and it's actually dangerous then a pox on anyone who releases it, especially to gain publicity at the expensive of network operators sleep and well being. May you never find a reliable route ever again. I personally like Gadi's work, but not as much as I like getting my
On Sat, May 17, 2008 at 04:47:02PM +0930, Matthew Moyle-Croft wrote: packets to their destination. I personally don't quite understand why netops keep buying proprietary, closed technology for routers, but I'm not and have never been a netop so I'm sure there's good reasons. To me it seems that if you need reliable router hardware, you can buy that from a vendor, but in theory I don't see why the software for routers couldn't be much more open. When I can, I reflash my WAPs with DD-WRT, because at least then I understand the system (and you can't secure what you don't understand), but I am not saying that's much of a comparison.
Have you read and security validated every line of open code you're running? Even if you've only read and security validated 99% of it, you're still trusting that the other 1% doesn't have any vulnerabilities in it.
There are people who routinely deal in absolutes. we generally call them mathematicians... The rest of us have to operate on a certain amount of uncertainty. Ken's goal I think in 1985 was to open people's eyes to an area of uncertainty which was then relatively poorly understood. It was infeasible in 1985 and certainly remains so outside the confines of some really narrowly focused areas to audit a significant percentage of the code you run.
Then again, even if you have audited every line of code, and it is 100% "secure", who's to say the compiler used to compile it is ... so you'll have to audit that too.
On Sat, 17 May 2008, Paul Wall wrote:
What if some good comes from this "root kit"?
For instance, what if it lets us fix things like DOM on non-Cisco XENPAKs and SFPs? Or lets us un-cripple our 6500 chassis to run the code we want?
Of course, given the messenger, I'm sure it's just hype to help bolster Gadi's security practice, and will prove to be no big deal.
A signed issue is now 25 bucks FOR YOU, Mister.
Paul
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
participants (15)
-
Aaron Glenn -
Christian -
Dragos Ruiu -
Gadi Evron -
Joel Jaeggli -
Jon Kibler -
Marc Manthey -
Mark Smith -
Matthew Moyle-Croft -
n3td3v -
Paul Ferguson -
Paul Wall -
Simon Lockhart -
Suresh Ramasubramanian -
travis+ml-nanogļ¼ subspacefield.org