Re: NSA able to compromise Cisco, Juniper, Huawei switches
Hi Folks - Clay Kossmeyer here from the Cisco PSIRT. We've published the following document in response to the original (Dec. 29) Der Spiegel article: http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-s... and are investing the claims in the Dec. 30 Der Spiegel article referencing 'persistent implants' for the PIX and ASA product lines under case number PSIRT-1384943056. Any vulnerabilities we discover will be disclosed via our standard vulnerability handling process documented here: http://www.cisco.com/web/about/security/psirt/security_vulnerability_policy.... I'm not currently subscribed to NANOG, so if you have a reply you'd like me to see, please copy me directly. Regards, Clay ----- Found some interesting news on one of the Australia news websites. http://www.scmagazine.com.au/News/368527,nsa-able-to-compromise-cisco-junipe... Regards, Steven.
Clay Kossmeyer here from the Cisco PSIRT.
shoveling kitty litter as fast as you can, eh?
http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-s...
"The article does not discuss or disclose any Cisco product vulnerabilities." this is disengenuous at best. from the nsa document copied in der spiegel and now many other places: "JETPLOW is a firmware persistence implant for Cisco PIX series and ASA firewalls ..." so in cisco kitty litter lingo, what would be "discuss[ing] or disclos[ing] any Cisco product vulnerabilities? the exploit code itself? randy
On 12/30/2013 3:51 PM, Randy Bush wrote:
Clay Kossmeyer here from the Cisco PSIRT.
shoveling kitty litter as fast as you can, eh?
http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-s...
"The article does not discuss or disclose any Cisco product vulnerabilities."
this is disengenuous at best. from the nsa document copied in der spiegel and now many other places:
"JETPLOW is a firmware persistence implant for Cisco PIX series and ASA firewalls ..."
so in cisco kitty litter lingo, what would be "discuss[ing] or disclos[ing] any Cisco product vulnerabilities? the exploit code itself?
randy
What is the vulnerability in Cisco product Randy? That a 3rd party can replace the firmware in your firewall? There isn't enough information to determine if this is a software vulnerability triggered with exploit code or wholesale firmware replacement. The document refers to an implant but not how it gets there. -- "The first rule of any game is to know that you're in one." -Sandy Lerner, co-founder, Cisco Systems
* Randy Bush:
Clay Kossmeyer here from the Cisco PSIRT.
shoveling kitty litter as fast as you can, eh?
http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-s...
"The article does not discuss or disclose any Cisco product vulnerabilities."
this is disengenuous at best. from the nsa document copied in der spiegel and now many other places:
"JETPLOW is a firmware persistence implant for Cisco PIX series and ASA firewalls ..."
There's a limit to what can reasonably be called a *product* vulnerability. If you physically plant a bug in a phone, does it exploit a vulnerability in the phone? I don't think so. Theoretically, the manufacturer could have filled it completely with glue. But the next step up is drilling out some of that to place the bug, and then you're looking at tamper evidence, and that's an extremely difficult matter. Routers are expected to be modular, so it's difficult to avoid that they have exposed buses with something that approaches DMA capability. On-site debugging hooks through JTAG ports or similar might be essential to reduce downtime in case of severe problems, so I doubt one can get rid of them. Same for firmware downgrade and recovery options. In the end, the defense has to be political, not technical. "We don't want to do this because it's wrong", and not "we can't do this because it's impossible". After all, what's possible can change very quickly. Appeasement in the form of lawful intercept turned out to be failure: even if you comply, it's likely that your own, domestic intelligence agencies consider your infrastructure, you and your colleagues legitimate targets.
There's a limit to what can reasonably be called a *product* vulnerability.
right. if the product was wearing a low-cut blouse and a short skirt, it's not. it's weasel words (excuse the idiom). shoveling kitty litter over a big steaming pile. let me insert a second advert for jake's 30c3 preso, https://www.youtube.com/watch?v=b0w36GAyZIA randy
+1 NSA states very clearly this is baked in and ³widely deployed². Either Cisco is not very happy with their government overlords today, or they are having long meetings at those oversized conference tables trying to figure out what to tell everyone. I¹m curious about the implications to the US DoD STIG¹s that are put out, as I¹m fairly sure they do not mention there is a backdoor that anyone who knows how to knock can access. My other question is.. How are they identifying unique ASA and PIX? Is there a fingerprint mechanism that tells it what¹s going on? I¹d think there would be quite a few admins out there with really weird syslog entries?? Randy is right here.. Cisco has some Œsplainin to do - we buy these devices as ³security appliances², not NSA rootkit gateways. I hope the .cn guys don¹t figure out what¹s going on here, I¹d imagine there are plenty of ASA¹s in the .gov infrastructures. //warren PS - I mentioned .cn specifically because of the Huawei aspect, in addition to the fact that it has been widely publicized we are in a ³cyber war² with them. On 12/31/13, 12:07 PM, "Randy Bush" <randy@psg.com> wrote:
There's a limit to what can reasonably be called a *product* vulnerability.
right. if the product was wearing a low-cut blouse and a short skirt, it's not.
it's weasel words (excuse the idiom). shoveling kitty litter over a big steaming pile.
let me insert a second advert for jake's 30c3 preso, https://www.youtube.com/watch?v=b0w36GAyZIA
randy
The best response I've seen to all this hype and I completely agree with Scott: "Do ya think that you wouldn't also notice a drastic increase in outbound traffic to begin with? It's fun to watch all the hype and things like that, but to truly sit down and think about what it would actually take to make something like this happen, especially on a sustained and "unnoticed" basis, is just asinine. Perhaps more work should be spent maintaining ones own equipment and network than debating the chances that the sky may actually be falling or the NSA hunting your ass down. ;) Just my two cents for the day! Happy New Year! Scott Morris, CCIEx4 (R&S/ISP-Dial/Security/Service Provider) #4713, CCDE #2009::D, CCNP-Data Center, CCNP-Voice, JNCIE-SP #153, JNCIE-ENT #102, JNCIS-QFX, CISSP, et al. IPv6 Gold Certified Engineer, IPv6 Gold Certified Trainer CCSI #21903, JNCI-SP, JNCI-ENT, JNCI-QFX swm@emanon.com Knowledge is power. Power corrupts. Study hard and be Eeeeviiiil......" Jonathan On Tue, Dec 31, 2013 at 11:16 AM, Warren Bailey < wbailey@satelliteintelligencegroup.com> wrote:
+1
NSA states very clearly this is baked in and ³widely deployed². Either Cisco is not very happy with their government overlords today, or they are having long meetings at those oversized conference tables trying to figure out what to tell everyone. I¹m curious about the implications to the US DoD STIG¹s that are put out, as I¹m fairly sure they do not mention there is a backdoor that anyone who knows how to knock can access.
My other question is.. How are they identifying unique ASA and PIX? Is there a fingerprint mechanism that tells it what¹s going on? I¹d think there would be quite a few admins out there with really weird syslog entries??
Randy is right here.. Cisco has some Œsplainin to do - we buy these devices as ³security appliances², not NSA rootkit gateways. I hope the .cn guys don¹t figure out what¹s going on here, I¹d imagine there are plenty of ASA¹s in the .gov infrastructures.
//warren
PS - I mentioned .cn specifically because of the Huawei aspect, in addition to the fact that it has been widely publicized we are in a ³cyber war² with them.
On 12/31/13, 12:07 PM, "Randy Bush" <randy@psg.com> wrote:
There's a limit to what can reasonably be called a *product* vulnerability.
right. if the product was wearing a low-cut blouse and a short skirt, it's not.
it's weasel words (excuse the idiom). shoveling kitty litter over a big steaming pile.
let me insert a second advert for jake's 30c3 preso, https://www.youtube.com/watch?v=b0w36GAyZIA
randy
-- Jonathan Greenwood II CCIE #22744
On Jan 1, 2014, at 2:34 AM, Jonathan Greenwood II <gwood83@gmail.com> wrote:
The best response I've seen to all this hype and I completely agree with Scott:
"Do ya think that you wouldn't also notice a drastic increase in outbound traffic to begin with? It's fun to watch all the hype and things like that, but to truly sit down and think about what it would actually take to make something like this happen, especially on a sustained and "unnoticed" basis, is just asinine.
Hopefully, this drives home the importance of all the various BCPs like iACLs, isolated jump-off boxes for interactive access, config-file management, and network telemetry - including visibility into DCN/OOB traffic. There are open-source tools out there which can be used for these purposes. It doesn't require a lot of capex, mainly opex - i.e., elbow-grease. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
The best response I've seen to all this hype and I completely agree with Scott:
"Do ya think that you wouldn't also notice a drastic increase in outbound traffic to begin with? It's fun to watch all the hype and things like that, but to truly sit down and think about what it would actually take to make something like this happen, especially on a sustained and "unnoticed" basis, is just asinine.
A drastic increase, definitely. Smaller increases (say a couple of Mbps on a link normally carrying 100 Mbps or more), doubtful. It all depends on the volume of the information you're looking for. Steinar Haug, Nethelp consulting, sthaug@nethelp.no
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/31/2013 12:33 PM, sthaug@nethelp.no wrote:
The best response I've seen to all this hype and I completely agree with Scott:
"Do ya think that you wouldn't also notice a drastic increase in outbound traffic to begin with? It's fun to watch all the hype and things like that, but to truly sit down and think about what it would actually take to make something like this happen, especially on a sustained and "unnoticed" basis, is just asinine.
A drastic increase, definitely. Smaller increases (say a couple of Mbps on a link normally carrying 100 Mbps or more), doubtful.
It all depends on the volume of the information you're looking for.
More than you know. As someone who has seen firsthand, in real time, an adversary exfiltrate documents and other data out of an organization which he has gained unauthorized internal access -- real professionals know how to blend in with the noise & fly under the radar successfully. - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 10.2.0 (Build 2317) Charset: utf-8 wj8DBQFSwywoq1pz9mNUZTMRAtFaAKDrbdnfnnPOP6G0DSRUxK4WmbtGhwCfRaQ/ V7MRFxg+dGwNKZgx4qK0Ogs= =XiSA -----END PGP SIGNATURE----- -- Paul Ferguson PGP Public Key ID: 0x63546533
On Jan 1, 2014, at 2:16 AM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
Randy is right here.. Cisco has some Œsplainin to do - we buy these devices as ³security appliances², not NSA rootkit gateways
<http://blogs.cisco.com/news/comment-on-der-spiegel-articles-about-nsa-tao-organization/> <http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131229-der-spiegel> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Explaining, not a denial written by their legal department. I find it insanely difficult to believe cisco systems has a backdoor into some of their product lines with no knowledge or participation. Given the fact that RSA had a check cut for their participation (sell outs..), would it be out of the realm of possibility cisco knowingly placed this into their product line? And would it be their mistake to come out with a “we had no idea!” rather than “guys with badges and court orders made us do it!”? Google has some deniability, as their networks were compromised without their knowledge. Placing code into a PC BIOS or IOS image is a far different beast than asking a fiber provider to give a split to a governmental agency. Secret squirrel wires with secret squirrel modulation techniques isn’t a surprise to me, what is a surprise to me is the level of acceptance the IT community has shown thus far on NANOG. On a side note, I found it unbelievable the NSA was so pissed off about aeronautical access being hard to capture. The initial article made it seem like they had already gotten ahold of the data, which would have really pissed me off. If it’s really that difficult, I have a NSA proof satellite platform with capacity should anyone need it.. ;) //warren On 12/31/13, 12:34 PM, "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On Jan 1, 2014, at 2:16 AM, Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:
Randy is right here.. Cisco has some Œsplainin to do - we buy these devices as ³security appliances², not NSA rootkit gateways
<http://blogs.cisco.com/news/comment-on-der-spiegel-articles-about-nsa-tao -organization/>
<http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisc o-sr-20131229-der-spiegel>
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Luck is the residue of opportunity and design.
-- John Milton
* Warren Bailey:
Explaining, not a denial written by their legal department. I find it insanely difficult to believe cisco systems has a backdoor into some of their product lines with no knowledge or participation.
As far as I understand it, these are firmware tweaks or implants sitting on a privileged bus (think PCI with busmaster DMA). Such things can be added after the device has left the factory by a sufficiently knowledgeable third party.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/31/2013 4:02 PM, Florian Weimer wrote:
* Warren Bailey:
Explaining, not a denial written by their legal department. I find it insanely difficult to believe cisco systems has a backdoor into some of their product lines with no knowledge or participation.
As far as I understand it, these are firmware tweaks or implants sitting on a privileged bus (think PCI with busmaster DMA). Such things can be added after the device has left the factory by a sufficiently knowledgeable third party.
That's really interesting. Where are these Cisco devices manufactured? - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 10.2.0 (Build 2317) Charset: utf-8 wj8DBQFSw1z/q1pz9mNUZTMRAvbIAKCYZn3slg1wMak/nlc/hb3ZHkS29wCg3ucb OJTl+SLgBtQDMGi+cTdDRtQ= =VAdw -----END PGP SIGNATURE----- -- Paul Ferguson PGP Public Key ID: 0x63546533
China. ;) lol Sent from my Mobile Device. -------- Original message -------- From: Paul Ferguson <fergdawgster@mykolab.com> Date: 12/31/2013 4:13 PM (GMT-08:00) To: nanog@nanog.org Subject: Re: NSA able to compromise Cisco, Juniper, Huawei switches -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/31/2013 4:02 PM, Florian Weimer wrote:
* Warren Bailey:
Explaining, not a denial written by their legal department. I find it insanely difficult to believe cisco systems has a backdoor into some of their product lines with no knowledge or participation.
As far as I understand it, these are firmware tweaks or implants sitting on a privileged bus (think PCI with busmaster DMA). Such things can be added after the device has left the factory by a sufficiently knowledgeable third party.
That's really interesting. Where are these Cisco devices manufactured? - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 10.2.0 (Build 2317) Charset: utf-8 wj8DBQFSw1z/q1pz9mNUZTMRAvbIAKCYZn3slg1wMak/nlc/hb3ZHkS29wCg3ucb OJTl+SLgBtQDMGi+cTdDRtQ= =VAdw -----END PGP SIGNATURE----- -- Paul Ferguson PGP Public Key ID: 0x63546533
On (2013-12-31 23:04 +0000), Warren Bailey wrote:
that RSA had a check cut for their participation (sell outs..), would it be out of the realm of possibility cisco knowingly placed this into their product line? And would it be their mistake to come out with a “we had no idea!” rather than “guys with badges and court orders made us do it!”?
Is this legal? Can NSA walk in to US based company and legally coerce to install such backdoor? If not, what is the incentive for private company to cooperate? If legal, consider risk to NSA. Official product ran inside company to add requested feature, hundred of people aware of it. Seems both expensive to order such feature and almost guaranteed to be exposed by some of the employees. Alternative method is to presume all software is insecure, hire 1 expert whose day job is to search for vulnerabilities in IOS. Much cheaper, insignificant risk. Which method would you use?
techniques isn’t a surprise to me, what is a surprise to me is the level of acceptance the IT community has shown thus far on NANOG.
This seems like generalization, majority opinion seems to be, government has no business spying on us. Someone contacted me yesterday, after reading how I'd love to see some of these attacks dissected and analysed to gain higher quality data than screenshot of PDF. He told me, he and his employer are cooperating with their vendor right now looking at attack done against router they operate and claimed they are aware of other operators being targeted. Unfortunately he couldn't share any specifics, so hopefully we'll soon have situation where someone can dissect publicly any of the attacks. If this is as widespread as claimed, and if we'll gain knowledge how to see if you are affected, there are potentially repercussions on geopolitical scale, as I'm sure many on these lists would go public and share information if they'd find being targeted. -- ++ytti
On Wed, 01 Jan 2014 11:55:37 +0200, Saku Ytti said:
Is this legal? Can NSA walk in to US based company and legally coerce to install such backdoor?
Well, legal or not... we will probably never know exactly what was said, but apparently the NSA was able to convince/coerce many of the 800 pound telecom gorillas to install taps and backdoors at the server end.
If not, what is the incentive for private company to cooperate?
The same incentive that was used to enforce secrecy on National Security Letters for many years - play nice or you'll end up in an oublette, with a trial to (maybe) be held behind closed doors, where you won't see any of the evidence against you because it's classified. Remember, the US sprouted this "indefinite detention" concept a while back, and still hasn't backtracked on it, because of its enormous usefulness as a cudgel to deal with "enemies of the State".
On Wed, Jan 1, 2014 at 3:55 AM, Saku Ytti <saku@ytti.fi> wrote:
Is this legal? Can NSA walk in to US based company and legally coerce to install such backdoor? If not, what is the incentive for private company to cooperate?
As evidenced by "Lavabit"; apparently, one thing that they CAN do is issue an order to the US based company to release their secret cryptography keys such as RSA secret keys to the government, including the secret keys that correspond to the public keys on their X509 certificates; possibly including certificates used for code signing and code distribution to users. AND maintain confidentiality that they were required to release keys. Recall, Lavabit was deemed in violation of the order: due to halting their service, after being forced to release the cryptography keys. The RSA secret keys can then be used to forge the company's signature on a payload containing a malicious copy of the firmware or operating system. And perform man in the middle attacks against web sites, and other software update infrastructure --- in order to distributed tampered with software with forged code signatures. -- -JH
On Wed, Jan 1, 2014 at 11:55 AM, Saku Ytti <saku@ytti.fi> wrote:
On (2013-12-31 23:04 +0000), Warren Bailey wrote:
that RSA had a check cut for their participation (sell outs..), would it be out of the realm of possibility cisco knowingly placed this into their product line? And would it be their mistake to come out with a “we had no idea!” rather than “guys with badges and court orders made us do it!”?
Is this legal? Can NSA walk in to US based company and legally coerce to install such backdoor? If not, what is the incentive for private company to cooperate?
As you might have seen from the beginning of time, people in power assume anything can go until proven otherwise.
On (2014-01-01 23:51 +0200), Eugeniu Patrascu wrote:
Is this legal? Can NSA walk in to US based company and legally coerce to install such backdoor? If not, what is the incentive for private company to cooperate?
As you might have seen from the beginning of time, people in power assume anything can go until proven otherwise.
This is mostly academic, as being legal or not being legal it's not appealing attack vector due to difficulties containing the information. But what I implied is, if it is legal, you'd have paper trail, like legal document from court. -- ++ytti
On Thu, Jan 2, 2014 at 10:01 AM, Saku Ytti <saku@ytti.fi> wrote:
On (2014-01-01 23:51 +0200), Eugeniu Patrascu wrote:
Is this legal? Can NSA walk in to US based company and legally coerce to install such backdoor? If not, what is the incentive for private company to cooperate?
As you might have seen from the beginning of time, people in power assume anything can go until proven otherwise.
This is mostly academic, as being legal or not being legal it's not appealing attack vector due to difficulties containing the information. But what I implied is, if it is legal, you'd have paper trail, like legal document from court.
I can't speak for NSA practices, but for example FBI asserted that they are entitled to put GPS trackers on cars owned by people they suspected of something without a court order. And they fought to the death in courts when the suspects brought suits against them for violating their rights with these practices. It would assume that other agencies employ the same tactics and strong-arm companies into doing their bidding with minimal paperwork. Let's not forget that NSA vets all the security vendors and products that the USG uses and it would be pretty easy for them to stop recommending SecurID tokens (main RSA business is authentication) for government use. The above presumption would have sounded crazy six months ago, but now...
Warren Bailey <wbailey@satelliteintelligencegroup.com>
I find it insanely difficult to believe cisco systems has a backdoor into some of their product lines with no knowledge or participation.
actually, i suspect a mix of both, the usg encouraging calea gone bad (while committing to bad-mouth huawei), and the TAO crew developing serious attacks based on unintended product vulnerabilities.
Google has some deniability, as their networks were compromised without their knowledge.
i doubt we will ever learn the extent of surprise vs culpability of google, apple, twitter, msoft, ... Saku Ytti <saku@ytti.fi>
Is this legal?
ROFL
If this is as widespread as claimed, and if we'll gain knowledge how to see if you are affected, there are potentially repercussions on geopolitical scale, as I'm sure many on these lists would go public and share information if they'd find being targeted.
we are dealing with a world in which there are attackers and victims and very few white hats to be seen. exposure via journalism, thanks @ioerror, wikileaks, ... and constructive hacking to make protocols and products more resistant are the main paths available to us. and if you want to be ambarrassed for our peers, see the ietf pissing all over itself deciding whether they can make simple statements that these things are attacks and the ietf needs to do something about its protocols. --- https://www.youtube.com/watch?v=cOCWTRJCnf0 randy
On Jan 1, 2014, at 2:07 AM, Randy Bush <randy@psg.com> wrote:
it's weasel words (excuse the idiom). shoveling kitty litter over a big steaming pile.
Clayton is responding to the ability that he's allowed, and he's using words very precisely. Here's Cisco's official responses, so far. <http://blogs.cisco.com/news/comment-on-der-spiegel-articles-about-nsa-tao-organization/> <http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131229-der-spiegel> I know both Clay and jns quite well, and they're both straight-shooters. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
* Randy Bush:
There's a limit to what can reasonably be called a *product* vulnerability.
right. if the product was wearing a low-cut blouse and a short skirt, it's not.
Uh-oh, is this an attempt at an argument based on a "blame the victim" rape analogy?
participants (13)
-
Clay Kossmeyer
-
Dobbins, Roland
-
Eugeniu Patrascu
-
Florian Weimer
-
Jimmy Hess
-
Jonathan Greenwood II
-
Paul Ferguson
-
Randy Bush
-
Saku Ytti
-
Sharif Torpis
-
sthaug@nethelp.no
-
Valdis.Kletnieks@vt.edu
-
Warren Bailey