Possible Sudden Uptick in ASA DOS?
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary. Another pair running the same code had the primary crash and fail in the same time window. So, three crashes in 4 hours in our environment. Open a TAC case on one of these for post-mortem analysis, and they interpreted the crash dump to point at a DOS bug first published in Oct. The very interesting thing; on the phone the TAC engineer said this was "the 10th one of these I've dealt with this morning". Here's the bug they reference: https://tools.cisco.com/bugsearch/bug/CSCul36176/?reffering_site=dumpcr Anyone else have observations to add on this? Mark Mayfield City of Roseville - AS 54371 Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office
On Wed 2015-Jul-08 16:58:24 +0000, Mark Mayfield <Mark.Mayfield@cityofroseville.com> wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Another pair running the same code had the primary crash and fail in the same time window.
So, three crashes in 4 hours in our environment.
Open a TAC case on one of these for post-mortem analysis, and they interpreted the crash dump to point at a DOS bug first published in Oct.
The very interesting thing; on the phone the TAC engineer said this was "the 10th one of these I've dealt with this morning".
Here's the bug they reference: https://tools.cisco.com/bugsearch/bug/CSCul36176/?reffering_site=dumpcr
Anyone else have observations to add on this?
Not sure about ASA-specific DoS and the bug you're pointing at, but we saw some NTP reflection this morning. Then there's the WSJ, NYSE, and UAL from this morning as well. Rough day on the internets?
Mark Mayfield City of Roseville - AS 54371 Network Systems Engineer
2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office
-- Hugo hugo@slabnet.com: email, xmpp/jabber PGP fingerprint (B178313E): CF18 15FA 9FE4 0CD1 2319 1D77 9AB1 0FFD B178 313E (also on textsecure & redphone)
On 08 Jul 2015, at 18:58, Mark Mayfield <Mark.Mayfield@cityofroseville.com> wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Not sure it’s related but I’ve read reports on FRNoG of ASAs crashing as well, seems related to a late leap second related issue. Regards, Michel
Really just people not patching their software after warnings more than six months ago: July-08 UPDATE: Cisco PSIRT is aware of disruption to some Cisco customers with Cisco ASA devices affected by CVE-2014-3383, the Cisco ASA VPN Denial of Service Vulnerability that was disclosed in this Security Advisory. Traffic causing the disruption was isolated to a specific source IPv4 address. Cisco has engaged the provider and owner of that device and determined that the traffic was sent with no malicious intent. Cisco strongly recommends that customers upgrade to a fixed Cisco ASA software release to remediate this issue. Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available. Jared Mauch
On Jul 8, 2015, at 1:15 PM, Michel Luczak <frnog@shrd.fr> wrote:
On 08 Jul 2015, at 18:58, Mark Mayfield <Mark.Mayfield@cityofroseville.com> wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Not sure it’s related but I’ve read reports on FRNoG of ASAs crashing as well, seems related to a late leap second related issue.
Regards, Michel
Hi Jared, thanks for update do you know provider/source ip of the source of the attack ? Colin
On 9 Jul 2015, at 12:27, Jared Mauch <jared@puck.nether.net> wrote:
Really just people not patching their software after warnings more than six months ago:
July-08 UPDATE: Cisco PSIRT is aware of disruption to some Cisco customers with Cisco ASA devices affected by CVE-2014-3383, the Cisco ASA VPN Denial of Service Vulnerability that was disclosed in this Security Advisory. Traffic causing the disruption was isolated to a specific source IPv4 address. Cisco has engaged the provider and owner of that device and determined that the traffic was sent with no malicious intent. Cisco strongly recommends that customers upgrade to a fixed Cisco ASA software release to remediate this issue.
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
Jared Mauch
On Jul 8, 2015, at 1:15 PM, Michel Luczak <frnog@shrd.fr> wrote:
On 08 Jul 2015, at 18:58, Mark Mayfield <Mark.Mayfield@cityofroseville.com> wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Not sure it’s related but I’ve read reports on FRNoG of ASAs crashing as well, seems related to a late leap second related issue.
Regards, Michel
My guess is a researcher. We saw the same issue in the past with a Cisco microcode bug and people doing ping record route. When it went across a LC with a very specific set of software it would crash. If you crashed just upgrade your code, don't hide behind blocking an IP as people now know what to send/do. It won't be long. Jared Mauch
On Jul 9, 2015, at 7:44 AM, Colin Johnston <colinj@gt86car.org.uk> wrote:
Hi Jared, thanks for update
do you know provider/source ip of the source of the attack ?
Colin
On 9 Jul 2015, at 12:27, Jared Mauch <jared@puck.nether.net> wrote:
Really just people not patching their software after warnings more than six months ago:
July-08 UPDATE: Cisco PSIRT is aware of disruption to some Cisco customers with Cisco ASA devices affected by CVE-2014-3383, the Cisco ASA VPN Denial of Service Vulnerability that was disclosed in this Security Advisory. Traffic causing the disruption was isolated to a specific source IPv4 address. Cisco has engaged the provider and owner of that device and determined that the traffic was sent with no malicious intent. Cisco strongly recommends that customers upgrade to a fixed Cisco ASA software release to remediate this issue.
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
Jared Mauch
On Jul 8, 2015, at 1:15 PM, Michel Luczak <frnog@shrd.fr> wrote:
On 08 Jul 2015, at 18:58, Mark Mayfield <Mark.Mayfield@cityofroseville.com> wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Not sure it’s related but I’ve read reports on FRNoG of ASAs crashing as well, seems related to a late leap second related issue.
Regards, Michel
you would think a researcher would stop once he realised effect being caused ? Colin
On 9 Jul 2015, at 14:08, Jared Mauch <jared@puck.nether.net> wrote:
My guess is a researcher.
We saw the same issue in the past with a Cisco microcode bug and people doing ping record route. When it went across a LC with a very specific set of software it would crash.
If you crashed just upgrade your code, don't hide behind blocking an IP as people now know what to send/do. It won't be long.
Jared Mauch
On Jul 9, 2015, at 7:44 AM, Colin Johnston <colinj@gt86car.org.uk> wrote:
Hi Jared, thanks for update
do you know provider/source ip of the source of the attack ?
Colin
On 9 Jul 2015, at 12:27, Jared Mauch <jared@puck.nether.net> wrote:
Really just people not patching their software after warnings more than six months ago:
July-08 UPDATE: Cisco PSIRT is aware of disruption to some Cisco customers with Cisco ASA devices affected by CVE-2014-3383, the Cisco ASA VPN Denial of Service Vulnerability that was disclosed in this Security Advisory. Traffic causing the disruption was isolated to a specific source IPv4 address. Cisco has engaged the provider and owner of that device and determined that the traffic was sent with no malicious intent. Cisco strongly recommends that customers upgrade to a fixed Cisco ASA software release to remediate this issue.
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
Jared Mauch
On Jul 8, 2015, at 1:15 PM, Michel Luczak <frnog@shrd.fr> wrote:
On 08 Jul 2015, at 18:58, Mark Mayfield <Mark.Mayfield@cityofroseville.com> wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Not sure it’s related but I’ve read reports on FRNoG of ASAs crashing as well, seems related to a late leap second related issue.
Regards, Michel
I’m sure they did. It could also have been any number of other things. I’m just guessing. It could have been someone trying to scan their enterprise too and went a bit rogue. Not everyone reads NANOG believe it or not :) Either way, if you haven’t upgraded for a 9 month old security advisory, shame on you. I don’t care what your change management process looks like, it’s bordering on network malpractice IMHO. - Jared
On Jul 9, 2015, at 10:09 AM, Colin Johnston <colinj@gt86car.org.uk> wrote:
you would think a researcher would stop once he realised effect being caused ?
Colin
On 9 Jul 2015, at 14:08, Jared Mauch <jared@puck.nether.net> wrote:
My guess is a researcher.
We saw the same issue in the past with a Cisco microcode bug and people doing ping record route. When it went across a LC with a very specific set of software it would crash.
If you crashed just upgrade your code, don't hide behind blocking an IP as people now know what to send/do. It won't be long.
Jared Mauch
On Jul 9, 2015, at 7:44 AM, Colin Johnston <colinj@gt86car.org.uk> wrote:
Hi Jared, thanks for update
do you know provider/source ip of the source of the attack ?
Colin
On 9 Jul 2015, at 12:27, Jared Mauch <jared@puck.nether.net> wrote:
Really just people not patching their software after warnings more than six months ago:
July-08 UPDATE: Cisco PSIRT is aware of disruption to some Cisco customers with Cisco ASA devices affected by CVE-2014-3383, the Cisco ASA VPN Denial of Service Vulnerability that was disclosed in this Security Advisory. Traffic causing the disruption was isolated to a specific source IPv4 address. Cisco has engaged the provider and owner of that device and determined that the traffic was sent with no malicious intent. Cisco strongly recommends that customers upgrade to a fixed Cisco ASA software release to remediate this issue.
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
Jared Mauch
On Jul 8, 2015, at 1:15 PM, Michel Luczak <frnog@shrd.fr> wrote:
On 08 Jul 2015, at 18:58, Mark Mayfield <Mark.Mayfield@cityofroseville.com> wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Not sure it’s related but I’ve read reports on FRNoG of ASAs crashing as well, seems related to a late leap second related issue.
Regards, Michel
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jared Mauch Sent: Thursday, July 09, 2015 9:08 AM To: Colin Johnston Cc: nanog@nanog.org Subject: Re: Possible Sudden Uptick in ASA DOS?
My guess is a researcher.
I wouldn't classify someone sending known malicious traffic towards someone else's network device attempting to crash it as a 'researcher'. Criminal is a better term. Chuck
On Jul 9, 2015, at 9:43 PM, Chuck Church <chuckchurch@gmail.com> wrote:
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jared Mauch Sent: Thursday, July 09, 2015 9:08 AM To: Colin Johnston Cc: nanog@nanog.org Subject: Re: Possible Sudden Uptick in ASA DOS?
My guess is a researcher.
I wouldn't classify someone sending known malicious traffic towards someone else's network device attempting to crash it as a 'researcher'. Criminal is a better term.
There are other terms for people who don’t maintain their equipment, it’s usually described as negligent. If my hardware were rebooting, I would be red-faced first about not having done something and not blaming someone outside. I don’t know if it was a researcher or something buggy sending packets or anything else. (I have no unique direct insight). What I do know is the ASAs under my control and purview had no issues. Take the free upgrade and move on folks. - Jared
In message <011d01d0bab1$e7890a00$b69b1e00$@gmail.com>, "Chuck Church" writes:
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jared Mauch Sent: Thursday, July 09, 2015 9:08 AM To: Colin Johnston Cc: nanog@nanog.org Subject: Re: Possible Sudden Uptick in ASA DOS?
My guess is a researcher.
I wouldn't classify someone sending known malicious traffic towards someone else's network device attempting to crash it as a 'researcher'. Criminal is a better term.
Chuck
At what point does a well formed but bug triggering packet go from "malicious" to "expected"? Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Fri, Jul 10, 2015 at 12:05:50PM +1000, Mark Andrews wrote:
In message <011d01d0bab1$e7890a00$b69b1e00$@gmail.com>, "Chuck Church" writes:
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jared Mauch Sent: Thursday, July 09, 2015 9:08 AM To: Colin Johnston Cc: nanog@nanog.org Subject: Re: Possible Sudden Uptick in ASA DOS?
My guess is a researcher.
I wouldn't classify someone sending known malicious traffic towards someone else's network device attempting to crash it as a 'researcher'. Criminal is a better term.
Chuck
At what point does a well formed but bug triggering packet go from "malicious" to "expected"?
Don't know. Lets say it was something else. i've seen well formatted things that crash BIND. When posting to bind-users list it caused people to wonder why I didn't contact the security team first. The ASA is mostly a black box, it could be any number of things from a kernel bug to IPSEC, SSH, etc.. that trigger the issue. I would say malformed packets are common. I saw trafic coming from a specific employee home link ending up corrupted when reaching our SIP server. The result was it would crash as the malformed SIP was improperly parsed. The root cause? The wireless link connecting the employee to a local water tower was taking errors and the UDP checksums still matched with the corruption. http://downloads.asterisk.org/pub/security/AST-2011-009.html Either way see above where i said it's a guess, I have no direct personal knowledge. I'm guessing someone running a honeypot or darknet would have packets from the researcher types. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
I would say it depends on the complexity and probability of it happening accidentally. An incorrect letter (language change perhaps) in a URL that crashes a web server might not be malicious. A crafted ESP or ISAKMP packet that was created in a Linux packet tool and 'randomly' hits your VPN I'd say is no accident. I agree with Jared, patch your stuff when the PSIRTs come out. But whether or not you're patched, if you're attacked, that person still is breaking the law. Think about leaving your car somewhere with the door open and keys in ignition. Someone steals it. They're still a criminal, even though you made their 'job' as easy as possible. Chuck -----Original Message----- From: Mark Andrews [mailto:marka@isc.org] Sent: Thursday, July 09, 2015 10:06 PM To: Chuck Church Cc: 'Jared Mauch'; 'Colin Johnston'; nanog@nanog.org Subject: Re: Possible Sudden Uptick in ASA DOS? In message <011d01d0bab1$e7890a00$b69b1e00$@gmail.com>, "Chuck Church" writes:
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jared Mauch Sent: Thursday, July 09, 2015 9:08 AM To: Colin Johnston Cc: nanog@nanog.org Subject: Re: Possible Sudden Uptick in ASA DOS?
My guess is a researcher.
I wouldn't classify someone sending known malicious traffic towards someone else's network device attempting to crash it as a 'researcher'. Criminal is a better term.
Chuck
At what point does a well formed but bug triggering packet go from "malicious" to "expected"? Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Thu, 09 Jul 2015 07:27:16 -0400, Jared Mauch <jared@puck.nether.net> wrote:
Really just people not patching their software after warnings more than six months ago:
A lot goes into "updates". Not the least of which is *knowing* about the issue. Then getting the patched code, then lab testing, then regulatory approval(s), then maintenance window(s)...
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
"Free" if you have a support contract. (the clause 3 "contact TAC" method is all too often a serious pain in the ass.) --Ricky
On Jul 9, 2015, at 5:35 PM, Ricky Beam <jfbeam@gmail.com> wrote:
On Thu, 09 Jul 2015 07:27:16 -0400, Jared Mauch <jared@puck.nether.net> wrote:
Really just people not patching their software after warnings more than six months ago:
A lot goes into "updates". Not the least of which is *knowing* about the issue. Then getting the patched code, then lab testing, then regulatory approval(s), then maintenance window(s)…
Not my first rodeo. Once again, it’s been since October 2014. If you failed to pay your credit card bill from October 2014 you can’t expect it to work either.
Cisco has released free software updates that address these vulnerabilities. Workarounds that mitigate some of these vulnerabilities are available.
"Free" if you have a support contract. (the clause 3 "contact TAC" method is all too often a serious pain in the ass.)
I’ve never had issues getting them to open a case for this hardware. You can either operate responsibly or not. I wouldn’t be surprised if the situation gets worse. Either way, upgrade/patch/silo as necessary. - Jared
On 09/07/2015 22:35, Ricky Beam wrote:
"Free" if you have a support contract.
No, free-as-in-beer. You register a guest CCO account, email tac@cisco.com, provide the device serial number (or output of "show hardware") and the bugid + Cisco PSIRT URL reference. Cisco TAC will then provide you with a download link with fixed software, at no cost to you. It's not a pain in the ass - it works fine. Nick
On 09-07-15 23:51, Nick Hilliard wrote:
On 09/07/2015 22:35, Ricky Beam wrote:
"Free" if you have a support contract. No, free-as-in-beer.
You register a guest CCO account, email tac@cisco.com, provide the device serial number (or output of "show hardware") and the bugid + Cisco PSIRT URL reference. Cisco TAC will then provide you with a download link with fixed software, at no cost to you. It's not a pain in the ass - it works fine.
Nick
And while that's the general procedure for almost all Cisco products, there is even an faster way for the ASA: - register a CCO account - in ASDM choose Tools > Check for ASA/ASDM Updates - follow the onscreen instructions Paul.
On Fri, Jul 10, 2015 at 3:31 PM, Paul Hoogsteder <mailings@meanie.nl> wrote:
On 09-07-15 23:51, Nick Hilliard wrote:
On 09/07/2015 22:35, Ricky Beam wrote:
"Free" if you have a support contract.
No, free-as-in-beer.
You register a guest CCO account, email tac@cisco.com, provide the device serial number (or output of "show hardware") and the bugid + Cisco PSIRT URL reference. Cisco TAC will then provide you with a download link with fixed software, at no cost to you. It's not a pain in the ass - it works fine.
Nick
And while that's the general procedure for almost all Cisco products, there is even an faster way for the ASA:
- register a CCO account - in ASDM choose Tools > Check for ASA/ASDM Updates - follow the onscreen instructions
Paul.
Hello Gentlemen, I had a crashing ASA 5585-S40 yesterday and it is still crashing today. Box is up to date, I have similar setups on LAX and on east coast and I only see the problem on west coast on circuits connected to Level3 traffic. I have a couple tickets still open with Cisco staff. They have added some dataplane protection which minimized the instability, but I dont know if it's a coincidence or effective, since it's not that often but 5585-S40 boxes are still crashing. If anyone got any update on what's going on please share. I have replaced one critical box with a Juniper one but I can't do it for all my sites promptly so. So far what I found is that it's related to protocol 132 (sctp?). I have tried to filter 132 but no success. I can't just filter source address since it's legit, and proto 132 filtered traffic stills reaching the box up the point it leads to the problem (if in fact it's sctp related). It looks like I'm back to 90's since it seems like a single packet attack. I can't see volumetric deviations, I can't see unusual patterns, proto 132 starts showing up and nothing goes wrong, suddenly I get the crash, no matter if it's been a couple minutes with some proto 132 traffic or if the traffic just started this second... the only "coincidence" is proto 132 popping up without any further specific pattern. Weird and keeps happening.
The bug that this crash impacts is in ASA was introduced in 9.1(4.3) and fixed in 9.1(5.1) and later. Are you inside the affected version range? If not, it's not the bug being discussed here. If so, you may wish to upgrade. Cheers, Christoph On 10 July 2015 at 12:56, Eddie Tardist <edtardist@gmail.com> wrote:
On Fri, Jul 10, 2015 at 3:31 PM, Paul Hoogsteder <mailings@meanie.nl> wrote:
On 09-07-15 23:51, Nick Hilliard wrote:
On 09/07/2015 22:35, Ricky Beam wrote:
"Free" if you have a support contract.
No, free-as-in-beer.
You register a guest CCO account, email tac@cisco.com, provide the device serial number (or output of "show hardware") and the bugid + Cisco PSIRT URL reference. Cisco TAC will then provide you with a download link with fixed software, at no cost to you. It's not a pain in the ass - it works fine.
Nick
And while that's the general procedure for almost all Cisco products, there is even an faster way for the ASA:
- register a CCO account - in ASDM choose Tools > Check for ASA/ASDM Updates - follow the onscreen instructions
Paul.
Hello Gentlemen,
I had a crashing ASA 5585-S40 yesterday and it is still crashing today. Box is up to date, I have similar setups on LAX and on east coast and I only see the problem on west coast on circuits connected to Level3 traffic. I have a couple tickets still open with Cisco staff. They have added some dataplane protection which minimized the instability, but I dont know if it's a coincidence or effective, since it's not that often but 5585-S40 boxes are still crashing.
If anyone got any update on what's going on please share. I have replaced one critical box with a Juniper one but I can't do it for all my sites promptly so.
So far what I found is that it's related to protocol 132 (sctp?). I have tried to filter 132 but no success. I can't just filter source address since it's legit, and proto 132 filtered traffic stills reaching the box up the point it leads to the problem (if in fact it's sctp related).
It looks like I'm back to 90's since it seems like a single packet attack. I can't see volumetric deviations, I can't see unusual patterns, proto 132 starts showing up and nothing goes wrong, suddenly I get the crash, no matter if it's been a couple minutes with some proto 132 traffic or if the traffic just started this second... the only "coincidence" is proto 132 popping up without any further specific pattern.
Weird and keeps happening.
On Fri, Jul 10, 2015 at 7:09 PM, Christoph Blecker <cblecker@gmail.com> wrote:
The bug that this crash impacts is in ASA was introduced in 9.1(4.3) and fixed in 9.1(5.1) and later. Are you inside the affected version range? If not, it's not the bug being discussed here. If so, you may wish to upgrade.
Which is the bug being discussed here? I am still in the dark. No, I am not in the affected range, the only bug I am aware related to proto 132 is back from 2013 and I don't suspect it's the same bug by reading the advisory (however it's the same problem, crashing system). This is why I am blindly looking for clues. Proto 132 is a correlation I made and assumed but not clear at all. If you are talking about any other bug please clarify or point me for further readings, I am still looking for a reaction. Thanks.
Cheers, Christoph
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 7/10/2015 4:00 PM, Eddie Tardist wrote:
On Fri, Jul 10, 2015 at 7:09 PM, Christoph Blecker <cblecker@gmail.com> wrote:
The bug that this crash impacts is in ASA was introduced in 9.1(4.3) and fixed in 9.1(5.1) and later. Are you inside the affected version range? If not, it's not the bug being discussed here. If so, you may wish to upgrade.
Which is the bug being discussed here? I am still in the dark. No, I am not in the affected range, the only bug I am aware related to proto 132 is back from 2013 and I don't suspect it's the same bug by reading the advisory (however it's the same problem, crashing system). This is why I am blindly looking for clues. Proto 132 is a correlation I made and assumed but not clear at all. If you are talking about any other bug please clarify or point me for further readings, I am still looking for a reaction.
Eddi: http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cis co-sa-20141008-asa - - ferg - -- Paul Ferguson PGP Public Key ID: 0x54DC85B2 Key fingerprint: 19EC 2945 FEE8 D6C8 58A1 CE53 2896 AC75 54DC 85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iF4EAREIAAYFAlWgX90ACgkQKJasdVTchbJwKwEAyG1UjDE1cGB/jrLnzQmiNNvO O/AHM2/D1rXrm8SVoAUBAKl14Wdyz+VgByPhEE+YCvuoWOdX8+7wUk/DfohEBb1k =0DDw -----END PGP SIGNATURE-----
On 8 Jul 2015, at 23:58, Mark Mayfield wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
See this preso: <https://app.box.com/s/a3oqqlgwe15j8svojvzl> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Thank you sir. I read your presentation quite some time ago, probably one of the first times you posted to the list. It has definitely informed many of my design processes; particularly with regard to server publishing, and been a major part of my supporting documentation in arguments with others at my organization over the last few years. Of course, these particular ASA implementations are for law enforcement applications, so we are mandated to implement in ways that auditors from the state and federal agencies approve of. However, this makes me consider the need to more aggressively ACL inbound traffic at the router level before these particular firewalls, which I can do, and may help mitigate such events, so thank you for the reminder! Mark Mayfield City of Roseville - AS 54371 Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Roland Dobbins Sent: Wednesday, July 08, 2015 12:18 To: nanog@nanog.org Subject: Re: Possible Sudden Uptick in ASA DOS? On 8 Jul 2015, at 23:58, Mark Mayfield wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
See this preso: <https://app.box.com/s/a3oqqlgwe15j8svojvzl> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On 9 Jul 2015, at 0:43, Mark Mayfield wrote:
However, this makes me consider the need to more aggressively ACL inbound traffic at the router level before these particular firewalls, which I can do, and may help mitigate such events,
Spot-on - reduce the state-surface as much as possible.
so thank you for the reminder!
Sorry for the repeat, but glad the preso was helpful! ;> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
This is pretty scary when you take into account that the NYSE is still down. Kirill Klimakhin Principal Consultant 120 Seventh Street Suite 202 Garden City, NY 11530 (C) 631-707-3303 (F) 631-982-0174 Kirill.Klimakhin@corebts.com www.corebts.com -----Original Message----- From: NANOG [mailto:nanog-bounces+kirill.klimakhin=corebts.com@nanog.org] On Behalf Of Mark Mayfield Sent: Wednesday, July 08, 2015 12:58 PM To: nanog@nanog.org Subject: Possible Sudden Uptick in ASA DOS? Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary. Another pair running the same code had the primary crash and fail in the same time window. So, three crashes in 4 hours in our environment. Open a TAC case on one of these for post-mortem analysis, and they interpreted the crash dump to point at a DOS bug first published in Oct. The very interesting thing; on the phone the TAC engineer said this was "the 10th one of these I've dealt with this morning". Here's the bug they reference: https://tools.cisco.com/bugsearch/bug/CSCul36176/?reffering_site=dumpcr Anyone else have observations to add on this? Mark Mayfield City of Roseville - AS 54371 Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office ________________________________ Important Notice: This email message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Core BTS. Core BTS specifically disclaims liability for any damage caused by any virus transmitted by this email.
We call all relax. The Commander-in-Chief of the USA has declared this to be a technical glitch, and not a security breach or attack. -- Todd Williams Network Engineer Tactical Network Operations Rackspace Hosting On Wed, Jul 08, 2015 at 05:45:55PM +0000, Klimakhin, Kirill wrote:
This is pretty scary when you take into account that the NYSE is still down.
Kirill Klimakhin Principal Consultant 120 Seventh Street Suite 202 Garden City, NY 11530 (C) 631-707-3303 (F) 631-982-0174 Kirill.Klimakhin@corebts.com www.corebts.com
-----Original Message----- From: NANOG [mailto:nanog-bounces+kirill.klimakhin=corebts.com@nanog.org] On Behalf Of Mark Mayfield Sent: Wednesday, July 08, 2015 12:58 PM To: nanog@nanog.org Subject: Possible Sudden Uptick in ASA DOS?
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Another pair running the same code had the primary crash and fail in the same time window.
So, three crashes in 4 hours in our environment.
Open a TAC case on one of these for post-mortem analysis, and they interpreted the crash dump to point at a DOS bug first published in Oct.
The very interesting thing; on the phone the TAC engineer said this was "the 10th one of these I've dealt with this morning".
Here's the bug they reference: https://tools.cisco.com/bugsearch/bug/CSCul36176/?reffering_site=dumpcr
Anyone else have observations to add on this?
Mark Mayfield City of Roseville - AS 54371 Network Systems Engineer
2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office
________________________________ Important Notice: This email message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Core BTS. Core BTS specifically disclaims liability for any damage caused by any virus transmitted by this email.
NANOG members: Hi there. This is Dario Ciccarone from the Cisco PSIRT - the Product Security Incident Response Team. This is to acknowledge we're aware of this issue, and we're working with all the appropriate parties. Indeed, it seems the culprit is Cisco bug ID CSCul36176 - which was released as part of the Cisco Security Advisory "Multiple Vulnerabilities in Cisco ASA Software ", which was published on October 8th, 2014. The full advisory is available at the following URL: http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-s... As I said, the Cisco PSIRT is working with the Cisco Technical Assistance Center on this matter, and we're analyzing the available information. The advisory will be updated to reflect the fact we're seeing active exploitation of this issue. NANOG members are welcome to contact us at psirt@cisco.com if they have any additional questions or concerns, or any information relevant to this issue. Thanks, Dario On 7/8/15 12:58 PM, Mark Mayfield wrote:
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary.
Another pair running the same code had the primary crash and fail in the same time window.
So, three crashes in 4 hours in our environment.
Open a TAC case on one of these for post-mortem analysis, and they interpreted the crash dump to point at a DOS bug first published in Oct.
The very interesting thing; on the phone the TAC engineer said this was "the 10th one of these I've dealt with this morning".
Here's the bug they reference: https://tools.cisco.com/bugsearch/bug/CSCul36176/?reffering_site=dumpcr
Anyone else have observations to add on this?
Mark Mayfield City of Roseville - AS 54371 Network Systems Engineer
2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office
participants (19)
-
Christoph Blecker -
Christopher Morrow -
Chuck Church -
Colin Johnston -
Dario Ciccarone -
Eddie Tardist -
Hugo Slabbert -
Jared Mauch -
Jared Mauch
-
Klimakhin, Kirill -
Mark Andrews -
Mark Mayfield -
Michel Luczak -
Nick Hilliard -
Paul Ferguson -
Paul Hoogsteder -
Ricky Beam -
Roland Dobbins -
Todd Williams