Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch? I'm trying here to gauge the length of time before this vulnerability is closed out. irwin
I don't see a lot of choice with this exploit. It's easy and quick. I knocked down a 7206 in less than a minute. Expedite where you can. BobG -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Irwin Lazar Sent: Friday, July 18, 2003 2:30 PM To: nanog@merit.edu Subject: Patching for Cisco vulnerability Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch? I'm trying here to gauge the length of time before this vulnerability is closed out. irwin
On Fri, Jul 18, 2003 at 12:29:30PM -0600, Irwin Lazar wrote:
Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch?
I'm trying here to gauge the length of time before this vulnerability is closed out.
most providers can easily go from (for example) 12.0(21)S3 to 12.0(21)S7 with less testing than from 12.0(21)S to 12.0(25)S The hurdles are still there to maintain the necessary customer notifications, etc.. but aside from that, I think the press is doing their job (good or bad) in that most customers are aware that there's something bad going on and people are moving to protect the internet infrastructure. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Fri, Jul 18, 2003 at 03:04:45PM -0400, Jared Mauch wrote:
most providers can easily go from (for example) 12.0(21)S3 to 12.0(21)S7 with less testing than from 12.0(21)S to 12.0(25)S
12.0(21)S* (at least S5 and above) have broken SNMP interface counters and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't want to lose money (accounting) are forced to upgrade to 12.0(25)S*. I guess they want to force all "conservative" ISPs to jump over the 12.0(22)S "barrier". Some things won't be forgotten (see also recent discussion about the new non-"Cisco"-GBIC blocking). Voting with pockets takes place. Regards, Daniel
On Fri, Jul 18, 2003 at 09:21:28PM +0200, Daniel Roesen wrote:
On Fri, Jul 18, 2003 at 03:04:45PM -0400, Jared Mauch wrote:
most providers can easily go from (for example) 12.0(21)S3 to 12.0(21)S7 with less testing than from 12.0(21)S to 12.0(25)S
12.0(21)S* (at least S5 and above) have broken SNMP interface counters and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't
Do you have a DDTS I can reference?
want to lose money (accounting) are forced to upgrade to 12.0(25)S*. I guess they want to force all "conservative" ISPs to jump over the 12.0(22)S "barrier".
I agree that Cisco should actually take more serious ownership of these issues within a customers network. They're selling us these software/hw and claiming that we can obtain a particular SLA level. Yet they can't seem to add in some code that says if (ifc->in_bps > ifc->phy_speed || ifc->out_bps > ifc->phy_speed) { crash_router(); } If they added this code, they'd find these bugs in their labs instead of in our networks. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Fri, Jul 18, 2003 at 03:31:25PM -0400, Jared Mauch wrote:
12.0(21)S* (at least S5 and above) have broken SNMP interface counters and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't
Do you have a DDTS I can reference?
Not handy, but from cisco-nsp Archives I've found CSCea35259 and CSCdy30984, and a reference to CSCea63754 which I can't take a look at in BugToolkit. Symptom: SNMP output octet counter stops counting traffic (except some control plane traffic it seems), with every few days jumping by weird amounts producing such funny things like 150mbps spikes on a FE interface. I've seen a box with a nicely loaded FE (30-70mbps) which took (reproducably) just about 48 hours to have this interface stop counting. If this would have been a customer interface, it would have meant "reload router every two nights or lose money". This bug is supposed to be (finally) fixed in 12.0(25)S1. Given that you a) don't want to lose money and b) don't want to do two whole-network upgrades within a short time, going to 12.0(21)S7 to fix the vulnerabilty is no real option, so people are more or less forced to put their networks on bigger risk by going from 12.0(21)S* to (25)S1. Regards, Daniel
--On Friday, July 18, 2003 21:57:57 +0200 Daniel Roesen <dr@cluenet.de> wrote:
On Fri, Jul 18, 2003 at 03:31:25PM -0400, Jared Mauch wrote:
12.0(21)S* (at least S5 and above) have broken SNMP interface counters and Cisco refuses to fix the bug in 12.0(21)S*, so people who don't
Do you have a DDTS I can reference?
Not handy, but from cisco-nsp Archives I've found CSCea35259 and CSCdy30984, and a reference to CSCea63754 which I can't take a look at in BugToolkit.
Symptom: SNMP output octet counter stops counting traffic (except some control plane traffic it seems), with every few days jumping by weird amounts producing such funny things like 150mbps spikes on a FE interface.
I've seen a box with a nicely loaded FE (30-70mbps) which took (reproducably) just about 48 hours to have this interface stop counting. If this would have been a customer interface, it would have meant "reload router every two nights or lose money".
This bug is supposed to be (finally) fixed in 12.0(25)S1.
Given that you a) don't want to lose money and b) don't want to do two whole-network upgrades within a short time, going to 12.0(21)S7 to fix the vulnerabilty is no real option, so people are more or less forced to put their networks on bigger risk by going from 12.0(21)S* to (25)S1.
I'm running 12.0(25.2)S, and it has the bug REALLY squashed. LER -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: ler@lerctr.org US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
if (ifc->in_bps > ifc->phy_speed || ifc->out_bps > ifc->phy_speed) { crash_router(); }
If they added this code, they'd find these bugs in their labs instead of in our networks.
I remember seeing an article claiming that Cisco´s automated regression testing does "more than 250000" tests before they release a piece of code. However, questions about the nature of these tests and if any tests sent more traffic than a random scripted ping went unanswered. Pete
On Fri, 18 Jul 2003 12:29:30 MDT, Irwin Lazar <ILazar@burtongroup.com> said:
I'm trying here to gauge the length of time before this vulnerability is closed out.
The core routers have been bouncing as they upgrade all this week. A lot of places will be putting the fixes in place during windows this weekend. And some 30% of the routers won't ever get upgraded. We're going to be finding vulnerable routers on eBay for years. See Eric Rescorla's paper on how fast the OpenSSL vulnerabilities of a while ago were actually fixed: To: BugTraq Subject: Security holes... Who cares? Date: Nov 15 2002 6:30PM Author: Eric Rescorla <ekr rtfm com> Message-ID: <200211151830.gAFIUr751517@sierra.rtfm.com> I'd like to announce the availability for downlaod of the following paper. Security holes... Who cares? Eric Rescorla RTFM, Inc. <http://www.rtfm.com/> We report on an observational study of user response following the OpenSSL remote buffer overflows of July 2002 and the worm that exploited it in September 2002. Immediately after the publication of the bug and its subsequent fix we identified a set of vulnerable servers. In the weeks that followed we regularly probed each server to determine whether it had applied one of the relevant fixes. We report two primary results. First, we find that administrators are generally very slow to apply the fixes. Two weeks after the bug announcement, more than two thirds of servers were still vulnerable. Second, we identify several weak predictors of user response and find that the pattern differs in the period following the release of the bug and that following the release of the worm. The paper can be downloaded from: http://www.rtfm.com/upgrade.pdf http://www.rtfm.com/upgrade.ps
On Fri, 2003-07-18 at 14:29, Irwin Lazar wrote:
Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch?
I think a lot of providers are doing the minimum testing required (upgrade, does it boot? can I ping?) and then rolling it out... I guess it's more of an implicit trust type deal rather than having routers running with an increased load because of ACL's and still having the possibility of a vulnerable router because something was overlooked.. Going forward, if you go the ACL route, every time you add a new interface you need to be sure to either apply the "any any" acl to it, or add the new ip's to the "big" block by ip acl ...
I'm trying here to gauge the length of time before this vulnerability is closed out.
I don't think it will ever truly go away.. there are lots of "older" routers that won't be able to support the newer code, albeit small routers like the 2500's, but they'll exist..
irwin
-- --------------------------- Jason H. Frisvold Backbone Engineering Supervisor Penteledata Engineering friz@corp.ptd.net RedHat Engineer - RHCE # 807302349405893 Cisco Certified - CCNA # CSCO10151622 MySQL Core Certified - ID# 205982910 --------------------------- "Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world." -- Albert Einstein [1879-1955]
I don't think it will ever truly go away.. there are lots of "older" routers that won't be able to support the newer code, albeit small routers like the 2500's, but they'll exist..
Yes I have some old routers (2500s) for which no code exists which is patched and small enough to sit in the flash/memory... acl acl ! Steve
On Fri, 2003-07-18 at 15:37, Stephen J. Wilcox wrote:
I don't think it will ever truly go away.. there are lots of "older" routers that won't be able to support the newer code, albeit small routers like the 2500's, but they'll exist..
Yes I have some old routers (2500s) for which no code exists which is patched and small enough to sit in the flash/memory... acl acl !
And given the low processing power of those routers, acl's hurt ... :(
Steve --
Jason H. Frisvold Backbone Engineering Supervisor Penteledata Engineering friz@corp.ptd.net RedHat Engineer - RHCE # 807302349405893 Cisco Certified - CCNA # CSCO10151622 MySQL Core Certified - ID# 205982910 --------------------------- "Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world." -- Albert Einstein [1879-1955]
This has me wondering if there are any BCPs that touch on the whole idea of filtering traffic destined to your router, or what the advisory called "infrastructure filtering". All in all, it seems like a good idea to block any direct access to router interfaces. But as some have probably found already, it's a big pain in the arse. If I recall correctly, Rob's Secure IOS Template touches on filtering known services (the BGP listener, snmp), but what are people's feelings on maintaining filters on all interfaces *after* loading a fixed IOS? Thanks, Charles -- Charles Sprickman spork@inch.com On Fri, 18 Jul 2003, Irwin Lazar wrote:
Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch?
I'm trying here to gauge the length of time before this vulnerability is closed out.
irwin
Some high-end boxes already have thing called "receive filter" which helps this a lot. Hope we see more of that or better yet router vendors stop processing packets they shouldn´t be processing anyway much earlier in the code path. "Be liberal what you accept" should not apply here. Pete ----- Original Message ----- From: "Charles Sprickman" <spork@inch.com> To: <nanog@merit.edu> Sent: Friday, July 18, 2003 11:20 PM Subject: Infrastructure Filtering (was Re: Patching for Cisco vulnerability)
This has me wondering if there are any BCPs that touch on the whole idea of filtering traffic destined to your router, or what the advisory called "infrastructure filtering". All in all, it seems like a good idea to block any direct access to router interfaces. But as some have probably found already, it's a big pain in the arse.
If I recall correctly, Rob's Secure IOS Template touches on filtering known services (the BGP listener, snmp), but what are people's feelings on maintaining filters on all interfaces *after* loading a fixed IOS?
Thanks,
Charles
-- Charles Sprickman spork@inch.com
On Fri, 18 Jul 2003, Irwin Lazar wrote:
Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to
ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch?
I'm trying here to gauge the length of time before this vulnerability is closed out.
irwin
Hi Charles, * spork@inch.com (Charles Sprickman) [Fri 18 Jul 2003, 22:21 CEST]:
If I recall correctly, Rob's Secure IOS Template touches on filtering known services (the BGP listener, snmp), but what are people's feelings on maintaining filters on all interfaces *after* loading a fixed IOS?
You'll have to weigh the benefits to the downsides. Benefits to filtering IP packets with those four protocols: - You're protecting your network, and possibly others, from failure - You can see pretty quickly when someone's trying to attack you or other networks Downsides: - You're hampering the use of several technologies - Possible impact on the load / forwarding capacity of your router (dependent on its architecture) Personally I'd try to filter packets destined for known router interfaces and let the rest pass through. And of course not run known-buggy software (famous last words...). -- Niels. -- <anselm> rather than calling it bluetooth the protocol should be called 'erikson wireless cellphone earpiece protocol' since that seems to be its only real use.
On Fri, 18 Jul 2003, Niels Bakker wrote:
Personally I'd try to filter packets destined for known router interfaces and let the rest pass through. And of course not run known-buggy software (famous last words...).
That would mean Windows (pick your flavor.) :-) Sorry, I couldn't resist. Curtis -- Curtis Maurand mailto:curtis@maurand.com http://www.maurand.com
On Fri, Jul 18, 2003 at 04:20:37PM -0400, Charles Sprickman wrote:
This has me wondering if there are any BCPs that touch on the whole idea of filtering traffic destined to your router, or what the advisory called "infrastructure filtering". All in all, it seems like a good idea to block any direct access to router interfaces. But as some have probably found already, it's a big pain in the arse.
If I recall correctly, Rob's Secure IOS Template touches on filtering known services (the BGP listener, snmp), but what are people's feelings on maintaining filters on all interfaces *after* loading a fixed IOS?
It shouldn't be done. transit internet providers should not be the edges firewalls. The edge? They can filter what they want, but you should not filter things for people that they don't know is being filtered. I can see a few clear cases where this is acceptable, and ms-sql was one of them. Something that might be of interest and of possible value for something like this in the future would be this: draft-marques-idr-flow-spec-00.txt Take a look at it. It would allow access-list/firewall-filters to be able to be deployed across your network in the matter of a few minutes instead of having to login to every router. Plus you could count the packets too via snmp. (well, assuming that works right ;-) ) - Jared
On Fri, 18 Jul 2003, Irwin Lazar wrote:
Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch?
I'm trying here to gauge the length of time before this vulnerability is closed out.
irwin
-- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
* jared@puck.Nether.net (Jared Mauch) [Fri 18 Jul 2003, 23:23 CEST]:
On Fri, Jul 18, 2003 at 04:20:37PM -0400, Charles Sprickman wrote:
If I recall correctly, Rob's Secure IOS Template touches on filtering known services (the BGP listener, snmp), but what are people's feelings on maintaining filters on all interfaces *after* loading a fixed IOS? It shouldn't be done. transit internet providers should not be the edges firewalls. The edge? They can filter what they want, but you should not filter things for people that they don't know is being filtered. I can see a few clear cases where this is acceptable, and ms-sql was one of them.
Good point. Still, transit networks' ingress routers could filter on destination addresses of nodes known not to run IP protocols 53/55/77/103 in order to protect them. I suppose most networks have a limited number of ranges they use for assigning space to loopback and point-to-point interfaces so this needn't be an extreme amount of administration. Regards, -- Niels.
On Fri, 18 Jul 2003, Niels Bakker wrote:
* jared@puck.Nether.net (Jared Mauch) [Fri 18 Jul 2003, 23:23 CEST]:
On Fri, Jul 18, 2003 at 04:20:37PM -0400, Charles Sprickman wrote:
If I recall correctly, Rob's Secure IOS Template touches on filtering known services (the BGP listener, snmp), but what are people's feelings on maintaining filters on all interfaces *after* loading a fixed IOS? It shouldn't be done. transit internet providers should not be the edges firewalls. The edge? They can filter what they want, but you should not filter things for people that they don't know is being filtered. I can see a few clear cases where this is acceptable, and ms-sql was one of them.
Good point. Still, transit networks' ingress routers could filter on destination addresses of nodes known not to run IP protocols 53/55/77/103 in order to protect them.
hrm, what nodes don't run 55/53/77/103? What do? Do you have a list? Could we have it? Seriously though... the edge networks (as Jared pointed out) should be able to decide what they want to filter and what they don't... perhaps some large ISP would decide you don't want any traffic from 212/8 or perhaps all porn? Or all religious material? You don't want someone deciding what you do and don't get... unless that someone is you :)
I suppose most networks have a limited number of ranges they use for assigning space to loopback and point-to-point interfaces so this needn't be an extreme amount of administration.
yes... inside my network I know what my loopbacks and links are, inside yours?? No idea... or Jared's or Tim Battles or...
* chris@UU.NET (Christopher L. Morrow) [Sat 19 Jul 2003, 01:03 CEST]:
hrm, what nodes don't run 55/53/77/103? What do? Do you have a list? Could we have it?
I'm sure you know what devices in your network run Mobile IP or Sun ND (to paraphrase Randy Bush, you can probably count them on the fingers of your nose). Router#conf t Router(config)#ip receive-acl 10 no-idiocy
Seriously though... the edge networks (as Jared pointed out) should be able to decide what they want to filter and what they don't... perhaps some large ISP would decide you don't want any traffic from 212/8 or perhaps all porn? Or all religious material? You don't want someone deciding what you do and don't get... unless that someone is you :)
That's why I said that transit networks could filter only towards their own infrastructure.
yes... inside my network I know what my loopbacks and links are, inside yours?? No idea... or Jared's or Tim Battles or...
Luckily it's not your responsibility to protect them (only to intervene when advised they're under attack, which I've heard you're doing a very good job at - but that aside). Regards, -- Niels. -- "The time of getting fame for your name on its own is over. Artwork that is only about wanting to be famous will never make you famous. Any fame is a bi-product of making something that means something. You don't go to a restaurant and order a meal because you want to have a shit." -- Banksy
On Sat, 19 Jul 2003, Niels Bakker wrote:
* chris@UU.NET (Christopher L. Morrow) [Sat 19 Jul 2003, 01:03 CEST]:
hrm, what nodes don't run 55/53/77/103? What do? Do you have a list? Could we have it?
I'm sure you know what devices in your network run Mobile IP or Sun ND (to paraphrase Randy Bush, you can probably count them on the fingers of your nose).
my nose has many fingers... wait, thats hairs! :) though I do agree... So, I must apologize for reading your message's intent in reverse.
Router#conf t Router(config)#ip receive-acl 10 no-idiocy
Seriously though... the edge networks (as Jared pointed out) should be able to decide what they want to filter and what they don't... perhaps some large ISP would decide you don't want any traffic from 212/8 or perhaps all porn? Or all religious material? You don't want someone deciding what you do and don't get... unless that someone is you :)
That's why I said that transit networks could filter only towards their own infrastructure.
Agreed, and it does, to some extent... As should anyone elses, eh? It makes sense that if you have either of the 2 main vendor's products you can accomplish this task easily and at 'no cost'
yes... inside my network I know what my loopbacks and links are, inside yours?? No idea... or Jared's or Tim Battles or...
Luckily it's not your responsibility to protect them (only to intervene when advised they're under attack, which I've heard you're doing a very good job at - but that aside).
We thank you, its a group effort... but as I said above, my apologies, this current event has me a bit punchy :)
As mentioned before, Receive Path ACL (rACL) is already in 12.0(21)S2 (and forward) for the GSR and 12.0(24)S for the 7500. This is one way of doing infrastructure filtering without packet filtering the data plane (customer traffic). The second phase of Receive Path ACL (rACL) is going everywhere. The marketing name is Control Plane Protocol (CPP) ... but it also takes care of any packet punted to the receive path (i.e. packet with destination address = to the router). It is MQC based (ACL + rate-limiting). Think of it as a "TCP wrapper" for the receive path - but with the rate-limiting. The rate limiting part is important. It will first show up in 12.2S (and forward) and then cross-port/back-port through customer pressure (talk to your Cisco Account Teams). You'll see it on everything for the small boxes (26XX) to switches (CAT6Ks) to the high end (GSRs). Personally, I see this "TCP Wrapper with Rate-Limit" around a router as something that is going to be a requirement for all vendors on the Net.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Charles Sprickman Sent: Friday, July 18, 2003 1:21 PM To: nanog@merit.edu Subject: Infrastructure Filtering (was Re: Patching for Cisco vulnerability)
This has me wondering if there are any BCPs that touch on the whole idea of filtering traffic destined to your router, or what the advisory called "infrastructure filtering". All in all, it seems like a good idea to block any direct access to router interfaces. But as some have probably found already, it's a big pain in the arse.
If I recall correctly, Rob's Secure IOS Template touches on filtering known services (the BGP listener, snmp), but what are people's feelings on maintaining filters on all interfaces *after* loading a fixed IOS?
Thanks,
Charles
-- Charles Sprickman spork@inch.com
On Fri, 18 Jul 2003, Irwin Lazar wrote:
Just out of curiosity, are folks just applying the Cisco patch or do you
go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch?
I'm trying here to gauge the length of time before this vulnerability is
closed out.
irwin
--On Friday, July 18, 2003 12:29:30 -0600 Irwin Lazar <ILazar@burtongroup.com> wrote:
Just out of curiosity, are folks just applying the Cisco patch or do you go through some sort of testing/validation process to ensure that the patch doesn't cause any other problems? Given typical change management procedures how long is taking you to get clearance to apply the patch?
I'm trying here to gauge the length of time before this vulnerability is closed out.
We had a phone conference with our Cisco people thursday lunch MEST and agreed on a testing scheme, where we would upgrade one of the (redundant) core routers and one access router (our network basically has two kinds of Cisco equipment, 12400 as core and 10700 as CPE) and let them run for an hour -- if no problems by then we'd roll the upgrade through the network, trying not to blackhole our customers. We went from 12.0(23)S1 to 12.0(23)S3, and it was mostly painless. -- Måns Nilsson Systems Specialist +46 70 681 7204 KTHNOC MN1334-RIPE We're sysadmins. To us, data is a protocol-overhead.
participants (15)
-
Barry Raveendran Greene -
Bob German -
Charles Sprickman -
Christopher L. Morrow -
Curtis Maurand -
Daniel Roesen -
Irwin Lazar -
Jared Mauch -
Jason Frisvold -
Larry Rosenman -
Måns Nilsson -
Niels Bakker -
Petri Helenius -
Stephen J. Wilcox -
Valdis.Kletnieks＠vt.edu