Hi all, I'm Bingyang Liu, a ph.d student in Tsinghua University. My thesis topic is on "source address validation". Although BCP38 was proposed more than ten years ago, IP spoofing still remains an attack vector [MIT-Spoofer] [ARBOR-Annual-Report] [Presentation on NANOG Meeting] [Discussion in NANOG ML]. I did a lot investigation, but still have no idea why so many ISPs haven't deploy BCP38. I enumerate three reasons I found, and I'd like your comments very much. 1. Stub ASes: They rely on their providers to filter, so they won't deploy BCP38 on their own. 2. Low tier transit ASes: They are most likely to deploy BCP38 on the interfaces towards their customers. 3. Large or tier1 ASes: Their peers and customers are also large. So uRPF may have false positive and ACLs are too large to manage. I also asked some ISP guys in IETF today, they all agreed that IP spoofing is an issue, but they may haven't deployed it. One key issue, I think, is about incentive. i.e. you can filter, but you'll still receive spoofing from providers and peers who haven't enforced BCP38. best Bingyang -- Bingyang Liu Network Architecture Lab, Network Center,Tsinghua Univ. Beijing, China Home Page: http://netarchlab.tsinghua.edu.cn/~liuby
On Mar 28, 2012, at 10:44 , Bingyang LIU wrote:
I'm Bingyang Liu, a ph.d student in Tsinghua University. My thesis topic is on "source address validation".
Although BCP38 was proposed more than ten years ago, IP spoofing still remains an attack vector [MIT-Spoofer] [ARBOR-Annual-Report] [Presentation on NANOG Meeting] [Discussion in NANOG ML].
I did a lot investigation, but still have no idea why so many ISPs haven't deploy BCP38. I enumerate three reasons I found, and I'd like your comments very much.
1. Stub ASes: They rely on their providers to filter, so they won't deploy BCP38 on their own. 2. Low tier transit ASes: They are most likely to deploy BCP38 on the interfaces towards their customers. 3. Large or tier1 ASes: Their peers and customers are also large. So uRPF may have false positive and ACLs are too large to manage.
I also asked some ISP guys in IETF today, they all agreed that IP spoofing is an issue, but they may haven't deployed it. One key issue, I think, is about incentive. i.e. you can filter, but you'll still receive spoofing from providers and peers who haven't enforced BCP38.
While those reasons are somewhat valid, they are not the main reasons. #1) Money. Whenever someone asks "why...?", the answer is usually "money". It costs money - CapEx if your equipment doesn't support RPF, and OpEx even if it does. Plus opportunity cost if your customers don't like it or you screw up, as those customers will find someone who doesn't filter and move. #2) Laziness. When the question is "why have [you|they] not...?", the second most common answer is laziness. Some call it "inertia", but reality is people are busy, lazy, etc. Please note the complete lack of smilies or other indication I am kidding or being sarcastic. There is also ignorance, stupidity, malice (yes, some people actually attack others or sell to those who do), etc. -- TTFN, patrick
In a message written on Wed, Mar 28, 2012 at 11:00:39AM -0400, Patrick W. Gilmore wrote:
#1) Money. Whenever someone asks "why...?", the answer is usually "money". It costs money - CapEx if your equipment doesn't support RPF, and OpEx even if it does. Plus opportunity cost if your customers don't like it or you screw up, as those customers will find someone who doesn't filter and move.
#2) Laziness. When the question is "why have [you|they] not...?", the second most common answer is laziness. Some call it "inertia", but reality is people are busy, lazy, etc.
While Patrick is spot on, there is a third issue which is related to money and laziness, but also has some unique aspects. BCP38 makes the assumption that the ISP does some "configuration" to insure only properly sourced packets enter the network. That may have been true when BCP38 was written, but no longer accurately reflects how networks are built and operated. To get source address validation widely deployed it needs to be baked into consumer CPE. The requirement needs to be a "default on" in the DOCSYS specs, for instance. Residential gateways need to come from the factory with unicast RPF turned on. BCP38 needs to be applied at the OEM level in equipment maufacturing, not at the operational level with ISP's. There are, simply, too many variations in CPE devices to expect ISP's to _configure_ them. Even when the configuration is "standardized" (like DOCSYS) ISP's have to think really hard about the operational impact of turning on a feature; and one buggy implementationc can scuttle an idea network wide. Which really comes back to Patrick's point #2. If the people who care about this want to see a positive change they need to stop badgering ISP's to implement BCP38 and start badgering Linksys/Netgear/D-Link/Motorola/Apple/Touchstone/SMC/Westtel to make unicast RPF a default part of their gateway implementation. More importantly, they need to get them to brand it as a _feature_, protect your computer from being used by hackers, our router insures they won't use up all of your data cap! Then it will be something they can sell, and thus something they will implement. As long as folks keep beating on (consumer) ISPs to implement BCP38, nothing will happen. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Leo, On Mar 28, 2012, at 8:13 AM, Leo Bicknell wrote:
#1) Money. #2) Laziness.
While Patrick is spot on, there is a third issue which is related to money and laziness, but also has some unique aspects.
BCP38 makes the assumption that the ISP does some "configuration" to insure only properly sourced packets enter the network. That may have been true when BCP38 was written, but no longer accurately reflects how networks are built and operated.
An interesting assertion. I haven't looked at how end-user networks are built recently. I had assumed there continue to be customer aggregation points within ISP infrastructure in which BCP38-type filtering could occur. You're saying this is no longer the case? What has replaced it?
BCP38 needs
to be applied at the OEM level in equipment maufacturing, not at the operational level with ISP's.
I don't believe this is either/or. I agree that BCP38 features should be turned on by default in CPE, however I believe it really needs to be enforced at the ISP level.
As long as folks keep beating on (consumer) ISPs to implement BCP38, nothing will happen.
Optimist. Actually, given the uptick in spoofing-based DoS attacks, the ease in which such attacks can be generated, recent high profile targets of said attacks, and the full-on money pumping freakout about anything with "cyber-" tacked on the front, I suspect a likely outcome will be proposals for legislation forcing ISPs to do something like BCP38. Regards, -drc
Hi David, Leo, Patrick and all, Considering the reasons you raised, do you think the following two things can happen? 1. Give BCP38 the only practical anti-spoofing technique, can an ISP well protect its customers by implementing BCP38? I don't think so, because I think BCP38 is accurate near the source but inaccurate near the destination, i.e. if its customer is the target of spoofing attack, its capability to filter is relatively low. 2. Even if ineffective near the destination, is an ISP willing to deploy it if it becomes easy to adopt and risk-free (no false positive)? Sorry for my stupid and naive questions. best Bingyang On Wed, Mar 28, 2012 at 5:45 PM, David Conrad <drc@virtualized.org> wrote:
Leo,
On Mar 28, 2012, at 8:13 AM, Leo Bicknell wrote:
#1) Money. #2) Laziness.
While Patrick is spot on, there is a third issue which is related to money and laziness, but also has some unique aspects.
BCP38 makes the assumption that the ISP does some "configuration" to insure only properly sourced packets enter the network. That may have been true when BCP38 was written, but no longer accurately reflects how networks are built and operated.
An interesting assertion. I haven't looked at how end-user networks are built recently. I had assumed there continue to be customer aggregation points within ISP infrastructure in which BCP38-type filtering could occur. You're saying this is no longer the case? What has replaced it?
BCP38 needs
to be applied at the OEM level in equipment maufacturing, not at the operational level with ISP's.
I don't believe this is either/or. I agree that BCP38 features should be turned on by default in CPE, however I believe it really needs to be enforced at the ISP level.
As long as folks keep beating on (consumer) ISPs to implement BCP38, nothing will happen.
Optimist.
Actually, given the uptick in spoofing-based DoS attacks, the ease in which such attacks can be generated, recent high profile targets of said attacks, and the full-on money pumping freakout about anything with "cyber-" tacked on the front, I suspect a likely outcome will be proposals for legislation forcing ISPs to do something like BCP38.
Regards, -drc
-- Bingyang Liu Network Architecture Lab, Network Center,Tsinghua Univ. Beijing, China Home Page: http://netarchlab.tsinghua.edu.cn/~liuby
While I'm a big fan of RFP, it does require that operators be "good citizens" for it to be effective. Like most of the Internet, it's built on a "web" of trust. On Wed, Mar 28, 2012 at 12:10 PM, Bingyang LIU <bjornliu@gmail.com> wrote:
Hi David, Leo, Patrick and all,
Considering the reasons you raised, do you think the following two things can happen?
1. Give BCP38 the only practical anti-spoofing technique, can an ISP well protect its customers by implementing BCP38? I don't think so, because I think BCP38 is accurate near the source but inaccurate near the destination, i.e. if its customer is the target of spoofing attack, its capability to filter is relatively low.
2. Even if ineffective near the destination, is an ISP willing to deploy it if it becomes easy to adopt and risk-free (no false positive)?
Sorry for my stupid and naive questions.
best Bingyang
On Wed, Mar 28, 2012 at 5:45 PM, David Conrad <drc@virtualized.org> wrote:
Leo,
On Mar 28, 2012, at 8:13 AM, Leo Bicknell wrote:
#1) Money. #2) Laziness.
While Patrick is spot on, there is a third issue which is related to money and laziness, but also has some unique aspects.
BCP38 makes the assumption that the ISP does some "configuration" to insure only properly sourced packets enter the network. That may have been true when BCP38 was written, but no longer accurately reflects how networks are built and operated.
An interesting assertion. I haven't looked at how end-user networks are built recently. I had assumed there continue to be customer aggregation points within ISP infrastructure in which BCP38-type filtering could occur. You're saying this is no longer the case? What has replaced it?
BCP38 needs
to be applied at the OEM level in equipment maufacturing, not at the operational level with ISP's.
I don't believe this is either/or. I agree that BCP38 features should be turned on by default in CPE, however I believe it really needs to be enforced at the ISP level.
As long as folks keep beating on (consumer) ISPs to implement BCP38, nothing will happen.
Optimist.
Actually, given the uptick in spoofing-based DoS attacks, the ease in which such attacks can be generated, recent high profile targets of said attacks, and the full-on money pumping freakout about anything with "cyber-" tacked on the front, I suspect a likely outcome will be proposals for legislation forcing ISPs to do something like BCP38.
Regards, -drc
-- Bingyang Liu Network Architecture Lab, Network Center,Tsinghua Univ. Beijing, China Home Page: http://netarchlab.tsinghua.edu.cn/~liuby
-- Ray Soucy Epic Communications Specialist Phone: +1 (207) 561-3526 Networkmaine, a Unit of the University of Maine System http://www.networkmaine.net/
1. Give BCP38 the only practical anti-spoofing technique, can an ISP well protect its customers by implementing BCP38? I don't think so, because I think BCP38 is accurate near the source but inaccurate near the destination, i.e. if its customer is the target of spoofing attack, its capability to filter is relatively low.
Nobody seems to have corrected this point. BCP38 is not intended to protect an ISP's customers. We're used to thinking in terms of protecting ourselves; you put locks on your front door or firewalls in front of your server. That's protecting yourself. If an ISP provides firewalling for their customers, then they are using it to protect the ISP's customers. BCP38 is intended to protect the *rest* of the Internet from *you* - or, more precisely, a bad guy who has taken over your connection. If your ISP implements BCP38, they are protecting everyone *else* from spoofed packets from your connection. It provides no protection for you, though. What provides protection for you is when *other* ISP's implement BCP38. If every other ISP except yours implemented BCP38, you'd be very well protected indeed. The problem here is that BCP38 assumes that service providers will work in the best interests of the Internet in general, implementing a filter that provides no measurable RoI for the SP. It's something that reduces everyone *else's* problems. It's good to implement on that basis, but most networks don't. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
In a message written on Wed, Mar 28, 2012 at 08:45:12AM -0700, David Conrad wrote:
An interesting assertion. I haven't looked at how end-user networks are built recently. I had assumed there continue to be customer aggregation points within ISP infrastructure in which BCP38-type filtering could occur. You're saying this is no longer the case? What has replaced it?
Well, RFC3704 for one has updated the methods and tactics since BCP38 was written. Remember BCP38 was before even "unicast RPF" as we know it existed. Some relevant points from 3704: 3. Clarifying the Applicability of Ingress Filtering What may not be readily apparent is that ingress filtering is not applied only at the "last-mile" interface between the ISP and the end user. It's perfectly fine, and recommended, to also perform ingress filtering at the edges of ISPs where appropriate, at the routers connecting LANs to an enterprise network, etc. -- this increases the defense in depth. 5. Security Considerations [snip] The closer to the actual source ingress filtering is performed, the more effective it is. One could wish that the first hop router would ensure that traffic being sourced from its neighboring end system was correctly addressed; a router further away can only ensure that it is possible that there is such a system within the indicated prefix. Therefore, ingress filtering should be done at multiple levels, with different level of granularity I'm not saying ISP's can't or couldn't do it, what I am saying, and RFC 3704 is repeating, is that it is cheaper/easier/faster and more reliable to do it as close to the edge as possible. "The edge" is not the edge of the ISP network, it is the edge of the entire network, that is the /last router in the topology/. Today that last router is owned and operated by the customer in most cases. So if a provider drops off a modem with your service that also does WiFi and the customer simply uses it, the provider is 100% responsible for doing BCP 38, in my estimation. But as soon as the consumer buys a routing device they become 100% on the hook for now operating the last mile, and it is that device where the primary filtering should take place. ISP's may still filter, for a defense in depth, but they are no longer the edge of the network and as such their responsibility is greatly diminished in my view. BCP38 was written when a point to point handoff to a single customer was standard, and that's easy to filter. Today a shared medium (like a cable modem network) is common and more importantly connects to more routers (home gateways), rathern than PC's. That's a funamental change since BCP38 was written. I'll also point out that operating systems fill a role here as well. Many OS's won't let you spoof a layer 2 MAC address (try to write a packet with a raw interface and it overwrites the source address) but are happy to let an application send a packet with source layer 3 address that is forged! Sure, malware could always hack the OS too, but it raises the bar. The community should demand that all OS's default to not allowing L3 sources that aren't configured on the box from leaving that box. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Wed, Mar 28, 2012 at 12:16, Leo Bicknell <bicknell@ufp.org> wrote:
Well, RFC3704 for one has updated the methods and tactics since BCP38 was written. Remember BCP38 was before even "unicast RPF" as we know it existed.
I think the concern of RFC3704/BCP84, i.e., multihoming, is the primary reason we don't see ingress filtering as much as we should. Almost any network worth its salt these days is multihomed, making strict RPF nearly impossible to pull off. Despite this, to my knowledge, Juniper is one of the only vendors that provides feasible-path RPF to deal with it. On Cisco and Brocade for example, you're stuck doing some dark voodoo magic with BGP weights & communities + strict RPF (refer to the previous money and laziness points) to accomplish something that SHOULD be basic. -- Darius Jahandarie
On Mar 28, 2012, at 9:39 AM, Darius Jahandarie wrote:
I think the concern of RFC3704/BCP84, i.e., multihoming, is the primary reason we don't see ingress filtering as much as we should.
I would be surprised if this were true. I'd argue that today, the vast majority of devices on the Internet (and certainly the ones that are used in massive D(D)oS attacks) are found hanging off singly-homed networks. Regards, -drc
On Wed, Mar 28, 2012 at 12:50, David Conrad <drc@virtualized.org> wrote:
I would be surprised if this were true.
I'd argue that today, the vast majority of devices on the Internet (and certainly the ones that are used in massive D(D)oS attacks) are found hanging off singly-homed networks.
Yes, but RPF can be implemented in places other than the customer edge. In those places, lack of widespread, easy, and vendor-supported feasible-path uRPF is what I believe really hurts things. Granted, this is along a different line than what the OP was talking about, but in terms of answering the question of "why don't we see ingress filtering as much as we should?", I think it's a large factor. -- Darius Jahandarie
Hi Darius, Yes, I agree that feasible RPF solves the problem in a lot of scenarios. However, in some other cases, the asymmetric routing is caused by static routing, traffic engineering, policy routing, etc., where the lengths of forward path and reverse path may differ, so feasible RPF may also fail (false positive). Bingyang On Wed, Mar 28, 2012 at 7:07 PM, Darius Jahandarie <djahandarie@gmail.com> wrote:
On Wed, Mar 28, 2012 at 12:50, David Conrad <drc@virtualized.org> wrote:
I would be surprised if this were true.
I'd argue that today, the vast majority of devices on the Internet (and certainly the ones that are used in massive D(D)oS attacks) are found hanging off singly-homed networks.
Yes, but RPF can be implemented in places other than the customer edge. In those places, lack of widespread, easy, and vendor-supported feasible-path uRPF is what I believe really hurts things.
Granted, this is along a different line than what the OP was talking about, but in terms of answering the question of "why don't we see ingress filtering as much as we should?", I think it's a large factor.
-- Darius Jahandarie
-- Bingyang Liu Network Architecture Lab, Network Center,Tsinghua Univ. Beijing, China Home Page: http://netarchlab.tsinghua.edu.cn/~liuby
Also, Don't forget that transit providers currently bill their customers to carry that spoofed/DoS traffic, why would they filter it when it's $$$$ on their balance sheets? -Drew -----Original Message----- From: Bingyang LIU [mailto:bjornliu@gmail.com] Sent: Wednesday, March 28, 2012 1:15 PM To: Darius Jahandarie Cc: NANOG list Subject: Re: BCP38 Deployment Hi Darius, Yes, I agree that feasible RPF solves the problem in a lot of scenarios. However, in some other cases, the asymmetric routing is caused by static routing, traffic engineering, policy routing, etc., where the lengths of forward path and reverse path may differ, so feasible RPF may also fail (false positive). Bingyang On Wed, Mar 28, 2012 at 7:07 PM, Darius Jahandarie <djahandarie@gmail.com> wrote:
On Wed, Mar 28, 2012 at 12:50, David Conrad <drc@virtualized.org> wrote:
I would be surprised if this were true.
I'd argue that today, the vast majority of devices on the Internet (and certainly the ones that are used in massive D(D)oS attacks) are found hanging off singly-homed networks.
Yes, but RPF can be implemented in places other than the customer edge. In those places, lack of widespread, easy, and vendor-supported feasible-path uRPF is what I believe really hurts things.
Granted, this is along a different line than what the OP was talking about, but in terms of answering the question of "why don't we see ingress filtering as much as we should?", I think it's a large factor.
-- Darius Jahandarie
-- Bingyang Liu Network Architecture Lab, Network Center,Tsinghua Univ. Beijing, China Home Page: http://netarchlab.tsinghua.edu.cn/~liuby
On 03/28/2012 09:16 AM, Leo Bicknell wrote:
In a message written on Wed, Mar 28, 2012 at 08:45:12AM -0700, David Conrad wrote:
An interesting assertion. I haven't looked at how end-user networks are built recently. I had assumed there continue to be customer aggregation points within ISP infrastructure in which BCP38-type filtering could occur. You're saying this is no longer the case? What has replaced it? Well, RFC3704 for one has updated the methods and tactics since BCP38 was written. Remember BCP38 was before even "unicast RPF" as we know it existed.
I'm not saying ISP's can't or couldn't do it, what I am saying, and RFC 3704 is repeating, is that it is cheaper/easier/faster and more reliable to do it as close to the edge as possible. "The edge" is not the edge of the ISP network, it is the edge of the entire network, that is the /last router in the topology/. Today that last router is owned and operated by the customer in most cases.
Yeahbut, the CPE isn't trusted. It would be _nice_ for customers to be bcp38 clueful as well, but I don't think it's _required_ for successful deployment from the ISP's standpoint. Even with a system like DOCSIS where the CPE is semi-trustworthy from a provisioning/etc standpoint, I don't think I'd _count_ on them. In any case, isn't RPF really cheap these days on edge aggregation routers? It's not like it's a new innovation or anything.
BCP38 was written when a point to point handoff to a single customer was standard, and that's easy to filter. Today a shared medium (like a cable modem network) is common and more importantly connects to more routers (home gateways), rathern than PC's. That's a funamental change since BCP38 was written.
DOCSIS was standardized in the mid to late 90's which more or less predates bcp 38, and it has always been able to handle multiple endpoints/modem. As I recall, there were specs for cable modem nics for individual machines, but they never took off. Mike
In a message written on Wed, Mar 28, 2012 at 09:52:49AM -0700, Michael Thomas wrote:
Yeahbut, the CPE isn't trusted. It would be _nice_ for customers to be bcp38 clueful as well, but I don't think it's _required_ for successful deployment from the ISP's standpoint. Even with a system like DOCSIS where the CPE is semi-trustworthy from a provisioning/etc standpoint, I don't think I'd _count_ on them.
None of the routers are "trusted" if your perspective is right. It's easy to find a path like: "Tier 1 ISP" - Regional ISP - Local Provider - Subscriber - User Techologically it may look like: Tier 1 T640 core network with 10GE handoff Regional Cisco GSR network with 1GE handoff Local 1006 to Arris CMTS Subscriber Motorola Cable Modem to NetGear SOHO Gateway User Patron with Airport Express sharing a wired connection to WiFi I don't trust any of the people in that list. More interesting from a BCP38 perspective who should be doing the filtering? If you were going to write it into law/regulation, where would you require it? Maybe all of them should, but can they from a technologial perspective? There's multi-homing in that chain somewhere. Do you require it at the first single homed place? If the subscriber is using a NetGear that does both ethernet and cell card backup and is thus multi-homed does that mean the user must do it? It's not even in my list, but re-asking my previous question why don't we go a step further and require the Operating System to do unicast RPF on-box? I think given the thorny set of issues that taking a step back and saying, "rather than a perfect solution, what gets us most of the way there the cheapest, and quick" is a good question to ask. I'm going to point to the local boxes. In my example the Netgear and Airport devices are in a posion to do super-cheap unicast RPF. They have (generally) one network behind them, and one way out. They are CPU based boxes for which this check requires no hardware changes. They don't even have enough interfaces in most cases to multi-home, so the chance of it breaking is nil. And yes, while the user may control both the end PC and these devices and thus be able to turn it off and circumvent all of this, that's really not the problem. The problem is infected machines spewing crap their owners don't know about, and just having a separate device upstream that stops it will do the job. The perfect is the enemy of the good in this case. Solving this at the consumer CPE level would remove 90-95% of the problem at zero hardware cost, a very small software cost, and a very small support cost and probably make us stop talking about this issue all together. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On 03/28/2012 12:03 PM, Leo Bicknell wrote:
None of the routers are "trusted" if your perspective is right.
It's easy to find a path like:
"Tier 1 ISP" - Regional ISP - Local Provider - Subscriber - User
Techologically it may look like:
Tier 1 T640 core network with 10GE handoff Regional Cisco GSR network with 1GE handoff Local 1006 to Arris CMTS Subscriber Motorola Cable Modem to NetGear SOHO Gateway User Patron with Airport Express sharing a wired connection to WiFi
I don't trust any of the people in that list. More interesting from a BCP38 perspective who should be doing the filtering? If you were going to write it into law/regulation, where would you require it?
I wasn't talking about laws/regulation, just responding to your comment that lack of RPF in CPE was the bigger problem which doesn't seem right to me. If I'm the owner of the CMTS network, I really shouldn't be trusting the CM to be doing filtering for me. Maybe it would be nice to have RPF checks in the CM to nip a spoofed DDOS before it ever gets on the HFC network, but I wouldn't count on the CM not being compromised too, which means I should probably be running RPF on the aggregation router as well.
Maybe all of them should, but can they from a technologial perspective? There's multi-homing in that chain somewhere. Do you require it at the first single homed place? If the subscriber is using a NetGear that does both ethernet and cell card backup and is thus multi-homed does that mean the user must do it? It's not even in my list, but re-asking my previous question why don't we go a step further and require the Operating System to do unicast RPF on-box?
It's been a long time since I've read bcp 38, but I thought its intent was if you can reasonably do it, you *should* do it. multipath obviously makes RPF problematic, but the vast majority of the edge networks aren't multi-homed, so they really should be running RPF just as a matter of... best common practice.
I think given the thorny set of issues that taking a step back and saying, "rather than a perfect solution, what gets us most of the way there the cheapest, and quick" is a good question to ask. I'm going to point to the local boxes. In my example the Netgear and Airport devices are in a posion to do super-cheap unicast RPF. They have (generally) one network behind them, and one way out. They are CPU based boxes for which this check requires no hardware changes. They don't even have enough interfaces in most cases to multi-home, so the chance of it breaking is nil. And yes, while the user may control both the end PC and these devices and thus be able to turn it off and circumvent all of this, that's really not the problem. The problem is infected machines spewing crap their owners don't know about, and just having a separate device upstream that stops it will do the job.
The perfect is the enemy of the good in this case. Solving this at the consumer CPE level would remove 90-95% of the problem at zero hardware cost, a very small software cost, and a very small support cost and probably make us stop talking about this issue all together.
Except for the small problem that getting cheap home router box manufacturers to do just about anything is a pushing on string exercise. So if I want to a) protect my network and b) be a good netizen, I'm still going to want to do BCP 38 regardless of whether others violate a, b or both. Right? Mike
In a message written on Wed, Mar 28, 2012 at 12:44:04PM -0700, Michael Thomas wrote:
Except for the small problem that getting cheap home router box manufacturers to do just about anything is a pushing on string exercise. So if I want to a) protect my network and b) be a good netizen, I'm still going to want to do BCP 38 regardless of whether others violate a, b or both. Right?
BCP38 has nothing to do with a), doing it on your own network doesn't really protect you from much of anything of note. It's all about b), being a good citizen, and having a leg to stand on when you try to convince others to do the same which will help protect you. But the home router vendors aren't as hard to make move as you think. True, the chance of them moving in response to the fact that BCP38 exists, or that NANOG wants them to is zero. Nada, zilch. However, there are some powerful companies that buy a lot of boxes from these vendors. That free-to-the-subscriber box with a Comcast, Verizon, Cox, Cable Vision, AT&T, SBC, or other provider label on it is just a rebranded version of one of these devices. If the guy buying several million dollars worth of the boxes showed up and demanded this feature, it would be done. Once it's done for them, it's a free "feature" they can market in the boxes at best-buy to try and recover more of their development costs. So in that sense we need to pressure the ISP's to implement BCP38! Maybe I'm back to agreeing with the OP! However we need to pressure them not to turn on RPF on their routers (although that's a fine thing too, defense in depth and all, if they can they should), but to pressure the vendors they are buying from to do it. The standards bodies should also be pressured as well, to get it into the specifications. I think some engineers need to ask some interesting questions, like how, in a box doing NAT to an outside IP, does it ever emit a packet not from that outside IP? The fact that you can spoof packets through some of the NAT implementations out there is mind-blowing to me. I'm telling you, if the big 10 ISP's would just add one bullet point to their RFP's for equipment: * Any device performing an IP routing function must default to strict mode unicast RPF for all connected networks as specified in RFC 3704 Section 2.2 as a method of implementing BCP38. We'd be done with this issue and move on to other things. Sure, there would still be spoofed packets, and yes, other types of operators (like free public wifi and such) still need to do the right BCP38 filtering when configuring their systems...but just having this on all residential gear gets rid of well over 90% of the crud we're all trying to stop. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Wed, 28 Mar 2012 13:36:49 -0700, Leo Bicknell said:
I think some engineers need to ask some interesting questions, like how, in a box doing NAT to an outside IP, does it ever emit a packet not from that outside IP? The fact that you can spoof packets through some of the NAT implementations out there is mind-blowing to me.
The mind-blowing part for me: Look at the MIT spoofing website, at what percent of the net's address space is spoofable. Then consider what percent of the net is behind a NAT (either consumer grade, or enterprise NAT). http://spoofer.csail.mit.edu/summary.php They're reporting that 20% or so (eyeballing) is unable to spoof due to a NAT. From that, and a guess of what % is *really* behind a NAT, we can make an estimate of how common this failure mode is.
On Mar 28, 2012, at 12:03 PM, Leo Bicknell wrote:
Tier 1 T640 core network with 10GE handoff Regional Cisco GSR network with 1GE handoff Local 1006 to Arris CMTS Subscriber Motorola Cable Modem to NetGear SOHO Gateway User Patron with Airport Express sharing a wired connection to WiFi ... If you were going to write it into law/regulation, where would you require it?
Seems to me that from a legislator's perspective, there is a pretty bright (as in "moth attracted to flame") line between "subscriber" and "provider".
Maybe all of them should, but can they from a technologial perspective?
Implementing telephone number portability was probably technologically more challenging for the telcos to deal with but that didn't stop the legislators from requiring it.
I think given the thorny set of issues that taking a step back and saying, "rather than a perfect solution, what gets us most of the way there the cheapest, and quick" is a good question to ask.
You don't think that question has already been asked? It has been a dozen years since BCP38 was published. Over that period, the Internet has grown immensely and with it, the threats the ability to trivially spoofing source addresses represents. As far as I can tell, there has been little to no improvement in mechanisms to reduce those threats, yet high profile attacks against governments, departments/ministries, commercial organizations, etc., have only increased. I figure at some point (likely after a particularly high-profile attack), politicians and their corporate masters are going to feel the need to be seen to "do something about the problem." I have some skepticism that 'something' is going to be an ideal solution.
The perfect is the enemy of the good in this case. Solving this at the consumer CPE level would remove 90-95% of the problem at zero hardware cost, a very small software cost, and a very small support cost and probably make us stop talking about this issue all together.
And the incentive for CPE manufacturers to invest in the small software cost is? Regards, -drc
In a message written on Wed, Mar 28, 2012 at 02:49:02PM -0700, David Conrad wrote:
On Mar 28, 2012, at 12:03 PM, Leo Bicknell wrote:
Tier 1 T640 core network with 10GE handoff Regional Cisco GSR network with 1GE handoff Local 1006 to Arris CMTS Subscriber Motorola Cable Modem to NetGear SOHO Gateway User Patron with Airport Express sharing a wired connection to WiFi ... If you were going to write it into law/regulation, where would you require it?
Seems to me that from a legislator's perspective, there is a pretty bright (as in "moth attracted to flame") line between "subscriber" and "provider".
The counterpoint I would offer is their are the most lobbiests and lawyers on the "provider" side of that equation, and indeed in the entire stack best I can tell.
And the incentive for CPE manufacturers to invest in the small software cost is?
The "provders" are large buyers of much of the CPE, and in some cases get to approve what CPE gets attached to their network. They can push this on the CPE manufacturers, and should. I suspect if legislators tried to push the issue their lobbiests and lawyers would attempt to stall and deflect, and that would be the direction. Many places already have laws that running an unsecured WiFi network is the subscriber's problem, not the providers. There's already operational and legal precident that the person running that end router should be responsible. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Wed, 28 Mar 2012, David Conrad wrote:
Actually, given the uptick in spoofing-based DoS attacks, the ease in which such attacks can be generated, recent high profile targets of said attacks, and the full-on money pumping freakout about anything with "cyber-" tacked on the front, I suspect a likely outcome will be proposals for legislation forcing ISPs to do something like BCP38.
Exactly. Either do it voluntarily or it will be done for you involuntarily at the federal level and you will have nobody but yourselves to blame. The choice is yours. -Dan
Yep, one way is to give economic penalty. But how about giving the _good_ ISPs economic reward? Say, some transit ISPs deploy anti-spoofing techniques (e.g. uRPF), but only filter those spoofing packets whose destination is the ASes having purchased their *anti-spoofing service* ? Bingyang On Wed, Mar 28, 2012 at 6:41 PM, <goemon@anime.net> wrote:
On Wed, 28 Mar 2012, David Conrad wrote:
Actually, given the uptick in spoofing-based DoS attacks, the ease in which such attacks can be generated, recent high profile targets of said attacks, and the full-on money pumping freakout about anything with "cyber-" tacked on the front, I suspect a likely outcome will be proposals for legislation forcing ISPs to do something like BCP38.
Exactly.
Either do it voluntarily or it will be done for you involuntarily at the federal level and you will have nobody but yourselves to blame.
The choice is yours.
-Dan
-- Bingyang Liu Network Architecture Lab, Network Center,Tsinghua Univ. Beijing, China Home Page: http://netarchlab.tsinghua.edu.cn/~liuby
On 3/28/12 11:45 AM, David Conrad wrote:
Actually, given the uptick in spoofing-based DoS attacks, the ease in which such attacks can be generated, recent high profile targets of said attacks, and the full-on money pumping freakout about anything with "cyber-" tacked on the front, I suspect a likely outcome will be proposals for legislation forcing ISPs to do something like BCP38.
in a note (which didn't go anywhere in particular) i pointed out that contract may address the same issue for which legislation may be proposed, at least for "contractual closures" (sorry, a term of my own, defined below) which share the property some jurisdictions have of a finite access provider universe. i mean "contractual closure" to be the performance guarantee (or non-performance guarantee) present in a set of contracts for a particular service. think "china", after first abstracting all the negatives associated with policy as a property of a distributed, shared, public resource, or "firewalls 4 (bcp defined) good". -e
Yeah, "contractual closures" might be a way to force the providers to deploy BCP38. However, when the customers become the target of a spoofing attack, the provider may not be able to protect its customers, because ingress filtering (including uRPF) is inefficient when done near the destination. In other words, an ISP can deploy BCP38 or whatever, but still cannot well protect its customers from spoofing attacks from other ASes. On Wed, Mar 28, 2012 at 6:54 PM, Eric Brunner-Williams <brunner@nic-naa.net> wrote:
On 3/28/12 11:45 AM, David Conrad wrote:
Actually, given the uptick in spoofing-based DoS attacks, the ease in which such attacks can be generated, recent high profile targets of said attacks, and the full-on money pumping freakout about anything with "cyber-" tacked on the front, I suspect a likely outcome will be proposals for legislation forcing ISPs to do something like BCP38.
in a note (which didn't go anywhere in particular) i pointed out that contract may address the same issue for which legislation may be proposed, at least for "contractual closures" (sorry, a term of my own, defined below) which share the property some jurisdictions have of a finite access provider universe.
i mean "contractual closure" to be the performance guarantee (or non-performance guarantee) present in a set of contracts for a particular service.
think "china", after first abstracting all the negatives associated with policy as a property of a distributed, shared, public resource, or "firewalls 4 (bcp defined) good".
-e
-- Bingyang Liu Network Architecture Lab, Network Center,Tsinghua Univ. Beijing, China Home Page: http://netarchlab.tsinghua.edu.cn/~liuby
On Wed, 28 Mar 2012, Bingyang LIU wrote:
the provider may not be able to protect its customers, because ingress filtering (including uRPF) is inefficient when done near the destination. In other words, an ISP can deploy BCP38 or whatever, but still cannot well protect its customers from spoofing attacks from other ASes.
The ASes which enable spoofing need to have some penalty imposed or they will never engage in correct behavior. Something like throwing all their traffic into scavenger class. If their customers start complaining to them, maybe then they will shape up. -Dan
On Wed, Mar 28, 2012 at 08:45:12AM -0700, David Conrad wrote:
Leo,
On Mar 28, 2012, at 8:13 AM, Leo Bicknell wrote:
#1) Money. #2) Laziness.
While Patrick is spot on, there is a third issue which is related to money and laziness, but also has some unique aspects.
BCP38 makes the assumption that the ISP does some "configuration" to insure only properly sourced packets enter the network. That may have been true when BCP38 was written, but no longer accurately reflects how networks are built and operated.
An interesting assertion. I haven't looked at how end-user networks are built recently. I had assumed there continue to be customer aggregation points within ISP infrastructure in which BCP38-type filtering could occur. You're saying this is no longer the case? What has replaced it?
uRFP was a trivial, 0-impact feature on the cisco VXR-based CMTS platform. Assert a simple statement in the default config (along with 'ips classless' and all your other standard config elements) and job done. It assisted in reducing our abuse desk workload by eliminating a class of attacks from us, so the trivial "cost" was worth it in opex. ISTR it being on the required feature list for additional CMTS evaluations but it has been many years since I touched that kit. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE / NewNOG
On Thu, 29 Mar 2012, Joe Provo wrote:
uRFP was a trivial, 0-impact feature on the cisco VXR-based CMTS platform. Assert a simple statement in the default config (along with 'ips classless' and all your other standard config elements)
uRPF: or as it's now used in ios, ip verify unicast source reachable-via rx ... I don't know what it would have to do with ip classless. It requires ip cef, but so do lots of other "features" including reasonably fast packet forwarding.
and job done. It assisted in reducing our abuse desk workload by eliminating a class of attacks from us, so the trivial "cost" was worth it in opex. ISTR it being on the required feature list for additional CMTS evaluations but it has been many years since I touched that kit.
uRPF stops your customers from sending forged source address packets. Since forged source address packets are rarely traced back to their actual source, I'm not sure how configuring it on your network would reduce your abuse desk workload at all. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Thu, Mar 29, 2012 at 07:31:26PM -0400, Jon Lewis wrote:
On Thu, 29 Mar 2012, Joe Provo wrote:
uRFP was a trivial, 0-impact feature on the cisco VXR-based CMTS platform. Assert a simple statement in the default config (along with 'ips classless' and all your other standard config elements)
uRPF: or as it's now used in ios, ip verify unicast source reachable-via rx ...
I don't know what it would have to do with ip classless.
Stated to counter 'config is hard' as there junk you have to do regardless. Add it to your standard specs and be done.
uRPF stops your customers from sending forged source address packets. Since forged source address packets are rarely traced back to their actual source, I'm not sure how configuring it on your network would reduce your abuse desk workload at all.
Guess we had better informed neighbors? :-) You caught the rhetoric; the cost was that trivial. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE / NewNOG
We are about to accept a 20MEG Ethernet feed via Comcast and their fiber plant as well as a BGP feed across the same link. I have a space GIGE interface on a 7206VXR and would like to know best practice for deploying for optimal performance across this interface. Any ideas and or direction would be extremely helpful as we are seeing some real issues such as. Direct connect (without BGP) to the CPE from Comcast (Fiber to Ethernet) via a laptop gives the level of performance we would expect, However as soon as we terminate to our router via the GIGE which is set to 100MB full duplex and all flow control turned off (Negotiation auto) per Comcast and connect up via a 100MB fast Ethernet interface directly connected we get a fraction of the speed when direct connected. Ideas? BRW
On 3/29/12 6:36 PM, Brian R. Watters wrote:
We are about to accept a 20MEG Ethernet feed via Comcast and their fiber plant as well as a BGP feed across the same link.
I have a space GIGE interface on a 7206VXR and would like to know best practice for deploying for optimal performance across this interface.
Any ideas and or direction would be extremely helpful as we are seeing some real issues such as.
Direct connect (without BGP) to the CPE from Comcast (Fiber to Ethernet) via a laptop gives the level of performance we would expect, However as soon as we terminate to our router via the GIGE which is set to 100MB full duplex and all flow control turned off (Negotiation auto) per Comcast and connect up via a 100MB fast Ethernet interface directly connected we get a fraction of the speed when direct connected.
From my own experience here with our 7200s, some of the PA based 100BaseT interfaces (ie: not on the IO module) can not negotiate 100-full, but rather only half. This leaves one end diff then the other and creates issues with performance. Try forcing both the laptop and router to 100-full and see if it helps. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
On Thu, 29 Mar 2012, Brielle Bruns wrote:
From my own experience here with our 7200s, some of the PA based 100BaseT interfaces (ie: not on the IO module) can not negotiate 100-full, but rather only half. This leaves one end diff then the other and creates issues with performance. Try forcing both the laptop and router to 100-full and see if it helps.
Those interfaces don't to auto-negotiation at all. That's why they default to 100 half. OP said they were using a Gig interface though. Maybe a copper 10/100/1000 port on an NPE-G<1|2>? I haven't used those, but I'd bet they support auto-negotiation. 1000baseT requires it. It'd be helpful to know how they've tested through the router, and if there are other connections routed through that VXR that are working at the expected rates. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On 3/29/12 6:53 PM, Jon Lewis wrote:
On Thu, 29 Mar 2012, Brielle Bruns wrote:
From my own experience here with our 7200s, some of the PA based 100BaseT interfaces (ie: not on the IO module) can not negotiate 100-full, but rather only half. This leaves one end diff then the other and creates issues with performance. Try forcing both the laptop and router to 100-full and see if it helps.
Those interfaces don't to auto-negotiation at all. That's why they default to 100 half. OP said they were using a Gig interface though. Maybe a copper 10/100/1000 port on an NPE-G<1|2>? I haven't used those, but I'd bet they support auto-negotiation. 1000baseT requires it.
I'm pretty sure the PA-FE-TX boards can do auto neg, just not 100 full (just tested with a 7507 with a VIP4 w a PA-FE-TX and a cheap 10BT hub - my 7206VXR is not powered up ATM). Believe it has something to do with the DEC ethernet chip they use (I have an older desktop that just happens to have the same DEC chipset that those do and has exactly the same problem). Based on what he said, I read his setup as having a G1 or G2 NPE, using the on-NPE gig to hook to comcast, and a PA-FE-TX in one of the PA slots. At least, that's how it sounded to me - why else use 100BaseT on a gige as most laptops and desktops in the past... 4-5 years or so have onboard gige? -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
On 3/29/12 7:06 PM, Brielle Bruns wrote:
I'm pretty sure the PA-FE-TX boards can do auto neg, just not 100 full (just tested with a 7507 with a VIP4 w a PA-FE-TX and a cheap 10BT hub - my 7206VXR is not powered up ATM).
Eh, just tried again to show someone and the link didn't even come up this time. I'll toss is up to an oddity or me misreading (likely the latter). My mistake. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
Your correct with your understanding of our setup, I also note on our NPE-G1 that the onboard GIGE interface will auto-negotiation and I do see the flow control is not supported via the other side (Comcast) but as soon as I refresh and view the GIG# interface again I note that flow control is turned back on and "no negotiation auto" is back on the interface cfg ?, this is certainly part of the issue .. is their a way to disable flow control on the onboard GIGE ? .. as stated Comcast does not want flow control on. Yes there are other ports on this router that perform without issue and as designed both other GIGE interfaces that are VLAN'ed and Serial interfaces that are both DS3 and a PA-4T bonded to 6MEG's. the GIGE cfg is as follows interface GigabitEthernet0/1 description Comcast Inet Feed Metro E 20MB bandwidth 100000 ip address 12.12.12.12 255.255.255.252 no ip unreachables no ip route-cache load-interval 30 duplex full speed 100 media-type rj45 no negotiation auto no cdp enable We have 2GB of memory on this router with a very light load on the CPU. On 3/29/12 6:53 PM, Jon Lewis wrote:
On Thu, 29 Mar 2012, Brielle Bruns wrote:
From my own experience here with our 7200s, some of the PA based 100BaseT interfaces (ie: not on the IO module) can not negotiate 100-full, but rather only half. This leaves one end diff then the other and creates issues with performance. Try forcing both the laptop and router to 100-full and see if it helps.
Those interfaces don't to auto-negotiation at all. That's why they default to 100 half. OP said they were using a Gig interface though. Maybe a copper 10/100/1000 port on an NPE-G<1|2>? I haven't used those, but I'd bet they support auto-negotiation. 1000baseT requires it.
I'm pretty sure the PA-FE-TX boards can do auto neg, just not 100 full (just tested with a 7507 with a VIP4 w a PA-FE-TX and a cheap 10BT hub - my 7206VXR is not powered up ATM). Believe it has something to do with the DEC ethernet chip they use (I have an older desktop that just happens to have the same DEC chipset that those do and has exactly the same problem). Based on what he said, I read his setup as having a G1 or G2 NPE, using the on-NPE gig to hook to comcast, and a PA-FE-TX in one of the PA slots. At least, that's how it sounded to me - why else use 100BaseT on a gige as most laptops and desktops in the past... 4-5 years or so have onboard gige? -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
On 3/29/12 7:32 PM, Brian R. Watters wrote:
Your correct with your understanding of our setup, I also note on our NPE-G1 that the onboard GIGE interface will auto-negotiation and I do see the flow control is not supported via the other side (Comcast) but as soon as I refresh and view the GIG# interface again I note that flow control is turned back on and "no negotiation auto" is back on the interface cfg ?, this is certainly part of the issue .. is their a way to disable flow control on the onboard GIGE ? .. as stated Comcast does not want flow control on.
Yes there are other ports on this router that perform without issue and as designed both other GIGE interfaces that are VLAN'ed and Serial interfaces that are both DS3 and a PA-4T bonded to 6MEG's.
How do you have the PA modules installed? Layout can make a huge difference on those given the bandwidth points system. http://www.cisco.com/en/US/docs/routers/7200/configuration/7200_port_adapter... -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
The GIGe is on-board with the NPE-G1 and from what I am told no bandwidth points to deal with .. the PA is in slot 4 with ZERO other traffic on that slot or the port, all other traffic that is of any real size is on the other two GIGE interfaces that are also on-board with the NPE-G1 blade. ----- Original Message ----- From: "Brielle Bruns" <bruns@2mbit.com> To: "NANOG list" <nanog@nanog.org> Sent: Thursday, March 29, 2012 6:42:11 PM Subject: Re: Comcast Ethernet Feed On 3/29/12 7:32 PM, Brian R. Watters wrote:
Your correct with your understanding of our setup, I also note on our NPE-G1 that the onboard GIGE interface will auto-negotiation and I do see the flow control is not supported via the other side (Comcast) but as soon as I refresh and view the GIG# interface again I note that flow control is turned back on and "no negotiation auto" is back on the interface cfg ?, this is certainly part of the issue .. is their a way to disable flow control on the onboard GIGE ? .. as stated Comcast does not want flow control on.
Yes there are other ports on this router that perform without issue and as designed both other GIGE interfaces that are VLAN'ed and Serial interfaces that are both DS3 and a PA-4T bonded to 6MEG's.
How do you have the PA modules installed? Layout can make a huge difference on those given the bandwidth points system. http://www.cisco.com/en/US/docs/routers/7200/configuration/7200_port_adapter... -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org -- Brian R. Watters Director 5718 East Shields Ave ■ Fresno, CA 93727 tel - (559) - 420-0205 ■ fax - (559) - 272-5266 Line website | My LinkedIn | email TwitterFacebookLinkedIn
On 30/03/2012, at 12:32 PM, Brian R. Watters wrote:
interface GigabitEthernet0/1 description Comcast Inet Feed Metro E 20MB bandwidth 100000 ip address 12.12.12.12 255.255.255.252 no ip unreachables no ip route-cache load-interval 30 duplex full speed 100 media-type rj45 no negotiation auto no cdp enable
Remove 'no ip route-cache'. This will be forcing all traffic via the slowest path possible.
This. We have a Comcast MetroE PtP connection between our office and our datacenter and I found that it was defaulting to half duplex. I forced both of my ends to 100 Full and it fixed my issue. Derek On Mar 29, 2012, at 8:47 PM, Brielle Bruns wrote:
On 3/29/12 6:36 PM, Brian R. Watters wrote:
We are about to accept a 20MEG Ethernet feed via Comcast and their fiber plant as well as a BGP feed across the same link.
I have a space GIGE interface on a 7206VXR and would like to know best practice for deploying for optimal performance across this interface.
Any ideas and or direction would be extremely helpful as we are seeing some real issues such as.
Direct connect (without BGP) to the CPE from Comcast (Fiber to Ethernet) via a laptop gives the level of performance we would expect, However as soon as we terminate to our router via the GIGE which is set to 100MB full duplex and all flow control turned off (Negotiation auto) per Comcast and connect up via a 100MB fast Ethernet interface directly connected we get a fraction of the speed when direct connected.
From my own experience here with our 7200s, some of the PA based 100BaseT interfaces (ie: not on the IO module) can not negotiate 100-full, but rather only half. This leaves one end diff then the other and creates issues with performance. Try forcing both the laptop and router to 100-full and see if it helps.
-- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
--- On Thu, 3/29/12, Brian R. Watters <brwatters@absfoc.com> wrote:
From: Brian R. Watters <brwatters@absfoc.com> Subject: Comcast Ethernet Feed To: "NANOG list" <nanog@nanog.org> Date: Thursday, March 29, 2012, 5:36 PM We are about to accept a 20MEG Ethernet feed via Comcast and their fiber plant as well as a BGP feed across the same link.
I have a space GIGE interface on a 7206VXR and would like to know best practice for deploying for optimal performance across this interface.
Any ideas and or direction would be extremely helpful as we are seeing some real issues such as.
Direct connect (without BGP) to the CPE from Comcast (Fiber to Ethernet) via a laptop gives the level of performance we would expect, However as soon as we terminate to our router via the GIGE which is set to 100MB full duplex and all flow control turned off (Negotiation auto) per Comcast and connect up via a 100MB fast Ethernet interface directly connected we get a fraction of the speed when direct connected.
Ideas?
BRW
A couple of questions - 1) What flavor of NPE are you using? 2) Is the GigE interface on the NPE-G1/G2 OR is this a PA? 3) Is the FaE ethernet interface that you appear to be connecting your laptop to, on a separate PA in chassis? 4) Have you verified you that "bandwidth-points" have not been exceeded for bus-1 and/or 2: slots 1,3,5 for bus1 and 2,4,6; also 0(if I/O controller is present. It is 600 points for bus1 and 600 for bus2. (A sh ver will provice the info) You can google: "Cisco 7200 Series Port Adapter Hardware Configuration Guidelines" for additional info. Finally, Have you *hard-coded* speed and duplex on any of you eth ints? Please don't! Let both ints auto-negotiate speed&duplex. after having done so, post the output of: sh int gi x/y and sh int fa x/y (hardcoding speed/duplex is sometimes required when dealing with brain-dead CPE. I have also seen other flavors of brain-dead CPE that *only* work when speed/duplex are set to auto) ./Randy
A couple of questions - 1) What flavor of NPE are you using? NPE-G1 2) Is the GigE interface on the NPE-G1/G2 OR is this a PA? 3) Is the FaE ethernet interface that you appear to be connecting your laptop to, on a separate PA in chassis? Laptop connected directly to router via slot 4 PA-FE-TX 4) Have you verified you that "bandwidth-points" have not been exceeded for bus-1 and/or 2: slots 1,3,5 for bus1 and 2,4,6; also 0(if I/O controller is present. It is 600 points for bus1 and 600 for bus2. PCI bus mb1 has 390 bandwidth points PCI bus mb2 has 500 bandwidth points Have you *hard-coded* speed and duplex on any of you eth ints? Please don't! GIGE has been both hard and auto .. same results .. Fast Ether has always been set @ auto Let both ints auto-negotiate speed&duplex. Comcast states that we are required to have a hard code FULL DUPLEX and SPEED 100 as well as flow control OFF however I can not appear to be able to disable it :( after having done so, post the output of: sh int gi x/y and sh int fa x/y (hardcoding speed/duplex is sometimes required when dealing with brain-dead CPE. I have also seen other flavors of brain-dead CPE that *only* work when speed/duplex are set to auto) ./Randy
On Thu, Mar 29, 2012 at 8:02 PM, Brian R. Watters <brwatters@absfoc.com> wrote:
A couple of questions -
1) What flavor of NPE are you using?
NPE-G1
2) Is the GigE interface on the NPE-G1/G2 OR is this a PA? 3) Is the FaE ethernet interface that you appear to be connecting your laptop to, on a separate PA in chassis?
Laptop connected directly to router via slot 4 PA-FE-TX
4) Have you verified you that "bandwidth-points" have not been exceeded for bus-1 and/or 2: slots 1,3,5 for bus1 and 2,4,6; also 0(if I/O controller is present. It is 600 points for bus1 and 600 for bus2.
PCI bus mb1 has 390 bandwidth points PCI bus mb2 has 500 bandwidth points
Have you *hard-coded* speed and duplex on any of you eth ints? Please don't!
GIGE has been both hard and auto .. same results .. Fast Ether has always been set @ auto
Let both ints auto-negotiate speed&duplex.
Comcast states that we are required to have a hard code FULL DUPLEX and SPEED 100 as well as flow control OFF however I can not appear to be able to disable it :(
If the Comcast side is hard-coded to 100/Full then you really only have one choice, set your side to 100/Full, as well. For the past decade, Cisco gear completely disables autonegotiation if you hard set the speed and duplex settings. Some equipment still participates in auto even when you hard set it. That's why you occasionally get duplex mismatches even when both sides are hard set. The side that participates in auto will expect to see an autonegotiating link partner. When it doesn't see one, it drops back to half duplex because it assumes it is connected to a hub (This is for Fast Ethernet.) So, if you connect a piece of Cisco gear and it is hard set to 100/full, you'll be fine. If you connect a laptop or some other device with a NIC that still participates in auto even when you hard set the settings, you won't get that to work well.
Never mind control and what Comcast says about hard-coding speed and duplex! The question is: What happens when you set the int facing Comcast CPE to auto? Does the link even come up? *IF* the link comes up, can you ping your next-hop? If you can, leave auto-neg on despite what what Comcast may say/require. Post a "sh int gix/y" and "sh int fax/y" If the above outputs are *clean*, I would say a TAC case is called for. --- On Thu, 3/29/12, Brian R. Watters <brwatters@absfoc.com> wrote:
From: Brian R. Watters <brwatters@absfoc.com> Subject: Re: Comcast Ethernet Feed To: "Randy" <randy_94108@yahoo.com> Cc: "NANOG list" <nanog@nanog.org> Date: Thursday, March 29, 2012, 7:02 PM
A couple of questions -
1) What flavor of NPE are you using?
NPE-G1
2) Is the GigE interface on the NPE-G1/G2 OR is this a PA? 3) Is the FaE ethernet interface that you appear to be connecting your laptop to, on a separate PA in chassis?
Laptop connected directly to router via slot 4 PA-FE-TX
4) Have you verified you that "bandwidth-points" have not been exceeded for bus-1 and/or 2: slots 1,3,5 for bus1 and 2,4,6; also 0(if I/O controller is present. It is 600 points for bus1 and 600 for bus2.
PCI bus mb1 has 390 bandwidth points PCI bus mb2 has 500 bandwidth points
Have you *hard-coded* speed and duplex on any of you eth ints? Please don't!
GIGE has been both hard and auto .. same results .. Fast Ether has always been set @ auto
Let both ints auto-negotiate speed&duplex.
Comcast states that we are required to have a hard code FULL DUPLEX and SPEED 100 as well as flow control OFF however I can not appear to be able to disable it :(
after having done so, post the output of:
sh int gi x/y and sh int fa x/y
(hardcoding speed/duplex is sometimes required when dealing with brain-dead CPE. I have also seen other flavors of brain-dead CPE that *only* work when speed/duplex are set to auto)
./Randy
On Thursday, March 29, 2012 7:03 PM, Brian R. Watters <mailto:brwatters@absfoc.com> wrote: [snip]
Fast Ether has always been set @ auto
Just in case you missed it, I would echo Brielle's earlier advice: please try forcing both laptop and the FE it's plugged into to 100/Full, auto disabled, and try your tests again. I feel like this thread has developed an unhealthy fixation with the GE <-> Comcast segment when it's just as likely that it's working perfectly fine and the problem is between Laptop <-> FE. :-) For whatever reason, I have historically had very bad luck/experience with 7200 FE interfaces and auto-negotiation, FWIW. -- Nathan Anderson First Step Internet, LLC nathana@fsr.com
On Thu, Mar 29, 2012 at 10:07 PM, Nathan Anderson <nathana@fsr.com> wrote:
On Thursday, March 29, 2012 7:03 PM, Brian R. Watters <mailto:brwatters@absfoc.com> wrote:
[snip]
Fast Ether has always been set @ auto
Just in case you missed it, I would echo Brielle's earlier advice: please try forcing both laptop and the FE it's plugged into to 100/Full, auto disabled, and try your tests again. I feel like this thread has developed an unhealthy fixation with the GE <-> Comcast segment when it's just as likely that it's working perfectly fine and the problem is between Laptop <-> FE. :-) For whatever reason, I have historically had very bad luck/experience with 7200 FE interfaces and auto-negotiation, FWIW.
Ditto! Plug laptop_2 into one of the FE ports and see if you get good speeds between the laptops. On the 7200, show interface for each of the FE ports, compare to what the laptops think it has. Our 7200's have FE ports that have to be hard coded to get full duplex. (We no longer have that issue since almost all 7200's have only GigE ports, we only have one 7204 with a FE port and it's the only one that's hard coded to duplex full). ...Todd -- Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. -- Martin Golding
What does Comcast do if you exceed 20 meg? We were told by AT&T that anything over the specified limit would be dropped, so we use rate limiting...not sure if Comcast does the same. Cody ----- Original Message ----- From: "Brian R. Watters" <brwatters@absfoc.com> To: "NANOG list" <nanog@nanog.org> Sent: Thursday, March 29, 2012 5:36:43 PM Subject: Comcast Ethernet Feed We are about to accept a 20MEG Ethernet feed via Comcast and their fiber plant as well as a BGP feed across the same link. I have a space GIGE interface on a 7206VXR and would like to know best practice for deploying for optimal performance across this interface. Any ideas and or direction would be extremely helpful as we are seeing some real issues such as. Direct connect (without BGP) to the CPE from Comcast (Fiber to Ethernet) via a laptop gives the level of performance we would expect, However as soon as we terminate to our router via the GIGE which is set to 100MB full duplex and all flow control turned off (Negotiation auto) per Comcast and connect up via a 100MB fast Ethernet interface directly connected we get a fraction of the speed when direct connected. Ideas? BRW Notice to Recipient: Information contained in this message may be privileged, confidential and protected from disclosure. If you are not an intended recipient, it is strictly prohibited to use, disseminate or copy this communication. If you have received this in error, please reply to the sender and then delete the message.Thank you.
The power of defaults. The few successful Internet security "best practice" changes have primarily resulted from changes to default settings, not trying to get ISPs, operators, sysadmins or users to change. Smurf attacks - change default directed-broadcast settings in dominant router vendors Open SMTP relays - changed default SMTP server settings in dominant SMTP software sources/vendors Windows network-level worms - changed default Windows XP/SP2 firewall settings to closed inbound Although it may take 10+ years for a product replacement cycle (Windows XP is taking a longer), the same laziness/money/ignorance reasons why its nearly impossible to get people to implement "best practices" is why a change to the default settings is so effective. The few times the new default doesn't work, the operator then has an incentive to change it. The times the default doesn't impact the operator, there is no incentive to change it. Expecting an average person (ISP, sysadmin, programmer, etc) to discover and understand many obscure configuration options which don't directly impact what they want to do isn't realistic. People tend to not pro-actively look for problems until it causes them a problem. Even worse, systems tend to revert back to defaults when a mistake or change to unrelated parts of the system are made without the user/operator realizing it. The "experts" are the people who created the open source software or vendors creating the product, not the users/customers. SSH is a rare example where operators pro-actively sought and changed their behaivor; but even then, there were probably more operators that went with the default.
participants (24)
-
Bingyang LIU -
Brian R. Watters -
Brielle Bruns -
Cody Grosskopf -
Darius Jahandarie -
David Conrad -
Derek Ivey -
Drew Weaver -
Eric Brunner-Williams -
goemon＠anime.net -
Ian Henderson -
Joe Greco -
Joe Provo -
John Neiberger -
Jon Lewis -
Leo Bicknell -
Michael Thomas -
Nathan Anderson -
Patrick W. Gilmore -
Randy -
Ray Soucy -
Sean Donelan -
Todd Lyons -
Valdis.Kletnieks＠vt.edu