- NANOG - lists.nanog.org

Re: Time to revise RFC 1771
by Sean Donelan 26 Jun '01

26 Jun '01

On Tue, 26 June 2001, Andrew Partan wrote: > On Tue, Jun 26, 2001 at 11:48:29AM -0700, Sean Donelan wrote: > > RESOLVED, the error handling mechanism should be revised to reject > > only the portion of the announcement in dispute instead of disrupting > > the entire BGP session. > > The big problem here is that it may be impossible to tell that just > this bit of <goo> is bad and that everything received after that > is fine. I'm willing to accept there are cases you can not recover, and the only acceptable alternative is to reboot (gee, Microsoft thinks that is the solution to everything). My objection is RFC 1771 does not give implementations the ability to continue when they can. Aborting the BGP session should be an option, and extreme option, but not the ONLY option. So should throwing away the bad data, alerting, and continuing with good data.

1 0

Re: Time to revise RFC 1771
by Dave Israel 26 Jun '01

26 Jun '01

This ignores three basic facts: 1) Networks tend to be homogenous in platform. 2) Platforms tend to accept their own implementation quirks 3) Networks peer at borders Therefore, under the "drop the session rule," my bad announcement gets to all my borders fine, and all my external peers who are not running forgiving/compatable implementations drop their connections to me and all my traffic to/from them hits the floor. One CRC error does not make PPP drop. Why make one route cause a catastrophic loss of connectivity? Report the bad route, drop it, and move on; let layer 8 resolve it. -Dave On 6/26/2001 at 13:08:46 -0700, Clayton Fiske said: > The basic issue is one of scale vs integrity. However, I think this > particular case is one in which the RFC-dictated behavior is the > correct choice. The problem is that one [set of] router[s] did not > follow such behavior and thus escalated the scale of the problem > significantly. > > Given that the malformed route in question was most likely originated > from a single router, the only damage that should have been done was > a loss of routability for networks behind that one router. While of > course that could be arguably a significant number of networks, I > think it's a safe assumption that X losing its peers is pretty much > always a smaller impact than all of X's peers losing -their- peers. > If network XYZ's routers have N peers each, the RFC-dictated > behavior gives us N peering sessions lost (assuming the offending > route was advertised to all peers), instead of N^2 (or greater) > sessions as was the case. > > I think the logic of dropping the session is sound. If a router > originates one malformed route, who's to say the rest of its routes > are correct? Perhaps other routes are corrupted, but not in ways > detectable by the router's sanity checks. Since the offending route > is indeed malformed, it's not unreasonable to stop trusting the > router from which it originated. Since it's likely[1] only a single > router is originating the route, dropping sessions to that one > router controls the blast radius[2]. > > This is not to say that the issue of scale is unimportant. It most > certainly is. However, again, if the first router(s) to receive > the route had behaved properly, the scale of the problem would > have been small. The only place you'd see a flap of 100,000 > routes is if the offending router was your upstream's. Everyone > else would only see (at most) a flap of the routes originated by > and/or behind that router (in BGP topology terms). > > Perhaps a knob to control the behavior would be an acceptable > compromise for some. I think it's a bad idea for two reasons. > First, it allows bugs such as this to go unfixed, because when > it happens people just adjust the knob to keep their BGP sessions > stable. Second, it circumvents the integrity control. If a router > has many corrupted routes, but only a few trigger the sanity > checks for malformation, the session stays alive and the remaining > corrupted routes are then propagated network-wide. While this may > seem like a paranoid philosophy, a little paranoia can be good > when considering the integrity of the larger whole. > > -c > > [1] = Yes, "likely" is a relative term. I know there are plenty of > cases where the same route is originated by multiple routers, > however the odds of more than one of them corrupting a route > at the same time are probably slim compared to the odds of > a single one doing so. > > [2] = In this specific case, as I understand it, the direct peers did > in fact drop the offending BGP session, however they propagated > the offending announcement to their peers before doing so. In > this case, of course, the blast radius is not controlled. > > -- Dave Israel Senior Manager, IP Backbone Intermedia Business Internet

2 1

Re: peering requirements (Re: DDOS anecdotes)
by Paul A Vixie 26 Jun '01

26 Jun '01

> maybe if we come up with a clear statement of the technical issue at hand, > those technical folk who manage to get along with their business folk can > make some forward progress. it maybe be a bit premature to throw the baby > out with the bath-water. ok, let's try it your way. anyone who wants to seriously discuss upgrading the general minimum peering agreement in use by their network so that it can include requirements about (a) not allowing spoofed source addresses to come from their network and (b) only peering, after date $TBD, with those networks who do likewise, should send me their PGP key. if i know you or if someone i know knows you, i'll add you to a private mailing list (@vix.com, not at my day job @paix.net). if you know that the right person to discuss this isn't a NANOG reader, then please forward my note internally and be prepared to (1) teach somebody pgp and (2) sign their key. (randy, i'm betting that the number of respondants will be a single digit.)

2 1

Time to revise RFC 1771
by Sean Donelan 26 Jun '01

26 Jun '01

RFC 1771's guidance on what to do with a announcement may have been filled with good intentions, however the few times it has been executed, it seems to create excessive harm to the Internet. I propose it is time to revise RFC 1771. WHEREAS, the error handling of RFC1771 is invoked where there is a dispute between two implementations of BGP. WHEREAS, these disputes may be caused by any number of reasons, including both implementations doing what they each believe are reasonable actions. WHEREAS, the resolution of these disputes may be time consuming, RESOLVED, the error handling mechanism should be revised to reject only the portion of the announcement in dispute instead of disrupting the entire BGP session. This would result in a small number of networks, most likely those who originated the disputed announcement or are closely associated with the announcer, being disrupted. But the portion of the Internet with generally accepted announcements would not be affected. This limits the colleteral damage from these disputed announcements to the small number of networks in dispute.

2 1

Re: OT: new NANOG shirt?
by George William Herbert 26 Jun '01

26 Jun '01

Someone should set up a new mailing list for the shooting run discussion. As much as I personally am interested, we're *really* off topic by now... -george william herbert gherbert(a)retro.com [PS: anyone have a Backhoe target already or do we need to get some printed?]

2 1

Re: Global BGP - 2001-06-23 - Vendor X's statement...
by Sean Donelan 26 Jun '01

26 Jun '01

On Tue, 26 June 2001, Robert E. Seastrom wrote: > > Killing 100,000 routes because you don't like one seems a bit excessive. > > Accepting a clearly defective route that may be a bellwether for a > router that has gone completely nuts seems a little bit naive and > trusting. Why do people believe it is so simple to decide what is the "bad" route? If the sender thought it was a bad route, they wouldn't have sent it in the first place. The reason they sent it was, 1) they thought it was an Ok route, 2) they didn't check it closely, 3) it changed after they checked it. You need both sides of the equation. Conservative sending: Do not send bad routes. Liberal receive: Accept the good routes, reject only the bad ones Conservative receive: If anything is wrong, reject everything (NOT THE INTERNET WAY) You seem to be saying, if you find ANY error (or even a disagreement) reject everything because you can trust nothing. This is exactly what the Internet motto cautions against. Yes, it may be a bellweather for something, but frequently the Internet has muddled along with 99.999% of things working. ISO/OSI loved to tear down sessions when it found a single bit which one side or the other thought was set wrong. The search for perfection is long, and life is short. I agree the bad route should not have been sent. However, I do not think the appropriate error action when there is a disagreement between implementations is the BGP death penalty. I believe the appropriate error handling mechanism is to accept those routes which pass your sanity check, reject those routes that do not, and ALERT the sender and operator something is wrong with some of the routes. Other implementations do not need to fix the bad route. They should not propagate the bad route. Rejecting 100,000 other good routes, which pass their sanity checks, is not a good thing.

1 0

RE: Global BGP - 2001-06-23 - Vendor X's statement...
by Richard A. Steenbergen 26 Jun '01

26 Jun '01

On Tue, Jun 26, 2001 at 11:56:21AM -0700, Sean Donelan wrote: > > On Tue, 26 June 2001, "Chance Whaley" wrote: > > How would you like Vendor X to liberally handle the situation? There is > > a point when being too liberal causes issue - like this one. The idea is > > that if the original peer followed the spec it would of been contained > > at the source and this would of never happened. Where is the line? > > Something about GIGO comes to mind. > > I would prefer implementations (not vendors) reject the one router > which they don't like, and accept the other 100,000+ routes in the > global Internet without flapping BGP sessions. > > Killing 100,000 routes because you don't like one seems a bit > excessive. There are only two situations where you will have a route that "isn't liked", 1) The sender does something wrong, 2) The receiver incorrectly handles something that is actually right. Either way, there is a fundamental flaw in someone's BGP implementation, and that needs to get resolved immediately. Flapping between the offender and its peer is probably an acceptable way to go about finding these, flapping over the entire internet because vendor C chooses to ignore the protocol spec and vendor F does not is probably not... -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

1 0

Re: peering requirements (Re: DDOS anecdotes)
by Paul A Vixie 26 Jun '01

26 Jun '01

> But please don't forget that in this particular DDoS event > there was no IP spoofing. > > So anti-spoofing precautions, either on administrative or technical > level, would be useless in this case. > > And this case is not so untypical. that doesn't matter to me. i, and people i'm various close to, am attacked several times daily. sometimes in a hard way, sometimes in a soft way, but almost always using spoofed addresses. tracking these hop by hop using mac addresses at exchange points only works if the stream is steady. it's not. > my .002$ i was not basing my recommendation for a general peering agreement upgrade on any specific attack. it's the pattern of attacks over the last decade that's got me bugged. any angry teenager with a $300 openbsd machine can bring down any part of the internet they're angry at. with impunity.

1 0

Re: Global BGP - 2001-06-23 - Vendor X's statement...
by rs＠seastrom.com 26 Jun '01

26 Jun '01

Sean Donelan <sean(a)donelan.com> writes: > I would prefer implementations (not vendors) reject the one router which > they don't like, and accept the other 100,000+ routes in the global Internet > without flapping BGP sessions. > > Killing 100,000 routes because you don't like one seems a bit excessive. Accepting a clearly defective route that may be a bellwether for a router that has gone completely nuts seems a little bit naive and trusting. ---rob

1 0

Global BGP - 2001-06-23
by lucifer＠lightbearer.com 26 Jun '01

26 Jun '01

Out of curiosity - did anyone see a duration of significanlt instability in the global routing tables on Saturday afternoon? Without violating NDA, all I can say is that it resembled a historic event involve a bad route, Ciscos, and Bay routers (only this time, it was a bad route, Ciscos, and <X> vendor whom I cannot name but is being soundly beaten with wet noodles to resolve the issue). The bad route, and instability, were seen across all of our transit vendors (all "household" names of transit service). Anyone else see this sort of event, or further details on the cause? -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer(a)lightbearer.com http://www.lightbearer.com/~lucifer

6 13