Wow, day late, and a dollar short. :) Joe Shaw - jshaw@insync.net NetAdmin - Insync Internet Services "Backhoes never sleep." - Patrick Greenwell On Thu, 6 Aug 1998, Joe Shaw wrote:

I just talked with Time Warner, and it's wreaking havoc on their Bay quipment. We've been experiencing frequent BGP resets which have made our connection flap all day, but they seem to indicate that they have it fixed. Apparently Cisco's handle it just fine, but the Bay stuff isn't dealing well with it. Hat's off to whoever found this one out. Has anyone else who's using Bay eqipment doing BGP seen this too?

Regards, Joe Shaw - jshaw@insync.net NetAdmin - Insync Internet Services "Backhoes never sleep." - Patrick Greenwell

On Thu, 6 Aug 1998, Neil J. McRae wrote:

...

On Thu, 06 Aug 1998 13:55:02 +0100 "Neil J. McRae" <neil@domino.org> wrote:

What I meant to add was, did anyone else see something like this?

...

Aug 6 13:46:20 BGP RECV 207.45.199.225+179 -> 207.45.199.226+1935 Aug 6 13:46:20 BGP RECV message type 2 (Update) length 71 Aug 6 13:46:20 BGP RECV flags 0x40 code Origin(1): Incomplete Aug 6 13:46:20 BGP RECV flags 0x40 code ASPath(2): 6453 701 65525 ((65523))

356

...

1 1691 Aug 6 13:46:20 BGP RECV flags 0x40 code NextHop(3): 207.45.199.225 Aug 6 13:46:20 BGP RECV 204.174.40/24, 204.239.26/24, 204.239.27/24, 204 .239.147/24 Aug 6 13:46:20 Aug 6 13:46:20 bnp_path_attr_eer: peer 207.45.199.225 (External AS 6453) bad up date send NOTIFY flag 0 type 0 err_subcode 11, data 0 Aug 6 13:46:20 NOTIFICATION sent to 207.45.199.225 (External AS 6453): code 3 ( Update Message Error) subcode 11 (AS path attribute problem) data Aug 6 13:46:20 Aug 6 13:46:20 BGP SEND 207.45.199.226+1935 -> 207.45.199.225+179 Aug 6 13:46:20 BGP SEND message type 3 (Notification) length 21 Aug 6 13:46:20 BGP SEND Notification code 3 (Update Message Error) subcode 1 1 ( AS path attribute problem) Aug 6 13:46:20

Regards, Neil. -- Neil J. McRae. Alive and Kicking. Domino: In the glow of the night neil@DOMINO.ORG NetBSD-1.3 released! ftp://ftp.uk.netbsd.org/pub/NetBSD Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>

-- Neil J. McRae. Alive and Kicking. neil@DOMINO.ORG NetBSD-1.3 released! ftp://ftp.uk.netbsd.org/pub/NetBSD Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>

On Thu, 06 Aug 1998 13:55:02 +0100 "Neil J. McRae" <neil@domino.org> wrote:

What I meant to add was, did anyone else see something like this?

...

Aug 6 13:46:20 BGP RECV 207.45.199.225+179 -> 207.45.199.226+1935 Aug 6 13:46:20 BGP RECV message type 2 (Update) length 71 Aug 6 13:46:20 BGP RECV flags 0x40 code Origin(1): Incomplete Aug 6 13:46:20 BGP RECV flags 0x40 code ASPath(2): 6453 701 65525 ((65523))

356

...

1 1691

Interestingly, after studying what this was it now looks like this is how UUNET and MCI interconnect. You guys need to be more careful about leaking this crud. Really though Cisco IOS shouldn't pass these routes on. [infact all routeing software should flag it as an alarm ignore it and not pass the message on]. Regards, Neil. -- Neil J. McRae. Alive and Kicking. Domino: In the glow of the night. neil@DOMINO.ORG NetBSD/sparc: 100% SpF (Solaris protection Factor) Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>

:: Victor L. Belov writes ::

AS8263 encounted the same problem today. The problem is that cisco handle this incorrect update normally, but it couses Bay Networks routers to crash =() Seems to be another problem on the Internet.

Bays don't crash (at least not in the general case ... for example, mine stayed up this time and the last time this happened), but they do send a NOTIFY and bring down the BGP session, as required by the RFC. (I believe gated does this also.) The reason this issue causes problems is that Cisco violates the RFC and passes the bad announcement around, so it eventually reaches most non-Cisco routers who properly terminate the BGP connection. If Cisco would do the NOTIFY upon receipt of the announcement, then the information wouldn't spread, and only one router's worth of peerings (i.e. the guy who "started" the bad annoucnement) would be lost.

Neil J. McRae wrote:

...

Aug 6 13:46:20 BGP RECV 207.45.199.225+179 -> 207.45.199.226+1935 Aug 6 13:46:20 BGP RECV message type 2 (Update) length 71 Aug 6 13:46:20 BGP RECV flags 0x40 code Origin(1): Incomplete Aug 6 13:46:20 BGP RECV flags 0x40 code ASPath(2): 6453 701 65525 ((65523)) 356 1 1691 Aug 6 13:46:20 BGP RECV flags 0x40 code NextHop(3): 207.45.199.225 Aug 6 13:46:20 BGP RECV 204.174.40/24, 204.239.26/24, 204.239.27/24, 204 .239.147/24 Aug 6 13:46:20 Aug 6 13:46:20 bnp_path_attr_eer: peer 207.45.199.225 (External AS 6453) bad up date send NOTIFY flag 0 type 0 err_subcode 11, data 0 Aug 6 13:46:20 NOTIFICATION sent to 207.45.199.225 (External AS 6453): code 3 ( Update Message Error) subcode 11 (AS path attribute problem) data Aug 6 13:46:20 Aug 6 13:46:20 BGP SEND 207.45.199.226+1935 -> 207.45.199.225+179 Aug 6 13:46:20 BGP SEND message type 3 (Notification) length 21 Aug 6 13:46:20 BGP SEND Notification code 3 (Update Message Error) subcode 11 ( AS path attribute problem) Aug 6 13:46:20

- Brett (brettf@netcom.com) ------------------------------------------------------------------------------ ... Coming soon to a | Brett Frankenberger .sig near you ... a Humorous Quote ... | brettf@netcom.com

Egads. I suppose I should have had more coffee, then read the update. =) The problem I mentioned is a separate case. Never mind me. /cah On Thu, Aug 06, 1998 at 09:39:50AM -0700, Craig A. Huegen wrote: ==>On Thu, Aug 06, 1998 at 10:48:53AM -0500, Brett Frankenberger wrote: ==> ==>==>Bays don't crash (at least not in the general case ... for example, ==>==>mine stayed up this time and the last time this happened), but they do ==>==>send a NOTIFY and bring down the BGP session, as required by the RFC. ==>==>(I believe gated does this also.) ==>==> ==>==>The reason this issue causes problems is that Cisco violates the RFC ==>==>and passes the bad announcement around, so it eventually reaches most ==>==>non-Cisco routers who properly terminate the BGP connection. If Cisco ==>==>would do the NOTIFY upon receipt of the announcement, then the ==>==>information wouldn't spread, and only one router's worth of peerings ==>==>(i.e. the guy who "started" the bad annoucnement) would be lost. ==> ==>The "bad announcement" that you're talking about is capability negotiation ==>for RFC 2283, multi-protocol extensions to BGP--used for MBGP. It's not ==>a bad announcement, but an OPEN message intended to support multiple ==>NLRI types. ==> ==>The intended behavior for 11.1(20)CC is that when it sees a NOTIFY ==>of code 2, subcode 4 (unknown authentication parmeter), it backs off ==>and uses the unicast IPv4 NLRI type only (and the OPEN format without the ==>options). Unfortunately, there is a case where the Cisco will respond ==>to the TCP session close by tearing down the peer before it ever sees ==>the NOTIFY message. ==> ==>CSCdk30915 addresses this, and it will be fixed in an early interim ==>of 11.1(20)CC. A temporary workaround is to enable "neighbor x.x.x.x ==>dont-capability-negotiate" on the Cisco. ==> ==>/cah

:: Andrew Bangs writes ::

On Auguest 6th Brett Frankenberge wrote:

...

:: Victor L. Belov writes ::

...

AS8263 encounted the same problem today. The problem is that cisco handle this incorrect update normally, but it couses Bay Networks routers to crash =() Seems to be another problem on the Internet.

Bays don't crash (at least not in the general case ... for example, mine stayed up this time and the last time this happened), but they do send a NOTIFY and bring down the BGP session, as required by the RFC. (I believe gated does this also.)

The reason this issue causes problems is that Cisco violates the RFC and passes the bad announcement around, so it eventually reaches most non-Cisco routers who properly terminate the BGP connection. If Cisco would do the NOTIFY upon receipt of the announcement, then the information wouldn't spread, and only one router's worth of peerings (i.e. the guy who "started" the bad annoucnement) would be lost.

Hmm. I'm seeing something similar tonight... seeing more than one of my upstreams send me junk, and my routers send back a notify and drop the session (and my reading of the RFC matches Brett's).

Since this isn't directly my upstream's problem I've edited them out of the log (actually, this could have come from more than one of my upstreams)

Nov 8 17:45:26 BGP RECV x.x.x.x+179 -> x.x.x.x+1161 Nov 8 17:45:26 BGP RECV message type 2 (Update) length 64 Nov 8 17:45:26 BGP RECV flags 0x40 code Origin(1): Incomplete Nov 8 17:45:26 BGP RECV flags 0x40 code ASPath(2): (0x02 0x07 0x0f 0x7f 0x02 0xbd 0x0d 0xa5 0x03 0x30 0x03 0x2f 0x03 0x2e) Nov 8 17:45:26 BGP RECV flags 0x40 code NextHop(3): x.x.x.x Nov 8 17:45:26 BGP RECV flags 0xc0 code Aggregator(7): 6218 206.53.128.254 Nov 8 17:45:26 BGP RECV 206.148.144/22 Nov 8 17:45:26 Nov 8 17:45:26 bnp_path_attr_eer: peer x.x.x.x (External AS yyyy) bad update send NOTIFY flag 0 type 0 err_subcode 11, data 0 Nov 8 17:45:26 NOTIFICATION sent to x.x.x.x (External AS yyyy): code 3 (Update Message Error) subcode 11 (AS path attribute problem) data Nov 8 17:45:26 Nov 8 17:45:26 BGP SEND x.x.x.x+1161 -> x.x.x.x+179 Nov 8 17:45:26 BGP SEND message type 3 (Notification) length 21 Nov 8 17:45:26 BGP SEND Notification code 3 (Update Message Error) subcode 11 (AS path attribute problem) Nov 8 17:45:26

(We saw the problem start around 1640 GMT tonight)

Problem at AS6218 perhaps ? (of course if this is the result of some random corruption that can't be relied on... )

Anyone else see anything ?

Oh, yeah. Both my upstreams run Cisco, and both are sending me this bogus announcement. UUNet is reporting major routing instability ... perhaps this is the cause ... (to get hard, you need Cisco's at the edges and something else in the middle ... if everything is RFC-compliant, then all you lose is the BGP session to wherever you get the bad route from, and if you aren't buying transit, that should only be a small fraction of your BGP sessions.) Has Cisco fixed this yet? ObConsipracyTheory: Cisco Marketing won't let them fix this because they think it's good when all the non-Ciscos start dropping BGP sessions. - Brett (brettf@netcom.com) ------------------------------------------------------------------------------ ... Coming soon to a | Brett Frankenberger .sig near you ... a Humorous Quote ... | brettf@netcom.com

:: Scott Gifford writes ::

...

...

...

Bays don't crash (at least not in the general case ... for example, mine stayed up this time and the last time this happened), but they do send a NOTIFY and bring down the BGP session, as required by the RFC. (I believe gated does this also.)

In case any Bay Networks users didn't already know this, reasonably new version of the system software have a switch to turn off this behavior:

1:TN]$g wfBgpPeerEntry.41.* wfBgpPeerEntry.wfBgpPeerASLoopDetect.157.130.101.182.157.130.101.181 = 2 wfBgpPeerEntry.wfBgpPeerASLoopDetect.204.70.16.38.204.70.16.37 = 1 wfBgpPeerEntry.wfBgpPeerASLoopDetect.204.70.100.66.204.70.100.65 = 1 wfBgpPeerEntry.wfBgpPeerASLoopDetect.209.54.51.230.209.54.51.229 = 2 wfBgpPeerEntry.wfBgpPeerASLoopDetect.209.54.101.238.209.54.101.237 = 2

[1:TN]$set wfBgpPeerEntry.wfBgpPeerASLoopDetect.204.70.100.66.204.70.100.65 2 (41) (interface) [1:TN]$commit

Set this flag to '2' for each interface to keep your router from tearing down BGP sessions when it finds a loop. Don't forget to commit afterwards, and then to do a "save config config" so it will take after you reboot.

This wasn't a loop. This was a malformed AS path. The length of the entire AS Path attribute was 14 bytes, and the length of the first AS Sequence segment was 7 AS's, even though there wasn't room to fit that many in 14 bytes. (7AS's * 2 bytes each + 2 bytes for the segment header, gives a minimum of 16 bytes needed.) Does this attribute also disable detection of malformed AS Paths? - Brett (brettf@netcom.com) ------------------------------------------------------------------------------ ... Coming soon to a | Brett Frankenberger .sig near you ... a Humorous Quote ... | brettf@netcom.com