Hello All, A customer of our's has had two major outages due to a bug in Cisco's software. The network is an MPLS VPN environment. The bug has occurred on our 6500(s) with VS-S720-10G Supervisors running s72033-adventerprisek9_wan-mz.122-33.SXH6.bin. These routers are configured as BGP route-reflectors. The first time the bug hit occurred when a new route-reflector client was added. The second time was due to a topology change on the network. On both occasions the only fix was removing the peer configurations on the RR and re-applying. Niether soft nor hard clears on the BGP neighbors worked, only the config removal. Once re-applied life was good. The bug itself was with the BGP updates sent by the RR. During the outage these updates did not include the Route Target Extended Community required by the route-reflector clients which identifies which VRF the route belongs too. Below is output on a client during the outage and after the config yanking. Notice the mysterious disappearance of the RT community. Looking to see if anyone has seen this issue particularly with this version of code. TAC is trying to tell me that this was a bug in a previous version but is fixed in the code I am running. Huh? Been running around in circles with them for a month so this is my act of desperation. Also if anyone is running a similar environment without issue I would be very interested in what version of code your using. Thanks to all who took the time to read this email... Happy Friday. *BAD:* CLN-MWB-2811-01#sh ip bgp vpnv4 all 10.180.33.22 BGP routing table entry for 102:102:10.180.33.20/30, version 1763889 Paths: (2 available, best #2, no table) Advertised to update-groups: 1 Local 10.180.20.1 (metric 9) from 10.180.20.3 (10.180.20.3) Origin incomplete, metric 0, localpref 100, valid, internal Extended Community: RT:102:102 Originator: 10.180.20.1, Cluster list: 0.0.255.245 mpls labels in/out nolabel/16 Local 10.180.20.1 (metric 9) from *10.180.20.1* (10.180.20.1) Origin incomplete, metric 0, localpref 100, valid, internal, best mpls labels in/out nolabel/16 *GOOD:* CLN-MWB-2811-01#sh ip bgp vpnv4 all 10.180.33.22 BGP routing table entry for 102:102:10.180.33.20/30, version 1765931 Paths: (2 available, best #1, table AMS) Advertised to update-groups: 1 Local 10.180.20.1 (metric 9) from *10.180.20.1* (10.180.20.1) Origin incomplete, metric 0, localpref 100, valid, internal, best *Extended Community: RT:102:102* mpls labels in/out nolabel/16 Local 10.180.20.1 (metric 9) from 10.180.20.3 (10.180.20.3) Origin incomplete, metric 0, localpref 100, valid, internal Extended Community: RT:102:102 Originator: 10.180.20.1, Cluster list: 0.0.255.245 mpls labels in/out nolabel/16 Cheers, -- Joe Renwick IP Network Consultant, CCIE #16465 GO NETFORWARD! Direct: 619-800-2055, Emergency Support: 800-719-0504 Is your network moving you forward?
On 2011-03-12, at 2:31 AM, Joe Renwick wrote:
These routers are configured as BGP route-reflectors.
...
...
...
Notice the mysterious disappearance of the RT community.
...
Interesting. I recently closed off a TAC case on a similar issue, but not an identical issue. In my case, it was 12.2(52)EY on an ME3600 and in my particular topology, an ME3600 wasn't announcing a plain ol' BGP community to one of it's two RRs. The extended communities were fine tho. Also, the announcements were being stuffed into two different update groups; the ME that was sending the 'good' announcement was announcing updates to update-group 1 and 2 and the ME that was announcing the 'bad' announcement was announcing updates to update-group 1 only. We didn't spend as much time as you clearly have troubleshooting the issue because we caught it before it was customer affecting. That said, at the time, I noticed the same thing; hard clearing the sessions didn't fix it. I didn't try to unconfigure the neighbour though; in my case, I was running EY on this switch and because the ME3600s are so new and EY1 was available and I knew that I'd have to reboot anyway to clear the issue, I decided to upgrade to EY1 and that seemed to clear up the problem. I haven't seen this resurface since. EY1 was available as soon as we started receiving our ME3600s, so as a policy we upgraded every one before it went into the field, except I had missed this one in particular. There were no open bugs pointing to my issue that the TAC engineer could find, but if you could pass me the case number, I'd like to give it to my engineer so he can see if your issue is somehow related to mine, just manifested in a slightly different way.
participants (2)
-
Jason Lixfeld -
Joe Renwick