Sent from my iPhone

On Apr 16, 2023, at 10:05, Mike Hammett <nanog@ics-il.net> wrote:


We did have our common upstream provider perform maintenance that then afterwards, had the traffic flowing on the right path. Later activity on our direct connection pushed it back to the common upstream. We haven't yet had the opportunity to bump our BGP session with the common upstream provider, but I suspect that will put the traffic back onto the right path. Seems like the router is just hanging onto the oldest BGP session it has, regardless of any other parameter or configuration.

This seems like a bug. We do intend on upgrading NX-OS, but that's on someone else's schedule.

Not that familiar  with nexus 3k but I would compare the route in the rib and that on the module.  If the platform is exhausting unicast route entries the control plan may  show the route when  module / asic doesn’t have it installed. It’s not always obviously when these things run out of tcam 

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/9-x/unicast/configuration/guide/l3_cli_nxos/l3_manage-routes.pdf

This is nexus 9k 
 
But I would expect it to be broadly similar  in terms of diagnostic on the other platforms

Nexus 3ks are Broadcom merchant silicon (trident/tomahawk so I’d expect the them to run out of fib in the mid tens of thousands of routes..



-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

From: "Mike Hammett" <nanog@ics-il.net>
To: "NANOG" <nanog@nanog.org>
Sent: Monday, April 3, 2023 12:21:29 AM
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? 



-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com