Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Is the BGP route getting installed into the rib? On Mon, Apr 3, 2023, 01:24 Mike Hammett <nanog@ics-il.net> wrote:
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
On 4/2/23 22:21, Mike Hammett wrote:
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
What does traceroute to that IP show? -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
On 4/2/23 22:21, Mike Hammett wrote:
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?
Another route to the destination from a routing protocol with a lower distance? What does "show ip route [destination]" look like? -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Jay Hennigan" <jay@west.net> To: nanog@nanog.org Sent: Monday, April 3, 2023 1:02:42 AM Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging On 4/2/23 22:21, Mike Hammett wrote:
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?
Another route to the destination from a routing protocol with a lower distance? What does "show ip route [destination]" look like? -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <nanog@nanog.org> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Matthew Huff" <mhuff@ox.com> To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <nanog@nanog.org> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus? From: Mike Hammett <nanog@ics-il.net> Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mhuff@ox.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________ From: "Matthew Huff" <mhuff@ox.com<mailto:mhuff@ox.com>> To: "Mike Hammett" <nanog@ics-il.net<mailto:nanog@ics-il.net>>, "NANOG" <nanog@nanog.org<mailto:nanog@nanog.org>> Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG <nanog-bounces+mhuff=ox.com@nanog.org<mailto:nanog-bounces+mhuff=ox.com@nanog.org>> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems. The snmp if index reported by the Nexus matches the if index in ElastiFlow. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Matthew Huff" <mhuff@ox.com> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus? From: Mike Hammett <nanog@ics-il.net> Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mhuff@ox.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Matthew Huff" < mhuff@ox.com > To: "Mike Hammett" < nanog@ics-il.net >, "NANOG" < nanog@nanog.org > Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG < nanog-bounces+mhuff=ox.com@nanog.org > On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG < nanog@nanog.org > Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
What about VRFs and/or policy based routing? switch-core1# show vrf VRF-Name VRF-ID State Reason default 1 Up -- management 2 Up -- switch-core1# show route-map route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 Match clauses: interface: Ethernet1/33 route-type: internal Set clauses: metric 40000000 10 255 1 1500 route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 Match clauses: interface: Ethernet1/34 route-type: internal Set clauses: metric 40000000 30 255 1 1500 route-map rmap_static_to_eigrp, permit, sequence 10 Match clauses: ip address prefix-lists: prefix_static_to_eigrp Set clauses: route-map rmap_static_to_eigrp_v6, permit, sequence 10 Match clauses: ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp Set clauses: From: Mike Hammett <nanog@ics-il.net> Sent: Monday, April 3, 2023 9:00 AM To: Matthew Huff <mhuff@ox.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems. The snmp if index reported by the Nexus matches the if index in ElastiFlow. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net> Cc: "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus? From: Mike Hammett <mailto:nanog@ics-il.net> Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mailto:mhuff@ox.com> Cc: NANOG <mailto:nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net>, "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG <mailto:nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <mailto:nanog@nanog.org> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Only two VRFs, default and manangement. IIRC, everything I saw before mentioned the default VRF. I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig through this a bit to see what's going on. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Matthew Huff" <mhuff@ox.com> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 8:06:51 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging What about VRFs and/or policy based routing? switch-core1# show vrf VRF-Name VRF-ID State Reason default 1 Up -- management 2 Up -- switch-core1# show route-map route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 Match clauses: interface: Ethernet1/33 route-type: internal Set clauses: metric 40000000 10 255 1 1500 route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 Match clauses: interface: Ethernet1/34 route-type: internal Set clauses: metric 40000000 30 255 1 1500 route-map rmap_static_to_eigrp, permit, sequence 10 Match clauses: ip address prefix-lists: prefix_static_to_eigrp Set clauses: route-map rmap_static_to_eigrp_v6, permit, sequence 10 Match clauses: ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp Set clauses: From: Mike Hammett <nanog@ics-il.net> Sent: Monday, April 3, 2023 9:00 AM To: Matthew Huff <mhuff@ox.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems. The snmp if index reported by the Nexus matches the if index in ElastiFlow. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net> Cc: "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus? From: Mike Hammett <mailto:nanog@ics-il.net> Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mailto:mhuff@ox.com> Cc: NANOG <mailto:nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net>, "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG <mailto:nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <mailto:nanog@nanog.org> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
I don't see any route-maps applied to interfaces, so there must not be any PBR going on. I only see ACLs, setting communities, setting local pref, etc. in the route maps that are applied to neighbors. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com From: "Mike Hammett" <nanog@ics-il.net> To: "Matthew Huff" <mhuff@ox.com> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 8:26:30 AM Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging Only two VRFs, default and manangement. IIRC, everything I saw before mentioned the default VRF. I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig through this a bit to see what's going on. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com From: "Matthew Huff" <mhuff@ox.com> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 8:06:51 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging What about VRFs and/or policy based routing? switch-core1# show vrf VRF-Name VRF-ID State Reason default 1 Up -- management 2 Up -- switch-core1# show route-map route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 Match clauses: interface: Ethernet1/33 route-type: internal Set clauses: metric 40000000 10 255 1 1500 route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 Match clauses: interface: Ethernet1/34 route-type: internal Set clauses: metric 40000000 30 255 1 1500 route-map rmap_static_to_eigrp, permit, sequence 10 Match clauses: ip address prefix-lists: prefix_static_to_eigrp Set clauses: route-map rmap_static_to_eigrp_v6, permit, sequence 10 Match clauses: ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp Set clauses: From: Mike Hammett <nanog@ics-il.net> Sent: Monday, April 3, 2023 9:00 AM To: Matthew Huff <mhuff@ox.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems. The snmp if index reported by the Nexus matches the if index in ElastiFlow. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net> Cc: "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus? From: Mike Hammett <mailto:nanog@ics-il.net> Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mailto:mhuff@ox.com> Cc: NANOG <mailto:nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net>, "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG <mailto:nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <mailto:nanog@nanog.org> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
You said that they are seeing traffic from another upstream…are you advertising the prefix to them? Are you advertising their prefix to your upstream? Looks like the route maps are involved in some dual redistribution…might want to make sure everything is matching correctly, and being advertised like you want. On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett <nanog@ics-il.net> wrote:
I don't see any route-maps applied to interfaces, so there must not be any PBR going on. I only see ACLs, setting communities, setting local pref, etc. in the route maps that are applied to neighbors.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"Mike Hammett" <nanog@ics-il.net> *To: *"Matthew Huff" <mhuff@ox.com> *Cc: *"NANOG" <nanog@nanog.org> *Sent: *Monday, April 3, 2023 8:26:30 AM
*Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
Only two VRFs, default and manangement. IIRC, everything I saw before mentioned the default VRF.
I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig through this a bit to see what's going on.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"Matthew Huff" <mhuff@ox.com> *To: *"Mike Hammett" <nanog@ics-il.net> *Cc: *"NANOG" <nanog@nanog.org> *Sent: *Monday, April 3, 2023 8:06:51 AM *Subject: *RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
What about VRFs and/or policy based routing?
switch-core1# show vrf VRF-Name VRF-ID State Reason
default 1 Up --
management 2 Up --
switch-core1# show route-map route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 Match clauses: interface: Ethernet1/33 route-type: internal Set clauses: metric 40000000 10 255 1 1500 route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 Match clauses: interface: Ethernet1/34 route-type: internal Set clauses: metric 40000000 30 255 1 1500 route-map rmap_static_to_eigrp, permit, sequence 10 Match clauses: ip address prefix-lists: prefix_static_to_eigrp Set clauses: route-map rmap_static_to_eigrp_v6, permit, sequence 10 Match clauses: ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp Set clauses:
From: Mike Hammett <nanog@ics-il.net> Sent: Monday, April 3, 2023 9:00 AM To: Matthew Huff <mhuff@ox.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems.
The snmp if index reported by the Nexus matches the if index in ElastiFlow.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net> Cc: "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus?
From: Mike Hammett <mailto:nanog@ics-il.net> Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mailto:mhuff@ox.com> Cc: NANOG <mailto:nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
It shows the desired result.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net>, "NANOG" <mailto: nanog@nanog.org> Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
switch-core1# sh forwarding route x.x.x.x
slot 1 =======
IPv4 routes for table default/base
------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install
------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29
switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal
From: NANOG <mailto:nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <mailto:nanog@nanog.org> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
sh ip bgp neighbor advertised-routes shows the only routes being advertised to Y are the routes that should be advertised to them. I checked a variety of other peers and have the expected results.
From my perspective:
Packets come in on port A, supposed to leave on port X, but they leave on port Y. All of the troubleshooting steps I've done on my own (or suggested by mailing lists) say the packets should be leaving on the desired port X.
From the customer's perspective, they're supposed to be coming from me on port X, but they're arriving on port Y, another network .
Port X in both scenarios is our direct connection, while port Y is a mutual upstream provider. Without knowing more about the specific platform, it seems to me like a bug in the platform. If all indicators (not just configurations, but show commands as well) say the packet should be leaving on X and it leaves on Y, then I'm not sure what else it could be, besides a bug. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "David Bass" <davidbass570@gmail.com> To: "Mike Hammett" <nanog@ics-il.net> Cc: "Matthew Huff" <mhuff@ox.com>, "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 9:12:52 PM Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging You said that they are seeing traffic from another upstream…are you advertising the prefix to them? Are you advertising their prefix to your upstream? Looks like the route maps are involved in some dual redistribution…might want to make sure everything is matching correctly, and being advertised like you want. On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett < nanog@ics-il.net > wrote: I don't see any route-maps applied to interfaces, so there must not be any PBR going on. I only see ACLs, setting communities, setting local pref, etc. in the route maps that are applied to neighbors. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com From: "Mike Hammett" < nanog@ics-il.net > To: "Matthew Huff" < mhuff@ox.com > Cc: "NANOG" < nanog@nanog.org > Sent: Monday, April 3, 2023 8:26:30 AM Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging Only two VRFs, default and manangement. IIRC, everything I saw before mentioned the default VRF. I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig through this a bit to see what's going on. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com From: "Matthew Huff" < mhuff@ox.com > To: "Mike Hammett" < nanog@ics-il.net > Cc: "NANOG" < nanog@nanog.org > Sent: Monday, April 3, 2023 8:06:51 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging What about VRFs and/or policy based routing? switch-core1# show vrf VRF-Name VRF-ID State Reason default 1 Up -- management 2 Up -- switch-core1# show route-map route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 Match clauses: interface: Ethernet1/33 route-type: internal Set clauses: metric 40000000 10 255 1 1500 route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 Match clauses: interface: Ethernet1/34 route-type: internal Set clauses: metric 40000000 30 255 1 1500 route-map rmap_static_to_eigrp, permit, sequence 10 Match clauses: ip address prefix-lists: prefix_static_to_eigrp Set clauses: route-map rmap_static_to_eigrp_v6, permit, sequence 10 Match clauses: ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp Set clauses: From: Mike Hammett < nanog@ics-il.net > Sent: Monday, April 3, 2023 9:00 AM To: Matthew Huff < mhuff@ox.com > Cc: NANOG < nanog@nanog.org > Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems. The snmp if index reported by the Nexus matches the if index in ElastiFlow. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto: mhuff@ox.com > To: "Mike Hammett" <mailto: nanog@ics-il.net > Cc: "NANOG" <mailto: nanog@nanog.org > Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus? From: Mike Hammett <mailto: nanog@ics-il.net > Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mailto: mhuff@ox.com > Cc: NANOG <mailto: nanog@nanog.org > Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto: mhuff@ox.com > To: "Mike Hammett" <mailto: nanog@ics-il.net >, "NANOG" <mailto: nanog@nanog.org > Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG <mailto: nanog-bounces+mhuff = ox.com@nanog.org > On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <mailto: nanog@nanog.org > Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
If you are both connected to the same upstream, but the customer wants traffic destined to the upstream to go through you (in and out), then they need to do something on their devices to try and affect the inbound path to their AS. From the upstream carrier in question they’ll take the best path to a prefix, which direct connection is generally going to be preferred over a transit AS (basic BGP best path algorithm stuff) unless there is some manipulation of the prefix advertisement happening. To confirm the path being taken you should be able to do a few trace routes from various locations as well as use looking glasses. Now the sflow data is an entirely different thing to analyze. On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett <nanog@ics-il.net> wrote:
sh ip bgp neighbor advertised-routes shows the only routes being advertised to Y are the routes that should be advertised to them. I checked a variety of other peers and have the expected results.
From my perspective:
Packets come in on port A, supposed to leave on port X, but they leave on port Y. All of the troubleshooting steps I've done on my own (or suggested by mailing lists) say the packets should be leaving on the desired port X.
From the customer's perspective, they're supposed to be coming from me on port X, but they're arriving on port Y, another network.
Port X in both scenarios is our direct connection, while port Y is a mutual upstream provider.
Without knowing more about the specific platform, it seems to me like a bug in the platform. If all indicators (not just configurations, but show commands as well) say the packet should be leaving on X and it leaves on Y, then I'm not sure what else it could be, besides a bug.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"David Bass" <davidbass570@gmail.com> *To: *"Mike Hammett" <nanog@ics-il.net> *Cc: *"Matthew Huff" <mhuff@ox.com>, "NANOG" <nanog@nanog.org> *Sent: *Monday, April 3, 2023 9:12:52 PM
*Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
You said that they are seeing traffic from another upstream…are you advertising the prefix to them? Are you advertising their prefix to your upstream?
Looks like the route maps are involved in some dual redistribution…might want to make sure everything is matching correctly, and being advertised like you want.
On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett <nanog@ics-il.net> wrote:
I don't see any route-maps applied to interfaces, so there must not be any PBR going on. I only see ACLs, setting communities, setting local pref, etc. in the route maps that are applied to neighbors.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"Mike Hammett" <nanog@ics-il.net> *To: *"Matthew Huff" <mhuff@ox.com> *Cc: *"NANOG" <nanog@nanog.org> *Sent: *Monday, April 3, 2023 8:26:30 AM
*Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
Only two VRFs, default and manangement. IIRC, everything I saw before mentioned the default VRF.
I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig through this a bit to see what's going on.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"Matthew Huff" <mhuff@ox.com> *To: *"Mike Hammett" <nanog@ics-il.net> *Cc: *"NANOG" <nanog@nanog.org> *Sent: *Monday, April 3, 2023 8:06:51 AM *Subject: *RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
What about VRFs and/or policy based routing?
switch-core1# show vrf VRF-Name VRF-ID State Reason
default 1 Up --
management 2 Up --
switch-core1# show route-map route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 Match clauses: interface: Ethernet1/33 route-type: internal Set clauses: metric 40000000 10 255 1 1500 route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 Match clauses: interface: Ethernet1/34 route-type: internal Set clauses: metric 40000000 30 255 1 1500 route-map rmap_static_to_eigrp, permit, sequence 10 Match clauses: ip address prefix-lists: prefix_static_to_eigrp Set clauses: route-map rmap_static_to_eigrp_v6, permit, sequence 10 Match clauses: ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp Set clauses:
From: Mike Hammett <nanog@ics-il.net> Sent: Monday, April 3, 2023 9:00 AM To: Matthew Huff <mhuff@ox.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems.
The snmp if index reported by the Nexus matches the if index in ElastiFlow.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net> Cc: "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus?
From: Mike Hammett <mailto:nanog@ics-il.net> Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mailto:mhuff@ox.com> Cc: NANOG <mailto:nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
It shows the desired result.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net>, "NANOG" <mailto: nanog@nanog.org> Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
switch-core1# sh forwarding route x.x.x.x
slot 1 =======
IPv4 routes for table default/base
------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install
------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29
switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal
From: NANOG <mailto:nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <mailto:nanog@nanog.org> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
Via all mechanisms I could find in the router, it thinks the best path is the direct path, the packets just don't go that way. The in traffic isn't a concern at this time, just the out (from my perspective). ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "David Bass" <davidbass570@gmail.com> To: "Mike Hammett" <nanog@ics-il.net> Cc: "Matthew Huff" <mhuff@ox.com>, "NANOG" <nanog@nanog.org> Sent: Tuesday, April 4, 2023 11:39:26 AM Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging If you are both connected to the same upstream, but the customer wants traffic destined to the upstream to go through you (in and out), then they need to do something on their devices to try and affect the inbound path to their AS. From the upstream carrier in question they’ll take the best path to a prefix, which direct connection is generally going to be preferred over a transit AS (basic BGP best path algorithm stuff) unless there is some manipulation of the prefix advertisement happening. To confirm the path being taken you should be able to do a few trace routes from various locations as well as use looking glasses. Now the sflow data is an entirely different thing to analyze. On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett < nanog@ics-il.net > wrote: sh ip bgp neighbor advertised-routes shows the only routes being advertised to Y are the routes that should be advertised to them. I checked a variety of other peers and have the expected results.
From my perspective:
Packets come in on port A, supposed to leave on port X, but they leave on port Y. All of the troubleshooting steps I've done on my own (or suggested by mailing lists) say the packets should be leaving on the desired port X.
From the customer's perspective, they're supposed to be coming from me on port X, but they're arriving on port Y, another network .
Port X in both scenarios is our direct connection, while port Y is a mutual upstream provider. Without knowing more about the specific platform, it seems to me like a bug in the platform. If all indicators (not just configurations, but show commands as well) say the packet should be leaving on X and it leaves on Y, then I'm not sure what else it could be, besides a bug. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com From: "David Bass" < davidbass570@gmail.com > To: "Mike Hammett" < nanog@ics-il.net > Cc: "Matthew Huff" < mhuff@ox.com >, "NANOG" < nanog@nanog.org > Sent: Monday, April 3, 2023 9:12:52 PM Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging You said that they are seeing traffic from another upstream…are you advertising the prefix to them? Are you advertising their prefix to your upstream? Looks like the route maps are involved in some dual redistribution…might want to make sure everything is matching correctly, and being advertised like you want. On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett < nanog@ics-il.net > wrote: <blockquote> I don't see any route-maps applied to interfaces, so there must not be any PBR going on. I only see ACLs, setting communities, setting local pref, etc. in the route maps that are applied to neighbors. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com From: "Mike Hammett" < nanog@ics-il.net > To: "Matthew Huff" < mhuff@ox.com > Cc: "NANOG" < nanog@nanog.org > Sent: Monday, April 3, 2023 8:26:30 AM Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging Only two VRFs, default and manangement. IIRC, everything I saw before mentioned the default VRF. I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig through this a bit to see what's going on. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com From: "Matthew Huff" < mhuff@ox.com > To: "Mike Hammett" < nanog@ics-il.net > Cc: "NANOG" < nanog@nanog.org > Sent: Monday, April 3, 2023 8:06:51 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging What about VRFs and/or policy based routing? switch-core1# show vrf VRF-Name VRF-ID State Reason default 1 Up -- management 2 Up -- switch-core1# show route-map route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 Match clauses: interface: Ethernet1/33 route-type: internal Set clauses: metric 40000000 10 255 1 1500 route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 Match clauses: interface: Ethernet1/34 route-type: internal Set clauses: metric 40000000 30 255 1 1500 route-map rmap_static_to_eigrp, permit, sequence 10 Match clauses: ip address prefix-lists: prefix_static_to_eigrp Set clauses: route-map rmap_static_to_eigrp_v6, permit, sequence 10 Match clauses: ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp Set clauses: From: Mike Hammett < nanog@ics-il.net > Sent: Monday, April 3, 2023 9:00 AM To: Matthew Huff < mhuff@ox.com > Cc: NANOG < nanog@nanog.org > Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems. The snmp if index reported by the Nexus matches the if index in ElastiFlow. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto: mhuff@ox.com > To: "Mike Hammett" <mailto: nanog@ics-il.net > Cc: "NANOG" <mailto: nanog@nanog.org > Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus? From: Mike Hammett <mailto: nanog@ics-il.net > Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mailto: mhuff@ox.com > Cc: NANOG <mailto: nanog@nanog.org > Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging It shows the desired result. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________________ From: "Matthew Huff" <mailto: mhuff@ox.com > To: "Mike Hammett" <mailto: nanog@ics-il.net >, "NANOG" <mailto: nanog@nanog.org > Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging switch-core1# sh forwarding route x.x.x.x slot 1 ======= IPv4 routes for table default/base ------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install ------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29 switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal From: NANOG <mailto: nanog-bounces+mhuff = ox.com@nanog.org > On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <mailto: nanog@nanog.org > Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com </blockquote>
Not that it's a "Fix" but have you tried rebooting the box? If this is a bug in the forwarding plane that might clear/rebuild it. And maybe it works correctly after that. Friend saw something similar on a Juniper MX with DPC cards that had run out of FIB space. It would show correctly in all places, but was failing to install in FIB (Router cut a error about it in the log). So the actual traffic didn't follow the same path. I saw you specifically referenced "forwarding" in one of your copy pastes. And it looked right. But I don't know enough about Cisco to say if that's really what is in FIB. Or just what it thinks is in FIB. On Tue, Apr 4, 2023 at 5:47 PM Mike Hammett <nanog@ics-il.net> wrote:
Via all mechanisms I could find in the router, it thinks the best path is the direct path, the packets just don't go that way.
The in traffic isn't a concern at this time, just the out (from my perspective).
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"David Bass" <davidbass570@gmail.com> *To: *"Mike Hammett" <nanog@ics-il.net> *Cc: *"Matthew Huff" <mhuff@ox.com>, "NANOG" <nanog@nanog.org> *Sent: *Tuesday, April 4, 2023 11:39:26 AM *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
If you are both connected to the same upstream, but the customer wants traffic destined to the upstream to go through you (in and out), then they need to do something on their devices to try and affect the inbound path to their AS. From the upstream carrier in question they’ll take the best path to a prefix, which direct connection is generally going to be preferred over a transit AS (basic BGP best path algorithm stuff) unless there is some manipulation of the prefix advertisement happening.
To confirm the path being taken you should be able to do a few trace routes from various locations as well as use looking glasses.
Now the sflow data is an entirely different thing to analyze.
On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett <nanog@ics-il.net> wrote:
sh ip bgp neighbor advertised-routes shows the only routes being advertised to Y are the routes that should be advertised to them. I checked a variety of other peers and have the expected results.
From my perspective:
Packets come in on port A, supposed to leave on port X, but they leave on port Y. All of the troubleshooting steps I've done on my own (or suggested by mailing lists) say the packets should be leaving on the desired port X.
From the customer's perspective, they're supposed to be coming from me on port X, but they're arriving on port Y, another network.
Port X in both scenarios is our direct connection, while port Y is a mutual upstream provider.
Without knowing more about the specific platform, it seems to me like a bug in the platform. If all indicators (not just configurations, but show commands as well) say the packet should be leaving on X and it leaves on Y, then I'm not sure what else it could be, besides a bug.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"David Bass" <davidbass570@gmail.com> *To: *"Mike Hammett" <nanog@ics-il.net> *Cc: *"Matthew Huff" <mhuff@ox.com>, "NANOG" <nanog@nanog.org> *Sent: *Monday, April 3, 2023 9:12:52 PM
*Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
You said that they are seeing traffic from another upstream…are you advertising the prefix to them? Are you advertising their prefix to your upstream?
Looks like the route maps are involved in some dual redistribution…might want to make sure everything is matching correctly, and being advertised like you want.
On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett <nanog@ics-il.net> wrote:
I don't see any route-maps applied to interfaces, so there must not be any PBR going on. I only see ACLs, setting communities, setting local pref, etc. in the route maps that are applied to neighbors.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"Mike Hammett" <nanog@ics-il.net> *To: *"Matthew Huff" <mhuff@ox.com> *Cc: *"NANOG" <nanog@nanog.org> *Sent: *Monday, April 3, 2023 8:26:30 AM
*Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
Only two VRFs, default and manangement. IIRC, everything I saw before mentioned the default VRF.
I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig through this a bit to see what's going on.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
------------------------------ *From: *"Matthew Huff" <mhuff@ox.com> *To: *"Mike Hammett" <nanog@ics-il.net> *Cc: *"NANOG" <nanog@nanog.org> *Sent: *Monday, April 3, 2023 8:06:51 AM *Subject: *RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
What about VRFs and/or policy based routing?
switch-core1# show vrf VRF-Name VRF-ID State Reason
default 1 Up --
management 2 Up --
switch-core1# show route-map route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 Match clauses: interface: Ethernet1/33 route-type: internal Set clauses: metric 40000000 10 255 1 1500 route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 Match clauses: interface: Ethernet1/34 route-type: internal Set clauses: metric 40000000 30 255 1 1500 route-map rmap_static_to_eigrp, permit, sequence 10 Match clauses: ip address prefix-lists: prefix_static_to_eigrp Set clauses: route-map rmap_static_to_eigrp_v6, permit, sequence 10 Match clauses: ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp Set clauses:
From: Mike Hammett <nanog@ics-il.net> Sent: Monday, April 3, 2023 9:00 AM To: Matthew Huff <mhuff@ox.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to looking at data for problems.
The snmp if index reported by the Nexus matches the if index in ElastiFlow.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net> Cc: "NANOG" <mailto:nanog@nanog.org> Sent: Monday, April 3, 2023 7:50:08 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp interfaces are mapped to the correct ones on the nexus?
From: Mike Hammett <mailto:nanog@ics-il.net> Sent: Monday, April 3, 2023 8:47 AM To: Matthew Huff <mailto:mhuff@ox.com> Cc: NANOG <mailto:nanog@nanog.org> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
It shows the desired result.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
________________________________________ From: "Matthew Huff" <mailto:mhuff@ox.com> To: "Mike Hammett" <mailto:nanog@ics-il.net>, "NANOG" <mailto: nanog@nanog.org> Sent: Monday, April 3, 2023 5:38:23 AM Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
switch-core1# sh forwarding route x.x.x.x
slot 1 =======
IPv4 routes for table default/base
------------------+-----------------------------------------+----------------------+-----------------+----------------- Prefix | Next-hop | Interface | Labels | Partial Install
------------------+-----------------------------------------+----------------------+-----------------+----------------- x.x.x.x/24 x.x.x.250 Ethernet1/29
switch-core1# show routing hash x.x.x.x y.y.y.y Load-share parameters used for software forwarding: load-share mode: address source-destination port source-destination Hash for VRF "default" Hashing to path *y.y.y.y Eth1/29 For route: y.y.y.0/24, ubest/mbest: 1/0 *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal
From: NANOG <mailto:nanog-bounces+mhuff=ox.com@nanog.org> On Behalf Of Mike Hammett Sent: Monday, April 3, 2023 1:21 AM To: NANOG <mailto:nanog@nanog.org> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.
I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.
However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.
If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
What started this investigation was a client complained of traffic coming from another upstream instead of our direct connection. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Mike Hammett" <nanog@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 12:21:29 AM Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
We did have our common upstream provider perform maintenance that then afterwards, had the traffic flowing on the right path. Later activity on our direct connection pushed it back to the common upstream. We haven't yet had the opportunity to bump our BGP session with the common upstream provider, but I suspect that will put the traffic back onto the right path. Seems like the router is just hanging onto the oldest BGP session it has, regardless of any other parameter or configuration. This seems like a bug. We do intend on upgrading NX-OS, but that's on someone else's schedule. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" <nanog@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 12:21:29 AM Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
We have upgraded NX-OS to a new major version and have the same results. Apr 20 09:36:05 UTC: %UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 consistency checker FAILED on slot 1 Apr 19 13:55:57 UTC: %IPFIB-2-FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM exhausted for IPV4 routes in LPM table show forwarding inconsistency ... 69. slot(1), vrf(default), prefix(198.XXX.XXX.0/YY), Route inconsistent in FIB Software. Well, those seem like problems. EASTGATE401-BGP-02# show ip route summary IP Route Table for VRF "default" Total number of routes: 292 Total number of paths: 292 Unicast paths: Best paths per protocol: Backup paths per protocol: am : 48 None local : 8 direct : 8 static : 5 broadcast : 19 bgp-XXXXX : 204 Number of routes per mask-length: /0 : 1 /8 : 1 /18: 2 /19: 4 /20: 26 /21: 24 /22: 35 /23: 18 /24: 100 /27: 1 /28: 1 /29: 1 /30: 4 /32: 74 It's choking on only 292 routes? Error Message FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM exhausted for IPV4 routes in LPM table Explanation The TCAM device in the Layer 3 forwarding ASIC has reached its system limits for IPv4 entries in the LPM table. Recommended Action No action is required. TCAM is exhausted and no action is recommended? Error Message UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 consistency checker FAILED on slot [chars] Explanation FIB Ipv4 route consistency checker Failed. Route database is consistent with hardware Recommended Action No action is required. The FIB is inconsistent and no action is recommended? EASTGATE401-BGP-02# show system internal forwarding route summary slot 1 ======= IPv4 hosts & routes summary on module 1 --------------------------------------------- Max host route entries : 8192 Total number of IPv4 host routes used: 72 Max LPM table entries : 7167 Total number of IPv4 LPM routes used : 16 I seem to be out of my depth here in that 292 is less than 7167, but yet it still fails. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com From: "Mike Hammett" <nanog@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 12:21:29 AM Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
"partial tables" There were only 292 IPv4 routes, which was expected. The IPv6 routes were expected to be less. There were 118k IPv6 routes in a box that could only handle... 4k? Fixed that errant IPv6 fill and all is well. Well, seemingly anyway. I won't rule out a coincidence, but these seems like a direct line from problem to resolution. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" <nanog@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, April 3, 2023 12:21:29 AM Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
participants (7)
-
David Bass -
Jay Hennigan -
Joelja Bogus -
Joshua Miller -
Matthew Huff -
Mike Hammett -
Nick Olsen