On Mon, Jun 22, 2020 at 9:43 PM Saku Ytti <saku@ytti.fi> wrote:

I can't tell you how common it is, because that type of visibility is not easy to acquire, But I can explain at least one scenario when it occasionally happens.

1) Imagine a ring of L2 metro ethernet 2) Ring is connected to two PE routers, for redundancy 3) Customers are connected to ring ports and backhauled over VLAN to PE

If there is very little traffic from Network=>Customer, the L2 metro forgets the MAC of customer subinterfaces (or VRRP) on the PE routers. Then when the client sends a packet to the Internet, the L2 floods it to all eligible ports, and it'll arrive to both PE routers, which will continue to forward it to the Internet.

Hi Saku, That's what spanning tree and its compatriots are for. Otherwise, ordinary broadcast traffic (like those arp packets) would travel in a loop, flooding the network and it would just about instantly collapse when you first turned it on. A slightly more likely scenario is a wifi link. 802.11 employs layer-2 retries across the wireless segment. When the packet is successfully transmitted but the ack is garbled, the packet may be sent a second time. Even then I wouldn't expect duplicated packets to be more than a very small fraction of a percent. Hal, if you're seeing a non-trivial amount of identical packets, my best guess is that the client is sending identical packets for some reason. NTP you say? How does iburst work during initial sync up? Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/

On 23/Jun/20 07:21, Saku Ytti wrote:

Metro: S1-S2-S3-S1 PE1: S1 PE2: S2 Customer: S3 STP blocking: ANY

S3 sends frame, it is unknown unicast flooded, S1+S2 both get it (regardless of which metro port blocks), which will send it via PE to Internet.

STP doesn't help, at all. Hope this helps.

In the above, is S3 part of the Metro-E ring, or simply downstream of S1 and S2? Mark.

On 23/Jun/20 07:32, Saku Ytti wrote:

Ring of 3 switches, minimum possible topology to explain the issue for people not familiar with L2.

To be clear, is the customer's device S3, or is S3 the ISP's device that terminates the customer's service? Mark.

On 23/Jun/20 07:52, Saku Ytti wrote:

S1-S2-S3-S1 is operator L2 metro-ring, which connects customers and 2xPE routers. It VLAN backhauls customers to PE.

Okay. In 2014, we hit a similar issue, although not in a ring. Our previous architecture was to interconnect edge routers via downstream, interconnected aggregation switches to which customers connected in order to support VRRP. Since customers do strange things, both edge routers received the same traffic, which caused pain. Since then, we don't support VRRP for customers any longer, nor do we interconnect aggregation switches that map to different edge routers. Your example scenario describes what we experienced back then. Mark.

On Tue, 23 Jun 2020 at 09:15, Sabri Berisha <sabri@cluecentral.net> wrote:

Yeah, except that unless you use static ARP entries, I can't come up with a plausible scenario in which this would happen for NTP. Assuming we're talking about a non-local NTP server, S3 will not send an NTP packet without first sending an ARP. Yes, your ARP will be flooded, but your NTP packet won't be transmitted until there is an ARP reply. By that time MACs have been learned, and the NTP packet will not be considered BUM traffic, right?

The plausible scenario is the one I explained. The crucial detail is MAC timeout (catalyst 300s) being shorter than ARP timeout (cisco 4h). So the device generating the packet knows the MAC address, the L2 does not. Hope this helps! -- ++ytti

On Tue, 23 Jun 2020 at 09:32, Sabri Berisha <sabri@cluecentral.net> wrote:

Aaah yes, fair point! Thanks $deity for default timers that make no sense.

Add low-traffic connection and default 1024s maxPoll of NTP and this duplication is guaranteed to happen for 97.9% of packets. -- ++ytti

On Tue, 23 Jun 2020 at 09:54, William Herrin <bill@herrin.us> wrote:

There's a link in the chain you haven't explained. The packet which entered at S3 has a unicast destination MAC address. That's what was in the arp table. If they're following the standards, only one of PE1 and PE2 will accept packets with that destination mac address. The other, recognizing that the packet is not addressed to it, drops it.

There are many reasons why practical devices (such as VXR) don't use MAC HW filters. Such as your PHY runs out of HW filter slots, or the HW does not support per-vlan HW filter, or there is 1 subinterface with EoMPLS configured or other type of L2 service, requiring reception of any DMAC. There are also many reasons why both routers have the DMAC in their HW filter, such as VRRP, HSRP.

for example. But then this ring configuration doesn't exist in an AWS VPC and I've not particularly observed a lot of packet duplication out of Amazon.

Amazon does nothing standard, it's all AMZN. Hope this helps! -- ++ytti

On 23/Jun/20 06:41, Saku Ytti wrote:

I can't tell you how common it is, because that type of visibility is not easy to acquire, But I can explain at least one scenario when it occasionally happens.

1) Imagine a ring of L2 metro ethernet 2) Ring is connected to two PE routers, for redundancy 3) Customers are connected to ring ports and backhauled over VLAN to PE

If there is very little traffic from Network=>Customer, the L2 metro forgets the MAC of customer subinterfaces (or VRRP) on the PE routers. Then when the client sends a packet to the Internet, the L2 floods it to all eligible ports, and it'll arrive to both PE routers, which will continue to forward it to the Internet. This requires an unfortunate (but typical) combination of ARP timeout and MAC timeout, so that sender still has ARP cache, while switch doesn't have MAC cache.

In the opposite direction this same topology can cause loops, when PE routers still have a customer MAC in the ARP table, but L2 switch doesn't have the MAC.

I wouldn't personally add code in applications to handle this case more gracefully.

My understanding of Layer 2-based Metro-E networks is that multi-directional traffic would be prevented by way of Spanning Tree. Mark.