December 2016 - Test - lists.nanog.org

Forwarding issues related to MACs starting with a 4 or a 6 (Was: [c-nsp] Wierd MPLS/VPLS issue)
by Job Snijders 07 Dec '16

07 Dec '16

Hi all, Ever since the IEEE started allocating OUIs (MAC address ranges) in a randomly distributed fashion rather then sequentially, the operator community has suffered enormously. Time after time issues pop up related to MAC addresses that start with a 4 or a 6. I believe IEEE changed their strategy to attempt to purposefully higher the chance of collisions with MAC squatters, to encourage people to register and pay the fee. The forwarded email at the bottom is yet another example of a widely deployed, but fundamentally broken ASIC. The switch can't forward VPLS frames which contain a payload where the inner packet is destined to a MAC starting with a 4 or a 6. This is with the switch operating in pure layer-2 mode, it doesn't know what MPLS or VPLS even are. The switch is dropping packets on the floor, based on their _payload_. Try selling such circuits to customers "discounted layer-2 service, some flows might not be forwarded". Had IEEE continued the sequential OUI allocations, it probably would've taken many years before we ever reached MACs starting with a 4 or a 6, but instead, in 2012 the first linecards started rolling out of factories with MACs burned in which start with a 4 or a 6, and this took some vendors by surpise. There have been quite some issues, both in hardware and software: Brocade produced a 24x10GE linecard to the market in 2013/2014, with limited FIB scale, meant for a BGP-free MPLS core, but the card can't keep flows together on LACP bundles if the inner packets in a pseudowire were destined for a 4 or 6 MAC. The result: out of order delivery, hurting performance. Cisco ASR 9k's had a bug where if a payload started with a 6, it assumed it would be an IPv6 packet, compare the calculated packet-length with the packet-length in the packet and obviously fail because an ethernet packet is not an IPv6 packet. The result: packets dropped on the floor. (Fixed in 4.3(0.32)I) The Nexus 9000 issue described at the top of this mail. Brocade IronWare had an issue related to packet reordering for flows inside pseudowires, fixed in 2013/2014. There are probably many more examples out there in the wild, slowly driving operators insane. At this moment, some issues related to MACs starting with a 4 or a 6 can be mitigated if you enable Pseudowire Control-Word (RFC 4385) _AND_ Flow-Aware Transport (RFC 6391). You need both to mitigate certain issues in multi-vendor networks (for instance if you have Cisco edge + Juniper core). But what to do when the ASIC won't forward the payload? As ISP you often don't control the payload. Unfortunatly, I don't think we've seen the end of this. The linecards bought in 2012 will trickle down to the grey/second-hand market about now, often without accompanying support contracts. In a world with increased complexity in our interconnectedness, and lack of visibility into the underlaying infrastructure (think remote peering, cloud connectivity, resellers reselling layer-2) it will hurt when some flows inexplicably fail to arrive. Dear IEEE, please pause assigning MAC addresses that start with a 4 or a 6 for the next 6 years. Or at least, next time you change the policy, consult the operational community. This 4/6 MAC issue was well documented in BCP128 back in 2007. The control-word drafts mentioned that there would be dragons related to 4 and 6 back in 2004. Dear Vendors, take this issue more serious. Realise that for operators these issues are _extremely_ hard to debug, this is an expensive time sink. Some of these issues are only visible under very specific, rare circumstances, much like chasing phantoms. So take every vague report of "mysterious" packetloss, or packet reordering at face value and immediately dispatch smart people to delve into whether your software or hardware makes wrong assumptions based on encountering a 4 or a 6 somewhere in the frame. And you, my fellow operators, please continue to publicly document these issues and possible workarounds. Kind regards, Job resources: c-nsp thread "Wierd MPLS/VPLS issue": https://puck.nether.net/pipermail/cisco-nsp/2016-December/thread.html https://www.nanog.org/meetings/nanog57/presentations/Tuesday/tues.general.S… BCP128: https://tools.ietf.org/html/bcp128 ----- Forwarded message from Simon Lockhart <simon(a)slimey.org> ----- Date: Fri, 2 Dec 2016 11:44:21 +0000 From: Simon Lockhart <simon(a)slimey.org> To: cisco-nsp(a)puck.nether.net Subject: Re: [c-nsp] Wierd MPLS/VPLS issue On Wed Nov 23, 2016 at 12:01:20PM +0000, Simon Lockhart wrote: > On Fri Nov 04, 2016 at 03:40:05PM +0000, Simon Lockhart wrote: > > To me, everything *looks* right, it's just that some VPLS traffic traversing > > the new link gets lost. > > For those who are interested... > > Well, I finally got to the bottom of this, and have pushed it to Cisco TAC > for a fix... Cisco TAC finally accepted the issue. Bug CSCvc33783 has been logged. Nexus BU has investigated. Response is... "[...] unfortunately this is an ASIC limitation on the Nexus 9000 switches and is therefore not fixable." If you want a Layer 2 switch that will forward all valid Ethernet frames, I'd suggest avoiding the Nexus 9000 range... Simon _______________________________________________ cisco-nsp mailing list cisco-nsp(a)puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ----- End forwarded message -----

11 20

Verizon Email Contacts
by ping murder 07 Dec '16

07 Dec '16

Hello, I need to connect with an admin w/ Verizon's email abuse team. Thanks, ping

1 0

Looking for a Comcast engineer
by Chris Lane 07 Dec '16

07 Dec '16

If a Comcast engineer is available mind hitting me up offline. I am not a customer, but a former customer of mine is still advertising my IP /24 out to you Comcast. the prefix belongs to our company and the former customer won't return my calls. I'd like you to remove prefix from filters. Much regards, Chris

1 1

Favorite Speed Test Systems
by Graham Johnston 06 Dec '16

06 Dec '16

For many years we have had a local instance of the Ookla speedtest.net on our network, and while it is pretty good some other tests seem include more detailed results. I am aware of the following speedtest systems that an operator can likely have a local instance of: * Speedtest.net * Sourceforge.net/speedtest * Dslreports.com/speedtest Are there others? What is your preferred one and why? Thanks, Graham

17 21

Canadian National Railway contact
by Andy Ringsmuth 06 Dec '16

06 Dec '16

If there happens to be someone here from the Canadian National Railway, or if someone knows someone there, could you hit me up off-list? Attempting to work through an e-mail block from us to them that I’ve been unsuccessful remedying so far. Much appreciated! ---- Andy Ringsmuth andy(a)newslink.com News Link – Manager Travel, Technology & Facilities 2201 Winthrop Rd., Lincoln, NE 68502-4158 (402) 475-6397 (402) 304-0083 cellular

2 1

Level3 / Cogent / NetFlix BGP Assistance
by Sam Norris 06 Dec '16

06 Dec '16

Hey all, In early October our traffic levels from NetFlix went from about half Level3 and half Cogent - pretty well balanced - to all on Level3. Standard multihomed BGP setup with minimal TE if I can help it. I am starting to run into problems with a full gigabit port on Level3 and only 100-200mbps on Cogent at that colo. I tried padding, I even tried 65000:2906 bgp community, but it seems like if our level3 port is up NetFlix chooses it solely. How can I load balance NetFlix traffic across my two ports so that I am not paying for overages and/or forced to up to a 10G port immediately? I have 3 other providers at that colo and cannot find any mix of bgp tricks to get some of it thru our other providers. I notice AS2906 has a localpref of 86 - odd ... The 108.175.47.0/24 block seems to be using as 2906 so I thought a 65000:2906 wouldn't announce those prefixes to them at all. I have about 10 prefixes being announced so I could implement a no-export on some of them to load balance but that doesn't work. Thx, Sam BGP routing table entry for 108.175.47.0/24 Paths: (2 available, best #1) 2906 AS-path translation: { 108.175.47.0/24 } ear3.LosAngeles1 (metric 100000) Origin IGP, metric 30334, localpref 86, valid, internal, best Community: 2906:51081 North_America Lclprf_86 United_States Level3_Peer Los_Angeles Level3:10984 Originator: ear3.LosAngeles1 2906 AS-path translation: { 108.175.47.0/24 } ear3.LosAngeles1 (metric 100000) Origin IGP, metric 30334, localpref 86, valid, internal Community: 2906:51081 North_America Lclprf_86 United_States Level3_Peer Los_Angeles Level3:10984 Originator: ear3.LosAngeles1

4 3

Need contact info for AS1798 - State of Oregon
by Tyler Applebaum 05 Dec '16

05 Dec '16

If you could contact me off-list, I'd appreciate it. - Tyler

1 0

Acquiring unused IP range. Some questions
by William McLendon 05 Dec '16

05 Dec '16

Hi everyone, we are about to acquire a block of IP’s from another organization that has unused space, and being fairly new to these procedures, I was hoping for some guidance. We have already been pre-approved by ARIN for the block size we are acquiring, and finalizing the deal with the current owner of the address space. First question is, once they initiate the transfer request to transfer the IP range to us, how long does that typically take for ARIN to complete? Secondly, our relationship with the IP block owner is a very good one, such that I think they would be ok with us advertising this block before we technically own it. My question is, what do they and we need to do to accomplish that in the proper way, so that the internet at large would accept the advertisement from a different ASN, and not view as some sort of hijacking, etc. I am guessing they may need to update some RADB or something like that, but i’ll be honest my knowledge of how those things work and their complete function is pretty slim. This would be a short term thing as we expect the purchase process to complete pretty quickly, but it would be advantageous to us to be able to advertise the space immediately. We just want to make sure we start off on the right foot. Thanks, Will

4 3

Wormly SMTP Test Message
by Wormly SMTP Test 04 Dec '16

04 Dec '16

This message was sent using the Wormly SMTP testing tool by this user: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 197.230.17.94

1 0

Wormly SMTP Test Message
by Wormly SMTP Test 04 Dec '16

04 Dec '16

This message was sent using the Wormly SMTP testing tool by this user: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 197.230.17.94

1 0