December 2016 - Test - lists.nanog.org

Internet Governance Forum DNS
by Joly MacFie 09 Dec '16

09 Dec '16

"www.intgovforum.org’s server DNS address could not be found." and http://downforeveryoneorjustme.com/www.intgovforum.org is negative. Any clues as to what's up? -- --------------------------------------------------------------- Joly MacFie 218 565 9365 Skype:punkcast -------------------------------------------------------------- -

3 3

load balancers convergence (Radware)
by Mihai 08 Dec '16

08 Dec '16

Hi, Sorry if this is not the right list to post but it's the last resort and any clue would be highly appreciated. I am new to LBs world and have the following active-standby topology (Alteon 5524): Router1------------Router2 | | | | eBGP eBGP | | | | Alteon1(act) Alteon2(stb) | | |_______________________| | | WEB servers - BGP local-pref is higher on the R1-A1 session. - VRRP priority is higher on Alteon1. - both Lbs advertise the same VIP address in BGP. - NAT is configured for WEB servers IPs. I am using these devices to load balance HTTP traffic and have the following issue: 1. Alteon1 fails and the traffic moves to Alteon2 without problems. 2. After Alteon1 recovers, it becomes the VRRP master due to higher priority, but the BGP session between Alteon1 and Router1 establishes after VRRP preemption (more than 1 minute after A1 becomes master) and the traffic gets dropped. I tried to use the hold-off timer to delay the VRRP preemption to match the BGP session establishment but still have ~30s downtime. Creating a direct link and BGP session between Alteons does not help as the traffic will be asymmetrical and is dropped on Alteon1. Regards

1 0

Cogent Router code updates during height of ecommerce season?
by Doc Flatline 08 Dec '16

08 Dec '16

Could. https://quickview.cloudapps.cisco.com/quickview/bug/CSCtd35382 https://github.com/Sab0tag3d/SIET

1 0

Forwarding issues related to MACs starting with a 4 or a 6 (Was: [c-nsp] Wierd MPLS/VPLS issue)
by Job Snijders 07 Dec '16

07 Dec '16

Hi all, Ever since the IEEE started allocating OUIs (MAC address ranges) in a randomly distributed fashion rather then sequentially, the operator community has suffered enormously. Time after time issues pop up related to MAC addresses that start with a 4 or a 6. I believe IEEE changed their strategy to attempt to purposefully higher the chance of collisions with MAC squatters, to encourage people to register and pay the fee. The forwarded email at the bottom is yet another example of a widely deployed, but fundamentally broken ASIC. The switch can't forward VPLS frames which contain a payload where the inner packet is destined to a MAC starting with a 4 or a 6. This is with the switch operating in pure layer-2 mode, it doesn't know what MPLS or VPLS even are. The switch is dropping packets on the floor, based on their _payload_. Try selling such circuits to customers "discounted layer-2 service, some flows might not be forwarded". Had IEEE continued the sequential OUI allocations, it probably would've taken many years before we ever reached MACs starting with a 4 or a 6, but instead, in 2012 the first linecards started rolling out of factories with MACs burned in which start with a 4 or a 6, and this took some vendors by surpise. There have been quite some issues, both in hardware and software: Brocade produced a 24x10GE linecard to the market in 2013/2014, with limited FIB scale, meant for a BGP-free MPLS core, but the card can't keep flows together on LACP bundles if the inner packets in a pseudowire were destined for a 4 or 6 MAC. The result: out of order delivery, hurting performance. Cisco ASR 9k's had a bug where if a payload started with a 6, it assumed it would be an IPv6 packet, compare the calculated packet-length with the packet-length in the packet and obviously fail because an ethernet packet is not an IPv6 packet. The result: packets dropped on the floor. (Fixed in 4.3(0.32)I) The Nexus 9000 issue described at the top of this mail. Brocade IronWare had an issue related to packet reordering for flows inside pseudowires, fixed in 2013/2014. There are probably many more examples out there in the wild, slowly driving operators insane. At this moment, some issues related to MACs starting with a 4 or a 6 can be mitigated if you enable Pseudowire Control-Word (RFC 4385) _AND_ Flow-Aware Transport (RFC 6391). You need both to mitigate certain issues in multi-vendor networks (for instance if you have Cisco edge + Juniper core). But what to do when the ASIC won't forward the payload? As ISP you often don't control the payload. Unfortunatly, I don't think we've seen the end of this. The linecards bought in 2012 will trickle down to the grey/second-hand market about now, often without accompanying support contracts. In a world with increased complexity in our interconnectedness, and lack of visibility into the underlaying infrastructure (think remote peering, cloud connectivity, resellers reselling layer-2) it will hurt when some flows inexplicably fail to arrive. Dear IEEE, please pause assigning MAC addresses that start with a 4 or a 6 for the next 6 years. Or at least, next time you change the policy, consult the operational community. This 4/6 MAC issue was well documented in BCP128 back in 2007. The control-word drafts mentioned that there would be dragons related to 4 and 6 back in 2004. Dear Vendors, take this issue more serious. Realise that for operators these issues are _extremely_ hard to debug, this is an expensive time sink. Some of these issues are only visible under very specific, rare circumstances, much like chasing phantoms. So take every vague report of "mysterious" packetloss, or packet reordering at face value and immediately dispatch smart people to delve into whether your software or hardware makes wrong assumptions based on encountering a 4 or a 6 somewhere in the frame. And you, my fellow operators, please continue to publicly document these issues and possible workarounds. Kind regards, Job resources: c-nsp thread "Wierd MPLS/VPLS issue": https://puck.nether.net/pipermail/cisco-nsp/2016-December/thread.html https://www.nanog.org/meetings/nanog57/presentations/Tuesday/tues.general.S… BCP128: https://tools.ietf.org/html/bcp128 ----- Forwarded message from Simon Lockhart <simon(a)slimey.org> ----- Date: Fri, 2 Dec 2016 11:44:21 +0000 From: Simon Lockhart <simon(a)slimey.org> To: cisco-nsp(a)puck.nether.net Subject: Re: [c-nsp] Wierd MPLS/VPLS issue On Wed Nov 23, 2016 at 12:01:20PM +0000, Simon Lockhart wrote: > On Fri Nov 04, 2016 at 03:40:05PM +0000, Simon Lockhart wrote: > > To me, everything *looks* right, it's just that some VPLS traffic traversing > > the new link gets lost. > > For those who are interested... > > Well, I finally got to the bottom of this, and have pushed it to Cisco TAC > for a fix... Cisco TAC finally accepted the issue. Bug CSCvc33783 has been logged. Nexus BU has investigated. Response is... "[...] unfortunately this is an ASIC limitation on the Nexus 9000 switches and is therefore not fixable." If you want a Layer 2 switch that will forward all valid Ethernet frames, I'd suggest avoiding the Nexus 9000 range... Simon _______________________________________________ cisco-nsp mailing list cisco-nsp(a)puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ----- End forwarded message -----

11 20

Verizon Email Contacts
by ping murder 07 Dec '16

07 Dec '16

Hello, I need to connect with an admin w/ Verizon's email abuse team. Thanks, ping

1 0

Looking for a Comcast engineer
by Chris Lane 07 Dec '16

07 Dec '16

If a Comcast engineer is available mind hitting me up offline. I am not a customer, but a former customer of mine is still advertising my IP /24 out to you Comcast. the prefix belongs to our company and the former customer won't return my calls. I'd like you to remove prefix from filters. Much regards, Chris

1 1

Favorite Speed Test Systems
by Graham Johnston 07 Dec '16

07 Dec '16

For many years we have had a local instance of the Ookla speedtest.net on our network, and while it is pretty good some other tests seem include more detailed results. I am aware of the following speedtest systems that an operator can likely have a local instance of: * Speedtest.net * Sourceforge.net/speedtest * Dslreports.com/speedtest Are there others? What is your preferred one and why? Thanks, Graham

17 21

Canadian National Railway contact
by Andy Ringsmuth 06 Dec '16

06 Dec '16

If there happens to be someone here from the Canadian National Railway, or if someone knows someone there, could you hit me up off-list? Attempting to work through an e-mail block from us to them that I’ve been unsuccessful remedying so far. Much appreciated! ---- Andy Ringsmuth andy(a)newslink.com News Link – Manager Travel, Technology & Facilities 2201 Winthrop Rd., Lincoln, NE 68502-4158 (402) 475-6397 (402) 304-0083 cellular

2 1

Level3 / Cogent / NetFlix BGP Assistance
by Sam Norris 06 Dec '16

06 Dec '16

Hey all, In early October our traffic levels from NetFlix went from about half Level3 and half Cogent - pretty well balanced - to all on Level3. Standard multihomed BGP setup with minimal TE if I can help it. I am starting to run into problems with a full gigabit port on Level3 and only 100-200mbps on Cogent at that colo. I tried padding, I even tried 65000:2906 bgp community, but it seems like if our level3 port is up NetFlix chooses it solely. How can I load balance NetFlix traffic across my two ports so that I am not paying for overages and/or forced to up to a 10G port immediately? I have 3 other providers at that colo and cannot find any mix of bgp tricks to get some of it thru our other providers. I notice AS2906 has a localpref of 86 - odd ... The 108.175.47.0/24 block seems to be using as 2906 so I thought a 65000:2906 wouldn't announce those prefixes to them at all. I have about 10 prefixes being announced so I could implement a no-export on some of them to load balance but that doesn't work. Thx, Sam BGP routing table entry for 108.175.47.0/24 Paths: (2 available, best #1) 2906 AS-path translation: { 108.175.47.0/24 } ear3.LosAngeles1 (metric 100000) Origin IGP, metric 30334, localpref 86, valid, internal, best Community: 2906:51081 North_America Lclprf_86 United_States Level3_Peer Los_Angeles Level3:10984 Originator: ear3.LosAngeles1 2906 AS-path translation: { 108.175.47.0/24 } ear3.LosAngeles1 (metric 100000) Origin IGP, metric 30334, localpref 86, valid, internal Community: 2906:51081 North_America Lclprf_86 United_States Level3_Peer Los_Angeles Level3:10984 Originator: ear3.LosAngeles1

4 3

Need contact info for AS1798 - State of Oregon
by Tyler Applebaum 05 Dec '16

05 Dec '16

If you could contact me off-list, I'd appreciate it. - Tyler

1 0