October 2021 - Test - lists.nanog.org

Anyone from Level3/CenturyLink/Lumen, possibly Comcast around?
by Brie 15 Oct '21

15 Oct '21

Hi all, So, having a... frustrating issue going on. Long wall of text ahead as I explain. 1 x CenturyLink/Lumen fiber in Boise 1 x CenturyLink/Lumen fiber in Cheyenne 1 x Comcast biz fiber in Denver IPsec VPN tunnels between all three sites, w/ OSPF for routing failover (which unfortunately doesn't help in this situation). Two days ago, Cheyenne to Denver (.196) traffic (both tcp and udp) were an issue initially. Failed over to routing Cheyenne VPN through Boise while we opened ticket with CL. Yesterday, Boise to Denver (.196) traffic started having exact same issue. Tests from another CL fiber in Boise (my own circuit, with legacy IP space and BGP) to Denver (.196) did not show same issues. Path appeared clean. Traceroutes from Office Boise to Denver (.196) had a noticeable difference from Personal Boise to Denver (.196): Office Boise -> Denver (.196) ---------------------- 3: sea-edge-15.inet.qwest.net 4: lag-4.ear3.Seattle1.Level3.net 5: ae-2-52.ear2.seattle1.level3.net <------ This hop 6: be-203-pe01.seattle.wa.ibone.comcast.net Personal Boise -> Denver (.196) ---------------------- 4: sea-edge-15.inet.qwest.net 5: lag-25.ear2.Seattle1.Level3.net 6: be-203-pe01.seattle.wa.ibone.comcast.net On a whim, tracerouted to another Denver router IP address (.199, IP alias on same interface) from Boise Office, and traceroute matched the traceroute from Personal Boise to Denver (.196) traceroute. No packet loss. Swapped VPN tunnels over to using another ip on same router (.199), same interface physical and logical, in Denver, and VPN was working again normally. This morning though, Cheyenne to Denver (.199) is having problems, while Boise to Denver (.199) isn't (for now). Already spent most of last night working with my partner in Denver replacing nearly everything on the Denver side with no change. Tests from the router above the main Denver VPN endpoint (.196) do not show any kind of issues or packet loss to it, so it doesn't appear the problem is inside of our network. I'm not inclined to think this is a Comcast issue, but I can't be sure. This sounds almost like a load balancing hashing issue, with one link in the bond group having issues, somewhere in one of our upstream's networks. We'll be opening a ticket in a bit through normal channels with CenturyLink/Lumen, but we're worried they're just going to close the ticket as not being their issue. Anyone know of an engineer at CenturyLink/Lumen/Level3 or even Comcast that might want to take a stab at this? I can provide a lot more detail. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

4 3

verizon fios, northeast, routing issues?
by Miles Fidelman 15 Oct '21

15 Oct '21

Any Verizon folks here? I've been having some rather weird network issues lately - just reading email via IMAP, from home. Over a 1gig FIOS connection to a machine in a nearby Tierpoint data center that has LOTS of good connectivity. I just tried some traceroutes, and got some interesting results: These originate on a machine connected to a 1gig FIOS feed, and end at a machine, located in a Tierpoint datacenter, about 10 miles from here. traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms 2 * * * 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms 8.477 ms 17.097 ms 5 * * * 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms 7 * * * 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms 8.331 ms 13.161 ms 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms Several things jump out: 1. alter.net is not a common path between here & there - usually a lower grade connection, when other backbones aren't working right 2. origin - alter.net - level.3 - endpoint is just bizarre, one would think that the regional FIOS network has a direct connection to level.3 (it also seems kind of odd that the packets are flowing from Acton MA, to Boston, and back out to Marlboro MA - there's an awful lot of fiber running along Rt. 495, and the networks are fairly dense around here) 3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts) All in all it's really mucking with both streaming services, and simply posting emails (SMTP timeouts). All of which leads me to wonder if there's something mucked up with Verizon's routing tables (or a particular network interface). Any insights (or fixes) to be had? Thanks, Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. ... unknown

4 5

Increase bandwidth usage in partial-mesh network?
by Adam Thompson 14 Oct '21

14 Oct '21

Looking for recommendtions or suggestions... I've got a downstream customer asking for help; they have a private internal network that I've taken to calling the "partial-mesh network from hell": it's got two partially-overlapping radio networks, mixed with islands of isolated fiber connectivity. Dynamic routing protocols (IS-IS, OSPF, EIGRP, etc.) generally will only select the _best_ path, they won't spread the load unless all paths are equal - and they are very unequal in this network, ECMP would likely fail horribly. The network is becoming bandwidth-limited, so they're wanting to make use of all available paths, not just the single "best" path. It's also remote and spread out, so adding new links or upgrading existing links is difficult and expensive. Oh, and their routers are overdue for a refresh, so acquiring replacement h/w is now possible. Has anyone come across any product or technology that can handle the multi-path-ness and the private-network-ness like a regular router, but also provides the intelligent per-flow path steering based on e.g. latency, like an SD-WAN device (and/or some firewalls)? Here's hoping, -Adam Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson(a)merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/>

9 9

NANOG 83: City Guide + Keynotes + More
by Nanog News 13 Oct '21

13 Oct '21

A NANOG 83 Attendees' Guide to Minneapolis Get ready to feel welcomed in the industrial landmark city, known as the birthplace of iconic musicians, incredible museums, + the "Minnesota nice" <https://theculturetrip.com/north-america/usa/minnesota/articles/15-things-k…> culture. Our next meeting, NANOG 83 (Nov. 1 - 3), will be located near the breathtaking end of the Mississippi River in Minnesota. When community members need a break from improving the Internet of tomorrow, here is a list of the "best of the best" to experience in Minneapolis. Meet our Keynote Speakers "Geeky Entrepreneur" + Government Intelligence Cybersecurity Pro *Meet Bert Hubert:* Bert (@PowerDNS_Bertis) is the founder of PowerDNS, a software that powers a significant fraction of the Internet. He also did government cybersecurity work and co-founded a software company in that field. These days, Bert does DNA research and is part of the governing board regulating the Dutch intelligence and security agencies. *Date:* Tuesday, Nov 2 at 10 am Central virtually, live Q&A. *Talk Title: *Who Controls the Internet? And Should They? *Abstract of Presentation:* This presentation discusses how control of the internet experience is moving more and more into the hands of browser and phone vendors. The advent of end-to-end encryption, also on control planes and metadata like DNS, means that no one else can influence the Internet - except in extremely heavy-handed and binary fashion... Meet our Keynote Speakers Chief Technology Officer & Vice President, Product Engineering & Infrastructure *Meet John Brzozowski: *@jjmbcom - Talks about #iot, #lorawan, and #loraalliance. John is an executive technology leader with a deep understanding of product development and infrastructure across multiple industries and diverse product and service offerings. *Date: *Wednesday, Nov 3 at 10 am Central virtually, live from the Grand Lakes Ballroom. *Talk Title: * IPv6 - The Next 10 years *Abstract of Presentation: *World IPv6 Day was in 2011, World IPv6 Launch in 2012. We will briefly reflect on the status of IPv6 deployment across eyeball and content networks - 10 years later. We will take a look at statistics across a wide range of public and private (cited) sources... We Have Missed You! Less than 3 Weeks until NANOG 83 - Register Now! *Reunited, and it feels so good. *We cannot wait to see you in person and/or virtually at NANOG 83 in less than three weeks! Complimentary registration is available for those in need and virtual only. *Have concerns about traveling during the pandemic?* Check out our Safety + Travel Guide. <https://www.nanog.org/events/nanog-83/safety-travel/> Time is running out, don't miss all the incredible networking opportunities and programming now. Let's Hack! Still, Time to Register for NANOG 83 Hybrid Hackathon *There's still time! *Register for the Hackathon today. Virtual kick-off is on Oct 22, + hacking is on Oct 30 - 31. *Hackathons are an essential part of NANOG conferences,* designed to be both fun and engaging. NANOG Hackathons are hands-on and educational at their core — directly supporting the most critical aspects of our mission. All levels are welcome to participate, and as always, registration is free.

1 0

CSR1000v + ASR1000 Code Upgrade Pleasure...
by Mark Tinka 13 Oct '21

13 Oct '21

Hi all. I thought I'd share our recent experiences, per subject, just in case others run into the same problems. So... we finally decided to try 17.3(4a)MD for the CSR1000v, after years of happy operation. Good Lord, what a drama! At first, we couldn't figure out why iBGP sessions to all Cisco boxes could not stand up. Then we realized it's because IS-IS to them could not stand up. Then we realized it's because BFD sessions could not stand up. But even after removing BFD, IS-IS remained down. After 3 days of searching, we finally landed on CSCuz58508. In case you don't have CCO access, it is the same issue as described here: https://community.cisco.com/t5/cisco-cloud-service-router-csr/b00ocg4q4e-cs… This was even more confusing for us, because our interface driver on VMware ESXi is vmxnet3. The bug ID suggests the problem is fixed in 16.3(2) and 16.4(1). So to be safe, we tested 16.12(5)MD, which allowed us to enable jumbo frames, but that only appeared to be a cosmetic thing. In the background, the box was simply dropping packets, silently. We found this out when we tried to copy other files to the node, and it would just hang without any feedback. Removing the jumbo frame support allowed the files to come through. We noticed that nodes still running 3.17(0)S did not have any issues with IS-IS or BFD, or MTU. However, this code was only ever released as an ED train (and to be fair, we've been having dodgy issues with it in recent years), so we decided to downgrade to 3.16(9)S (which is actually an upgrade from 3.17(00)S, since the 3.16 train is an MD release, with the latest release being March 2019, vs. July 2017 for 3.17(4)SED). With that, no more MTU issues, BFD and IS-IS are happy, iBGP is happy. We definitely won't be wasting any more time trying to make Denali, Gibraltar, Fuji, Everest or Amsterdam work on our CSR1000v complement. Needless to say, moving the ASR1000 platform to 17.3 has also come with its own avenue of pleasure, what with all the ROMMON, CPLD and FPGA upgrade mess that is. What the documentation says and what happens in real life are two very different things. It has taken us a week to come up with our own working procedure to upgrade just one box, worse if it's a dual-RP system. Mark.

1 0

Linux WiFi Package Issues
by Pascal Masha 12 Oct '21

12 Oct '21

Hello All, I have been wondering whether there is any known issue with the Linux WiFi package since the last 3 days or so? I'm Ubuntu 20.04.3 LTS 64bit Distro and WiFi has been dropping almost every 5 minutes. A colleague on another Linux Disto also contacted me about the same thing. Has anyone in the community experienced the same issue? Regards Paschal Masha

3 2

Re: S.Korea broadband firm sues Netflix after traffic surge
by Jared Brown 12 Oct '21

12 Oct '21

Mark Tinka wrote: > Someone can correct me if I'm wrong, but the way I know BitTorrent to > work is the file is downloaded to disk, unarchived and then listed as > ready to watch. That's not how it works. Several streaming BitTorrent clients specifically request blocks in order so that you can start watching immediately. Not that you need a special client, it works pretty well with the standard client as well on a well seeded torrent, as blocks are generally requested more or less in order. > It also assumes the device has all the necessary apps > and codecs needed to render the file. Well, yes. Or you could just stream content that is guaranteed to be compatible with the device used. > On the other hand, BitTorrent could just make an Apple > TV/PS4/PS5/Xbox/whatever-device-you-use app as well. They could, and they might even have, I forget, but there is little demand for such a thing as a centralized CDN strategy works better. > But I doubt that > will work, unless someone can think up a clever way to modify BitTorrent > to suit today's network architectures. Unless network topology is somehow exposed, this isn't possible. All anybody can do is use latency, IP and ASN information as a proxy. Nothing is stopping a BitTorrent client from being selective about its peers. The current peer selection algorithm optimizes for throughput, not adjecency or topology. - Jared

4 3

Re: S.Korea broadband firm sues Netflix after traffic surge
by Jared Brown 12 Oct '21

12 Oct '21

Doug Barton wrote: > One incentive I haven't seen anyone mention is that ISPs don't want to > charge customers what it really costs to provide them access. For the sake of argument, let's assume this is true. For this to work, I am really trying hard to ignore inconvenient facts like: "South Korea’s SK Telecom (SKT) has reported operating revenues of KRW4.818 trillion (USD4.2 billion) for the quarter ended 30 June 2021, up 4.7% year-on-year, with it saying that the increase was ‘due to continued solid growth trends in all business areas’. SKT’s operating income in Q2 2021 totalled KRW397 billion, up 10.8% on an annualised basis..." https://www.commsupdate.com/articles/2021/08/12/sk-telecom-reports-revenue-… Nevertheless, let's go with the hypothesis that service is provided below cost. Providing access is mostly fixed costs, as there are very few consumables in running a network. IP transit costs aren't an issue, since Netflix will do settlement free peering. This leaves the internal network of SK Telecom as the problem and cost center. There would have been no marginal cost if SK Telecom's own network was capable of handling the traffic of its customers. So basically SK Telecom is mad at Netflix for forcing equipment upgrades faster than budgeted. Should Netflix have to pay for SK Telecom sucking at traffic planning and budgeting? - Jared

3 2

Re: S.Korea broadband firm sues Netflix after traffic surge
by Jared Brown 12 Oct '21

12 Oct '21

Mark Tinka wrote: >> Well, yes. Or you could just stream content that is guaranteed to be compatible with the device used. > > People on this list would bother to check compatibility. > > Jane + Thatho just point & click. Since we aren't talking about random pirated content, but p2p streaming from a major content provider it would obviously be point & click. - Jared

2 1

ARIN BoT ballot petition underway..
by Ron da Silva 11 Oct '21

11 Oct '21

Just bringing to attention of the broader audience here, and since aria-discuss no longer exists there’s also been some discussion on ppml... Dan Alexandar and I are both exercising the petition process to be added to the ballot for the ARIN Board of Trustees election. There are two seats up and only 3 candidates on the ballot produced by the nominating committee. If you are the DMR (or know the DMR for your org), please support by simply logging into your ARIN account and following the “vote now” link. https://www.arin.net/announcements/20211004_petitions/?fbclid=IwAR1yPdai7ZF… <https://www.arin.net/announcements/20211004_petitions/?fbclid=IwAR1yPdai7ZF…> Thanks in advance! -ron

1 0