- Test - lists.nanog.org

interesting troubleshooting
by Nimrod Levy 24 Mar '20

24 Mar '20

I just ran into an issue that I thought was worth sharing with the NANOG community. With recently increased visibility on keeping the Internet running smoothly, I thought that sharing this small experience could benefit everyone. I was contacted by my NOC to investigate a LAG that was not distributing traffic evenly among the members to the point where one member was congested while the utilization on the LAG was reasonably low. Looking at my netflow data, I was able to confirm that this was caused by a single large flow of ESP traffic. Fortunately, I was able to shift this flow to another path that had enough headroom available so that the flow could be accommodated on a single member link. With the increase in remote workers and VPN traffic that won't hash across multiple paths, I thought this anecdote might help someone else track down a problem that might not be so obvious. Please take this message in the spirit in which it was intended and refrain from the snarky "just upgrade you links" comments. -- Nimrod

14 27

Re: COVID-19 vs. our Networks
by Clayton Zekelman 24 Mar '20

24 Mar '20

Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. Who determines what is "mission critical"? Our mission as an ISP is to deliver Internet to our customers. If they want to play online games or watch video, who am I to say that isn't critical to THEIR mission? The last thing we need are a bunch of kids in quarantine that have NOTHING to do because Mike Bolitho thinks their entertainment isn't part of the "mission" of the Internet. About the only thing that might be useful is something to smooth out the big jumps in utilization on game releases - but even that is something that can be managed by adding capacity. To quote Jay Leno - Crunch All You Want, We'll Make More. At 12:16 PM 14/03/2020, Mike Bolitho wrote: >Basically that. It's probably more streaming >services that could crowd out what would be >considered "mission critical" infrastructure. >Maybe the Netflixs and HulusÂ of the world will >limit 4K streaming or something along those >lines. Basically cap resolution to 720p for the time being. > >- Mike Bolitho > > >On Sat, Mar 14, 2020 at 1:06 AM Hugo Slabbert ><<mailto:hugo@slabnet.com>hugo(a)slabnet.com> wrote: > >Â The impact of all these bored school kids on > the networks due to gaming might cause some > issues. I know that if I'm working from home > and my videoconferencing slows down because of > someones gaming, I'm taking the necessary > action (read, change some rules on my firewall).Â > >People are welcome to do whatever they want on >their own networks. I just didn't get the >suggestion that online gaming services would >shut down. Or were you saying, Mike, that online >gaming would crowd out other services and so "shut down" those other services? > >On Fri., Mar. 13, 2020, 21:42 Owen DeLong ><<mailto:owen@delong.com>owen(a)delong.com> wrote: >You donât have kids, do you > >They have the attention span off Koi these days. >Theyâll play most games for about 15 minutes >or so before downloading the next one. (At least >thatâs been my observation of behavior among >my GFâs daughter and her friends). > >Owen > > >>On Mar 13, 2020, at 20:31 , Darin Steffl >><<mailto:darin.steffl@mnwifi.com>darin.steffl(a)mnwifi.com> wrote: >> >>Playing games doesn't take much bandwidth. >>Downloading games does. So as long as everyone >>already has their games and there's no updates, >>playing the game is typically under 100 kbps >>which is negligible compared to streaming video which takes 1 to 25 mbps.Â >> >>On Fri, Mar 13, 2020, 8:52 PM Sabri Berisha >><<mailto:sabri@cluecentral.net>sabri(a)cluecentral.net> wrote: >>Hi, >> >>I don't know where y'all live, but here in the >>SF Bay Area, pretty much all public and private >>schools have closed down. My school district >>(in Santa Clara County) will be closed until Spring Break. >> >>The impact of all these bored school kids on >>the networks due to gaming might cause some >>issues. I know that if I'm working from home >>and my videoconferencing slows down because of >>someones gaming, I'm taking the necessary >>action (read, change some rules on my firewall).Â >> >>Thanks, >> >>Sabri >> >> >>----- On Mar 13, 2020, at 4:12 PM, Hugo >>Slabbert <<mailto:hugo@slabnet.com>hugo(a)slabnet.com> wrote: >>I think under circumstances like this, I could >>definitely see some of the online based games shutting services down. >> >>Â >>How so? >> >>Signed, >> >>Someone who works for an online gaming company and has heard nothing of this. >> >>--Â >>Hugo SlabbertÂ Â Â Â | email, xmpp/jabber: >><mailto:hugo@slabnet.com>hugo(a)slabnet.com >>pgp key: B178313EÂ Â | also on Signal >> >> >>On Fri, Mar 13, 2020 at 2:52 PM Mike Bolitho >><<mailto:mikebolitho@gmail.com>mikebolitho(a)gmail.com> wrote: >>I think under circumstances like this, I could >>definitely see some of the online based games shutting services down. >> >>- Mike Bolitho >> >> >>On Fri, Mar 13, 2020 at 2:41 PM Ahmed Borno >><<mailto:amaged@gmail.com>amaged(a)gmail.com> wrote: >>Its already happening in Italy, and now that >>schools are shutting down here as well, its going to get interesting:Â >><https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-…>https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-… >> >>The ultimate traffic test is coming, looking >>forward to hearing about it on this thread. >> >>Maybe its a good time to start a communication >>channel between content providers/gaming companies and ISPs/CDNs. >> >> >>On Fri, Mar 13, 2020 at 11:22 AM Rubens Kuhl >><<mailto:rubensk@gmail.com>rubensk(a)gmail.com> wrote: >> >> >>On Thu, Mar 12, 2020 at 3:46 PM >><mailto:g@1337.io>g(a)1337.io <<mailto:lists@1337.io>lists(a)1337.io> wrote: >>With talk of there being an involuntary >>statewide (WA) and then national quarantines >>(house arrest) for multiple weeks, has anyone >>put thought into the impacts of this on your >>networks if/when this comes to fruition? >> >>We're already pushing the limits with >>telecommuters / those that are WFH, but I can >>only imagine what things will look like with >>everyone stuck at home for any duration of time. >> >> >> >>People will turn to you and every other ISP >>hoping you keep them online. So besides demand >>issues, keeping your network up will be important to a whole lot of people.Â >> >> >>Rubens >>Â >> -- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409

23 57

California full facilities CLEC
by Ben Cannon 24 Mar '20

24 Mar '20

Need a small yet large COVID traffic load related favor from a California Full-Facilities CLEC/CLC, pls contact me off list if you can help. Connecting at-risk citizens. -Ben

1 0

Fwd: Internet operations during pandemics
by Christopher Morrow 23 Mar '20

23 Mar '20

Did other folk on nanog-l see the nLnog-l note copied here? I wonder how folk are planning for things (noted in the slides) o supply chain for parts/equipment Wait, I can't get me a new shiny shipped because what?? o ongoing rollout of new equipment I'm deploying next week in KIX, I'm currently in LAX how do I get there? equipment arrives.. in between...oops! o noc/etc support staff omg.. wait, I can't have my noc staff in the same room? our 'wfh' solution is ... wait, where is that? how do i get their phone queue sent to them? omg :( <sadness!> o services capacity crunches I love my shiny new dns service.. .wait, why is there a smoking hole where my dns servers were? I think some of this has been discussed (shifts in peaks, leveling of peaks) Some hasn't really... I expect that at least sharing some 'err, our WFH changed now we do: X, Y , Z and use M to get N solved' could be super cool to discuss/share and iterate for better solutions for all of our users. thoughts? :) thanks! -chris (note all the hard work in this message is not mine... thanks Job!) ---------- Forwarded message --------- From: Job Snijders <job(a)ntt.net> Date: Wed, Mar 18, 2020 at 6:02 PM Subject: Internet operations during pandemics To: <nlnog(a)nlnog.net> Dear all, I threw together a slidedeck today on the potential impact and second order effects of COVID-19 on Internet network operations. http://instituut.net/~job/netops_during_pandemics.pdf I hope we together over time can add and extend projections in the deck on what will happen and how we can mitigate the negative effects on Internet operations. We have to answer questions such as: 1) what problems already exist today because of a few weeks of C19? 2) What problems are still coming? Will those be localized or globally? 3) What possible workarounds can we plan for those problems? I would appreciate feedback, comments, corrections or whatever you want to tell me. None of us have been in this situation before, so my guess is as good as yours. Kind regards, Job

4 5

Salesforce Peering / Network Engineer
by Louis D 23 Mar '20

23 Mar '20

Hello Nanog, If there is someone on list from Salesforce Network Engineering please reply to me off list about an issue with reaching Salesforce the DE-CIX in New York. Thanks, Lou

1 0

Frontier Pennsylvania
by Matt Hoppes 23 Mar '20

23 Mar '20

Does anyone have a contact for Frontier Central PA OSP contact? There is a line that has been down for over 8 months that I have been unable to get them to hang. It is across a driveway and roadway.

2 2

Sunday traffic curiosity
by Andy Ringsmuth 23 Mar '20

23 Mar '20

Fellow NANOGers, Not a big deal by any means, but for those of you who have traffic data, I’m curious what Sunday morning looked like as compared to other Sundays. Sure, Netflix and similar companies have no doubt seen traffic increase, but I’m wondering if an influx of church service streaming was substantial enough to cause a noticeable traffic increase. We livestream our services and have been for about a year or so, but normally average just a handful of viewers. Today, we were around 150 watching live. ---- Andy Ringsmuth andy(a)andyring.com

5 4

Re: Sunday traffic curiosity
by John Kristoff 22 Mar '20

22 Mar '20

On Sun, 22 Mar 2020 19:17:59 +0000 Grant Taylor via NANOG <nanog(a)nanog.org> wrote: > What was wrong with Internet scale multicast? Why did it get abandoned? There are about 20 years of archives to weed through, and some of our friends are still trying to make this happen. I expect someone (Hi Lenny) to appear any moment and mention AMT. So my take isn't universally accepted, but it won't be too far from what you'll hear from many. Brief summary off the top of my head: 1. Complexity. Both in protocol mechanisms and the requirements in network devices (i.e. snooping, state, troubleshooting). 2. Security. Driven in part by #1, threats abound. SSM can eliminate some of this and you can design a receiver-only model that removes most of the remaining problems - congratulations you just reinvented over the air broadcast TV. Even if you don't do interdomain IP multicast, you still may be at risk: <https://ccronline.sigcomm.org/wp-content/uploads/2017/01/p27-sargent.pdf> 3. Need and business drivers. Still far from compelling to build and support all this to make it worthwhile for all but a few niche environments. Support and expertise in this area is also very thin. Your inquiry demonstrates this. I stopped teaching it to students. What remains is becoming even less well supported than it has been. There is almost no interdomain IP multicast monitoring being done anymore. There is scant actual content being delivered, all the once popular stuff is gone. The number of engineers who know this stuff are dwindling and some that do know something about it are removing at least some parts of it: <https://tools.ietf.org/html/draft-ietf-mboned-deprecate-interdomain-asm-07> John

2 1

China’s Slow Transnational Network
by Pengxiong Zhu 22 Mar '20

22 Mar '20

Hi all, We are a group of researchers at University of California, Riverside who have been working on measuring the transnational network performance (and have previously asked questions on the mailing list). Our work has now led to a publication in Sigmetrics 2020 and we are eager to share some interesting findings. We find China's transnational networks have extremely poor performance when accessing foreign sites, where the throughput is often persistently low (e.g., for the majority of the daytime). Compared to other countries we measured including both developed and developing, China's transnational network performance is among the worst (comparable and even worse than some African countries). Measuring from more than 400 pairs of mainland China and foreign nodes over more than 53 days, our result shows when data transferring from foreign nodes to China, 79% of measured connections has throughput lower than the 1Mbps, sometimes it is even much lower. The slow speed occurs only during certain times and forms a diurnal pattern that resembles congestion (irrespective of network protocol and content), please see the following figure. The diurnal pattern is fairly stable, 80% to 95% of the transnational connections have a less than 3 hours standard deviation of the slowdown hours each day over the entire duration. However, the speed rises up from 1Mbps to 4Mbps in about half an hour. We are able to confirm that high packet loss rates and delays are incurred in the foreign-to-China direction only. Moreover, the end-to-end loss rate could rise up to 40% during the slow period, with ~15% on average. There are a few things noteworthy regarding the phenomenon. First of all, all traffic types are treated equally, HTTP(S), VPN, etc., which means it is discriminating or differentiating any specific kinds of traffic. Second, we found for 71% of connections, the bottleneck is located inside China (the second hop after entering China or further), which means that it is mostly unrelated to the transnational link itself (e.g., submarine cable). Yet we never observed any such domestic traffic slowdowns within China. Assuming this is due to congestion, it is unclear why the infrastructures within China that handles transnational traffic is not even capable to handle the capacity of transnational links, e.g., submarine cable, which maybe the most expensive investment themselves. Here is the link to our paper: https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf We appreciate any comments or feedback. -- Best, Pengxiong Zhu Department of Computer Science and Engineering University of California, Riverside

11 25

Fwd: Re: COVID-19 vs. our Networks
by Alexandre Petrescu 21 Mar '20

21 Mar '20

(photo removed, the admins have it, dont ask me in private) -------- Message transféré -------- Sujet : Re: COVID-19 vs. our Networks Date : Sat, 21 Mar 2020 14:20:56 +0100 De : Alexandre Petrescu <alexandre.petrescu(a)gmail.com> Pour : nanog(a)nanog.org LF/HF Le 21/03/2020 à 12:28, Florian Weimer a écrit : > * Mike Hammett: > > [...] > Relaxing copyright and patent restrictions might also help, at least > in the medium term. I agree. Related to copyright and patent restrictions, is this: Currently the situation is that I cant get to see the ARN (acid rybo nucleic) sequence of this virus; the data is present on gisaid.org, situated in Germany, at MAx Planck Institute, but my request to see it did not succeed - silence. FRiends tell me that silence is normal because I am not a specialist, and those who are specialist do have access. I disagree. I want to know what it looks like. This virus with a crown might hurt me as much as it might hurt the specialists, there is no difference, we are all Sapiens. I want to know what it looks like. The only thing I could find is during a TV news report, see below. That sequence of A, T, G, C, and their particular ordering, is what makes it bad bad. That's the enemy. Alex, LF/HF 2 (it means low stress)

1 0