- NANOG - lists.nanog.org

RE: Backhoe season?
by Aaron Gould 27 Mar '20

27 Mar '20

Yeah Darron, we lost some san Antonio connectivity to Houston via dallas or somewhere.... twice in the past few days, affecting different things for us -Aaron -----Original Message----- From: Darron Legnon [mailto:darron@commZoom.com] Sent: Thursday, March 26, 2020 1:11 PM To: Aaron Gould; 'William Herrin'; nanog(a)nanog.org Subject: RE: Backhoe season? I've had contractors for Zayo hitting multiple fiber routes over the past 3 months numerous times affecting us, AT&T, CenturyLink, Windstream, Fiberlight, etc in south Texas. Actually had one cut yesterday around 3pm and lasted till 1am. -----Original Message----- From: NANOG <nanog-bounces(a)nanog.org> On Behalf Of Aaron Gould Sent: Thursday, March 26, 2020 1:03 PM To: 'William Herrin' <bill(a)herrin.us>; nanog(a)nanog.org Subject: RE: Backhoe season? CAUTION: This email originated from outside your organization. Exercise caution when opening attachments or clicking links, especially from unknown senders. I heard, and am seeing that construction type jobs don't seem to be affected much with the virus shutdown. I mean I see guys building homes and working on roads all around me... furthermore, we've heard of a couple fiber cuts that have brought portions of our network down a couple times in the last week or so. -Aaron -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of William Herrin Sent: Thursday, March 26, 2020 12:57 PM To: nanog(a)nanog.org Subject: Backhoe season? Howdy, With so much work shut down, I'm curious how backhoe season is shaping up this year? How do the circuit and fiber outage numbers look? Regards, Bill Herrin -- William Herrin bill(a)herrin.us https://bill.herrin.us/ This e-mail and any files transmitted with it are the property of COMMZOOM, are confidential, and are intended solely for the use of the individual or entity to which this e-mail is addressed. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited.

4 3

CISA critical infrastructure letters
by Sean Donelan 25 Mar '20

25 Mar '20

The CISA critical infrastructure letters are a courtesy request letter. If people abuse its purpose, local officials do not need to extend any courtesy and can deny access. The CISA letter is only for "providing emergency communications sustainment and restoration support to critical communications infrastructure facilities." It is NOT a general purpose, ignore anything or go anywhere letter. Do NOT abuse the courtesy or no one will extend the courtesy.

4 4

Re: CISA critical infrastructure letters
by Scott Weeks 25 Mar '20

25 Mar '20

I got these. One each for travel and fuel. I could fake one in 15 minutes or so. Heck, I could probable find one online and modify it in less time than that! Because of that I don't see the usefulness. scott

4 3

Re: free collaborative tools for low BW and losy connections
by Scott Weeks 25 Mar '20

25 Mar '20

Thanks, my facepalm moment of the day (so far; it's only 7:30am here) is... Use tools from the past when the connections everywhere were losy and slow. They already mentioned RT. I'll mention that and NNTP/UUCP/etc. scott

1 0

interesting troubleshooting
by Nimrod Levy 24 Mar '20

24 Mar '20

I just ran into an issue that I thought was worth sharing with the NANOG community. With recently increased visibility on keeping the Internet running smoothly, I thought that sharing this small experience could benefit everyone. I was contacted by my NOC to investigate a LAG that was not distributing traffic evenly among the members to the point where one member was congested while the utilization on the LAG was reasonably low. Looking at my netflow data, I was able to confirm that this was caused by a single large flow of ESP traffic. Fortunately, I was able to shift this flow to another path that had enough headroom available so that the flow could be accommodated on a single member link. With the increase in remote workers and VPN traffic that won't hash across multiple paths, I thought this anecdote might help someone else track down a problem that might not be so obvious. Please take this message in the spirit in which it was intended and refrain from the snarky "just upgrade you links" comments. -- Nimrod

14 27

Re: COVID-19 vs. our Networks
by Clayton Zekelman 24 Mar '20

24 Mar '20

Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. Who determines what is "mission critical"? Our mission as an ISP is to deliver Internet to our customers. If they want to play online games or watch video, who am I to say that isn't critical to THEIR mission? The last thing we need are a bunch of kids in quarantine that have NOTHING to do because Mike Bolitho thinks their entertainment isn't part of the "mission" of the Internet. About the only thing that might be useful is something to smooth out the big jumps in utilization on game releases - but even that is something that can be managed by adding capacity. To quote Jay Leno - Crunch All You Want, We'll Make More. At 12:16 PM 14/03/2020, Mike Bolitho wrote: >Basically that. It's probably more streaming >services that could crowd out what would be >considered "mission critical" infrastructure. >Maybe the Netflixs and HulusÂ of the world will >limit 4K streaming or something along those >lines. Basically cap resolution to 720p for the time being. > >- Mike Bolitho > > >On Sat, Mar 14, 2020 at 1:06 AM Hugo Slabbert ><<mailto:hugo@slabnet.com>hugo(a)slabnet.com> wrote: > >Â The impact of all these bored school kids on > the networks due to gaming might cause some > issues. I know that if I'm working from home > and my videoconferencing slows down because of > someones gaming, I'm taking the necessary > action (read, change some rules on my firewall).Â > >People are welcome to do whatever they want on >their own networks. I just didn't get the >suggestion that online gaming services would >shut down. Or were you saying, Mike, that online >gaming would crowd out other services and so "shut down" those other services? > >On Fri., Mar. 13, 2020, 21:42 Owen DeLong ><<mailto:owen@delong.com>owen(a)delong.com> wrote: >You donât have kids, do you > >They have the attention span off Koi these days. >Theyâll play most games for about 15 minutes >or so before downloading the next one. (At least >thatâs been my observation of behavior among >my GFâs daughter and her friends). > >Owen > > >>On Mar 13, 2020, at 20:31 , Darin Steffl >><<mailto:darin.steffl@mnwifi.com>darin.steffl(a)mnwifi.com> wrote: >> >>Playing games doesn't take much bandwidth. >>Downloading games does. So as long as everyone >>already has their games and there's no updates, >>playing the game is typically under 100 kbps >>which is negligible compared to streaming video which takes 1 to 25 mbps.Â >> >>On Fri, Mar 13, 2020, 8:52 PM Sabri Berisha >><<mailto:sabri@cluecentral.net>sabri(a)cluecentral.net> wrote: >>Hi, >> >>I don't know where y'all live, but here in the >>SF Bay Area, pretty much all public and private >>schools have closed down. My school district >>(in Santa Clara County) will be closed until Spring Break. >> >>The impact of all these bored school kids on >>the networks due to gaming might cause some >>issues. I know that if I'm working from home >>and my videoconferencing slows down because of >>someones gaming, I'm taking the necessary >>action (read, change some rules on my firewall).Â >> >>Thanks, >> >>Sabri >> >> >>----- On Mar 13, 2020, at 4:12 PM, Hugo >>Slabbert <<mailto:hugo@slabnet.com>hugo(a)slabnet.com> wrote: >>I think under circumstances like this, I could >>definitely see some of the online based games shutting services down. >> >>Â >>How so? >> >>Signed, >> >>Someone who works for an online gaming company and has heard nothing of this. >> >>--Â >>Hugo SlabbertÂ Â Â Â | email, xmpp/jabber: >><mailto:hugo@slabnet.com>hugo(a)slabnet.com >>pgp key: B178313EÂ Â | also on Signal >> >> >>On Fri, Mar 13, 2020 at 2:52 PM Mike Bolitho >><<mailto:mikebolitho@gmail.com>mikebolitho(a)gmail.com> wrote: >>I think under circumstances like this, I could >>definitely see some of the online based games shutting services down. >> >>- Mike Bolitho >> >> >>On Fri, Mar 13, 2020 at 2:41 PM Ahmed Borno >><<mailto:amaged@gmail.com>amaged(a)gmail.com> wrote: >>Its already happening in Italy, and now that >>schools are shutting down here as well, its going to get interesting:Â >><https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-…>https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-… >> >>The ultimate traffic test is coming, looking >>forward to hearing about it on this thread. >> >>Maybe its a good time to start a communication >>channel between content providers/gaming companies and ISPs/CDNs. >> >> >>On Fri, Mar 13, 2020 at 11:22 AM Rubens Kuhl >><<mailto:rubensk@gmail.com>rubensk(a)gmail.com> wrote: >> >> >>On Thu, Mar 12, 2020 at 3:46 PM >><mailto:g@1337.io>g(a)1337.io <<mailto:lists@1337.io>lists(a)1337.io> wrote: >>With talk of there being an involuntary >>statewide (WA) and then national quarantines >>(house arrest) for multiple weeks, has anyone >>put thought into the impacts of this on your >>networks if/when this comes to fruition? >> >>We're already pushing the limits with >>telecommuters / those that are WFH, but I can >>only imagine what things will look like with >>everyone stuck at home for any duration of time. >> >> >> >>People will turn to you and every other ISP >>hoping you keep them online. So besides demand >>issues, keeping your network up will be important to a whole lot of people.Â >> >> >>Rubens >>Â >> -- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409

23 57

California full facilities CLEC
by Ben Cannon 24 Mar '20

24 Mar '20

Need a small yet large COVID traffic load related favor from a California Full-Facilities CLEC/CLC, pls contact me off list if you can help. Connecting at-risk citizens. -Ben

1 0

Fwd: Internet operations during pandemics
by Christopher Morrow 23 Mar '20

23 Mar '20

Did other folk on nanog-l see the nLnog-l note copied here? I wonder how folk are planning for things (noted in the slides) o supply chain for parts/equipment Wait, I can't get me a new shiny shipped because what?? o ongoing rollout of new equipment I'm deploying next week in KIX, I'm currently in LAX how do I get there? equipment arrives.. in between...oops! o noc/etc support staff omg.. wait, I can't have my noc staff in the same room? our 'wfh' solution is ... wait, where is that? how do i get their phone queue sent to them? omg :( <sadness!> o services capacity crunches I love my shiny new dns service.. .wait, why is there a smoking hole where my dns servers were? I think some of this has been discussed (shifts in peaks, leveling of peaks) Some hasn't really... I expect that at least sharing some 'err, our WFH changed now we do: X, Y , Z and use M to get N solved' could be super cool to discuss/share and iterate for better solutions for all of our users. thoughts? :) thanks! -chris (note all the hard work in this message is not mine... thanks Job!) ---------- Forwarded message --------- From: Job Snijders <job(a)ntt.net> Date: Wed, Mar 18, 2020 at 6:02 PM Subject: Internet operations during pandemics To: <nlnog(a)nlnog.net> Dear all, I threw together a slidedeck today on the potential impact and second order effects of COVID-19 on Internet network operations. http://instituut.net/~job/netops_during_pandemics.pdf I hope we together over time can add and extend projections in the deck on what will happen and how we can mitigate the negative effects on Internet operations. We have to answer questions such as: 1) what problems already exist today because of a few weeks of C19? 2) What problems are still coming? Will those be localized or globally? 3) What possible workarounds can we plan for those problems? I would appreciate feedback, comments, corrections or whatever you want to tell me. None of us have been in this situation before, so my guess is as good as yours. Kind regards, Job

4 5

Salesforce Peering / Network Engineer
by Louis D 23 Mar '20

23 Mar '20

Hello Nanog, If there is someone on list from Salesforce Network Engineering please reply to me off list about an issue with reaching Salesforce the DE-CIX in New York. Thanks, Lou

1 0

Frontier Pennsylvania
by Matt Hoppes 23 Mar '20

23 Mar '20

Does anyone have a contact for Frontier Central PA OSP contact? There is a line that has been down for over 8 months that I have been unable to get them to hang. It is across a driveway and roadway.

2 2