March 2020 - Test - lists.nanog.org

Fwd: Your message to NANOG awaits moderator approval
by Alexandre Petrescu 16 Mar '20

16 Mar '20

fyi my figure of the map was cut by mailman software at nanog -------- Message transféré -------- Sujet : Your message to NANOG awaits moderator approval Date : Mon, 16 Mar 2020 20:37:58 +0000 De : nanog-owner(a)nanog.org Pour : alexandre.petrescu(a)gmail.com Your mail to 'NANOG' with the subject Re: COVID-19 vs. our Networks Is being held until the list moderator can review it for approval. The reason it is being held: Message body is too big: 288150 bytes with a limit of 100 KB Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL: https://mailman.nanog.org/mailman/confirm/nanog/546fda0d60c3e64480862eb9af6…

1 0

Re: COVID-19 vs. our Networks
by Clayton Zekelman 16 Mar '20

16 Mar '20

The access facility and the underlying long haul are telecommunications services. The application provided using that facility may or may not be. The congestion you were experiencing was not with the telecommunications facility itself, but with the application running on it, and was as you state, outside of your network - on a CDN hosted service. Your argument is with your third party hosted service. Their argument is with their CDN. Internet exchange points are not regulated. Interconnections between ISPs and CDNs are private agreements, and are always at risk of congestion and commercial dispute between the parties. There is a long history of this. If you have a direct layer 2 or 3 private line to your hosted service provider's CDN, and it was not performing as per the SLA, then you need to take that up with them. If the underlying telecommunications facility failed, and was classified as critical infrastructure, and not restored in a timely manner, then you need to take that up with the provider of that infrastructure. I'm not trying to be difficult, but the fact remains that there is a distinction between telecommunications services, and Internet services. The fact that Internet services (and I'm not talking about any one particular DIA circuit, but the rather the global network of networks) work so well most of the time, such that people tend to start treating it as a substitute for telecommunications services is pretty impressive. There are cost/benefit tradeoffs for using cloud hosted services and public Internet infrastructure. You save money by not operating the data centre yourself, but you pay for it in reliability. Your organization may have made that choice, but to say that because you chose to put critical applications on Internet infrastructure, other users of the Internet should take a back seat to your needs seems to be a bit of a stretch. Again, if your provider sold this to you as something that was NOT relying on public Internet, and was Layer 2/3 private managed services with dedicated bandwidth, then you need to have a conversation with them. At 02:01 PM 14/03/2020, Mike Bolitho wrote: >First of all, we use a mixture of layer 2/3 >private lines and DIA circuits. You don't know >our infrastructure, stop being condescending. It >goes against the spirit of this mailing list. > >Second, yes, the Internet is protected. Both >public and private lines. I know this because we >have TSP coded circuits and I spent four years >at a Tier I ISP servicing TSP coded circuits > >Third, the trouble we had was a third party >service having congestion issues. They are >hosted by the same CDN as Call of Duty. The >problem was both outside of our control and our >third party service'sÂ control. The chokepoint >was between ISPs/IXPs and the CDN. I've seen >this time and time again while working at the >aforementionedÂ ISP. Saturated links on >ISP/IXP/CDN networks. This is where the TSP code >comes in. In this day and age of cloud services, >it is financially unfeasible for every company >to have a private line to every single cloudÂ >provider. That's preposterousÂ to even suggest. > >- Mike Bolitho > > >On Sat, Mar 14, 2020 at 10:40 AM Clayton >Zekelman <<mailto:clayton@mnsi.net>clayton(a)mnsi.net> wrote: > > >The Internet is not a telecommunications >service, according to your FCC.Â If you want >predictability, buy WAN circuits, not Internet >circuits.Â Â If your provider is co-mingling >Internet and WAN traffic (i.e. circuits with >defined endpoints vs. public Internet or VPN), >then you need to talk to them about their prioritization. > >If you have mission critical applications, put >them on mission critical infrastructure, not the public Internet. > >Oh, that's right - Internet circuits are cheaper than WAN circuits > -- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409

4 3

access to sealandqualityfoods.com from Calgary
by David Charlebois 16 Mar '20

16 Mar '20

Hi Anyone in the Calgary area able to do a quick test? I'm troubleshooting access to sealandqualityfoods.com (resolving as 34.95.13.225 for me). Site is fine from where I am in Ontario, but reported down in Calgary. I do not know what, if any errror messages are being seen from a browser in Calgary. Just looking for more details is the server responding, is there are routing issue, etc. I'm reviewing some BGP looking-glass output now to determine if there is evidence of an issues Thanks in advance. Dave

1 0

Re: backtracking forged packets?
by konrad＠zemek.io 16 Mar '20

16 Mar '20

On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote: > I don't recommend filtering the SYN-ACK packets. That's what Octolus did, > and the result was leaving half-open SYN_RECV connections on all the nodes > used for reflection. That has two downsides: > > - the reflectors will retry the SYN-ACK (several times), which increases > your PPS load (amplifying the attack) > - the providers may notice the large number of SYN_RECV connections from > your network and put you on a blacklist I work at Path Network; we're providing DDoS mitigation services for Octolus. I wanted to follow up your message with a couple of points from our POV. The filter we have running for Octolus is a more generic stateful TCP filter, designed primarily to curb spoofed TCP floods. The bigger attacks we see are on the magnitude of 100s of Gbps/Mpps, and a simple fact is that outright dropping an out-of-state packet is multiple times less expensive for us than creating and responding with a RST. We see spoofed TCP attacks much more frequently than a SYN-ACK reflection, and in fact from our automation's point of view this looks exactly like a SYN-ACK flood from randomized source addresses. Aside from the resources cost, sending back RSTs for non-reflected attacks of respectable volume might also not be appreciated by some networks. Knowing the specifics of this long-running reflection attacks, we're considering how to reply with RSTs so to not leave lingering SYN_RECVs on reflectors' side. On 2020-03-15 16:36, Jean | ddostest.me <jean at ddostest.me> wrote: > I believe that Oculus blocked the RST and not the SYN/ACK. The SYN-ACKs are dropped; letting them reach the end servers and dropping outgoing RSTs instead would make for poor mitigation. :) On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote (cont.): > I don't want to leave you with the impression that it's hopeless... these > attacks aren't impossible to stop --- it just requires convincing the > transit providers to care. > > Damian > > On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me via NANOG < > nanog at nanog.org> wrote: > > Hi Bill, > > thanks for sharing the data. Indeed, I can't offer you a way to backtrack > the spoofed packets. > > Anyway, I'm not sure what could you do legally as there is a very high > chance that these people are not in the USA and the CFAA won't apply to > them. > > Here is what I would do if I was in your situation. > > Since these packets are spoof and malformed, I would block all SYN/ACK > based on the length. > > Depending on your hardware, it's very easy to inspect *only the SYN/ACK > by length* if you have modern hardware. On linux/unix, it's also very > straightforward. I'm not sure for windows though. > > Here is the details of the analysis: > > Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, > SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be > others, but let's use the basic one. > > In your case, there are none. There is only MSS and the SYN length is 44 > bytes. These are spoof packets maybe generated by either TCP-AMP like > reported earlier. > > I would try to block all SYN/ACK coming toward your network with a length > of 44 bytes or lower. But, this is weird because it should be 54 bytes. > Maybe there is some offloading of some sort in your gear. > > Now depending on your hardware, it could work or it could kill your router > as it will punt the cpu. I guess you have some modern gear. > > What I do when I am not sure about the length, I start to accept and log > at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins. > > Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower > than X bytes. It won't kill or block anything legitimate if you do it > properly. :) > > What will happen is that you will not reply to these spoof SYN/ACK with a > RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your > service providers will be happy and should not penalize you. > > I'm pretty sure that it will help you as it did for me in the past. > > Let me know if it's not clear and/or which part is foggy and I'll try to > give more details and better explanation. > > Regards, > > Jean St-Laurent > On 2020-03-14 11:46, William Herrin wrote: > > On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me via NANOG<nanog at nanog.org> <nanog at nanog.org> wrote: > > can you post some forged packets please? You can send them offlist if > you prefer. > > Hi Jean, > > Here are a couple examples (PDT this morning): > > 08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none], > proto ICMP (1), length 56) > 45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable - > admin prohibited filter, length 36 > IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP > (6), length 40) > 199.33.225.218.9851 > 45.89.93.26.443: [|tcp] > 0x0000: 4500 0038 277c 0000 3701 28da 2d59 5d1a > 0x0010: c721 e1da 030d 4b61 0000 0000 4500 0028 > 0x0020: 277c 4000 4506 dae4 c721 e1da 2d59 5d1a > 0x0030: 267b 01bb a057 e903 > > 08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto > TCP (6), length 44) > 104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a > (correct), seq 1216155085, ack 11765035, win 29200, options [mss > 1156], length 0 > 0x0000: 4500 002c 0000 4000 3606 e564 6857 4e5f > 0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b > 0x0020: 6012 7210 c97a 0000 0204 0484 > > I have observed no consistency in the remote IP addresses. I receive > no more than a few of each and they don't line up with particular > networks. Remote ports are heavily 80, 443, 22, 25, etc. but a > smattering of less common ports too. I'm not seeing any RSTs at all > nor any port-unreachables. Lots of syn/acks and a few time exceeded > and host unreachables. I don't know what to make of that. > > > On Sat, Mar 14, 2020 at 1:46 AM Andrew Smith<andrew.william.smith at gmail.com> <andrew.william.smith at gmail.com> wrote: > > Look inside the ICMP Unreachable backscatter at the truncated original packet that caused the unreachable message. > > Clever! I wouldn't have thought of that. Unfortunately as in the > example above, the TTLs in the packets encapsulated in ICMP are not > especially close to one of the common boundaries. > > Regards, > Bill Herrin > > -- > William Herrinbill at herrin.ushttps://bill.herrin.us/ > >

1 0

COVID-19 vs. our Networks
by g＠1337.io 16 Mar '20

16 Mar '20

With talk of there being an involuntary statewide (WA) and then national quarantines (house arrest) for multiple weeks, has anyone put thought into the impacts of this on your networks if/when this comes to fruition? We're already pushing the limits with telecommuters / those that are WFH, but I can only imagine what things will look like with everyone stuck at home for any duration of time.

27 33

WFH: Measuring and fixing latency congestion and slow problems
by Sean Donelan 14 Mar '20

14 Mar '20

Trying to put together a one-pager to help I.T. people at organizations with a surge of work from home users. First, I get that ISPs customer care centers and network operation centers are trying their best. And may be having their own staffing issues. Snow days and hurricanes aren't nationwide (at least for the USA). No specific ISPs, but generally these users are connected to the major corporate and residential ISPs. Looking for construction suggestions, besides complaining to the ISP customer care centers. Hearing about Work From Home experiences the last couple of days from a few dozen large organizations. The most common complaint is the dreaded indeterminate "Its slow." Heard complaints about both cloud applications and classic corporate VPN access. Most of the users are non-technical professionals, e.g. lawyers, business people, etc. with better than average residential broadband services (50Mbps to 150Mbps coax, vdsl, fios, etc). But not extreme broadband (not 1Gbps fiber). What suggestions for diagnosing "slow" complaints with Classic corporate VPN Online business apps (brand names as examples) Google Docs and Sheets Office365 Outlook and Word What things can users and I.T. people check first? How to figure out which provider to complain too? And rule out providers that aren't the problem.

1 0

Fwd: WARNING!!! WARNING!!! WARNING!!!
by nanog08＠mulligan.org 14 Mar '20

14 Mar '20

Well - this is new. I have gotten plenty of the "we have broken into your computer (or account)", but a Death threat? And where is the bitcoin address to send my $100K. Maybe I should start a gofundme to get the money. Geoff PS - I did send this to aol abuse (not that it will do ANY good) -------- Forwarded Message -------- Subject: WARNING!!! WARNING!!! WARNING!!! Date: Sat, 14 Mar 2020 05:28:26 -0700 From: DRAGON BLADE <luffy(a)emailtool-seller100.xyz> Reply-To: dragon000blade.org(a)aol.com To: geoff(a)proto6.com I have to warn you someone has paid me to kill you..I've being monitoring you for 4days now...i think you almost noticed me 2days ago...LISTEN...You have less than, 48hours to pay $100,000USD...!! i see you as a nice person if not for the people you keep around... that is why i am doing this! I'm giving you a chance to escape or else I finish my job but I can move and run away!! If you think $100,000USD is too much for your life, very well then!!! I assure you,after taking this risk by telling you my assignment and having compassion on you at first because of GOD, if you act like you are smart?? I will cut you all piece by piece to make you die very painfully...I assure you****** Good-luck! NOTE: I'm Russian,if you like you can inform the police or anybody, death is promised and this time your precious family members will be involved i assure you!! There' s already a chip and your calls and all is being monitored i have to let you know!! You must have to send me $100,000USD as soon as possible before 48hours or else I cannot be sorry for you!! I await your reply {if you like} for payment information!!! But you can try to dare me (us)!!! GOODLUCK!!! E-MAIL:: dragon000blade.org(a)aol.com

2 1

Re: WIKI documentation Software?
by Craig 14 Mar '20

14 Mar '20

Lol, Sharepoint,,,. Arggg, yea NOT going to happen , We’ve managed to avoid using that. On Sat, Mar 14, 2020 at 10:50 AM Nicholas Oas <nicholas.oas(a)gmail.com> wrote: > Seconding Confluence. Stay away from Sharepoint. > > On Sat, Mar 14, 2020 at 8:09 AM Craig <cvuljanic(a)gmail.com> wrote: > >> Wanted to ask what WIKI software teams are using to save documentation to >> / how to's for staff, etc. >> >> pro's >> con's >> >> We have an older wiki bare-metal wiki server, that I want to get replaced >> before it kicks the bucket and was looking into various ones. >> >> thanks; >> >> CPV >> >> >>

3 2

Work from Home and other dynamics
by Jared Mauch 14 Mar '20

14 Mar '20

I’m wondering what general trends people have seen with the recent reduction in travel and increased work from home activities. I’m expecting that a number of networks are seeing increased traffic demand. Enterprises are likely adding VPN licenses for staff that are now remote, etc.. I would expect increase (and decreases) similar to weekend traffic patterns. I’m expecting there will be more IPv6 traffic similar to what is seen during the Christmas/New Years holiday on this traffic as well: https://www.google.com/intl/en/ipv6/statistics.html What interesting dynamics are you seeing? - Jared

6 5

oinky attribute
by Randy Bush 13 Mar '20

13 Mar '20

Prefix Send failed ! 103.199.169.0/24 bgp_rt_trace_too_big_message:1209 path attribute too big. Cannot build update. lovely randy

1 0