- Test - lists.nanog.org

AT&T is suspending broadband data caps for home internet customers due to coronavirus
by Sean Donelan 17 Mar '20

17 Mar '20

The first data cap waiver I've seen due to coronavirus. I expect other ISPs to quickly follow. https://www.vice.com/en_us/article/v74qzb/atandt-suspends-broadband-usage-c… AT&T is the first major ISP to confirm that it will be suspending all broadband usage caps as millions of Americans bunker down in a bid to slow the rate of COVID-19 expansion. Consumer groups and a coalition of Senators are now pressuring other ISPs to follow suit.

14 16

Power outage LA 600 W 7th - TELX
by A. Pishdadi 17 Mar '20

17 Mar '20

Greetings, We got an email last night after some alerts from monitoring system. Power on the B UPS feed has been offline till this second and going. Apparently they can not bypass UPS and put the load on the generator. Which is crazy in itself being that it's a data center.. Luckily we have everything on redundant power. Does anyone from DRT or Telx subscribe here that can look into this more for me? Message me off list if possible. Thanks AP

1 0

Re: COVID-19 vs. our Networks
by Scott Weeks 17 Mar '20

17 Mar '20

--- alexandre.petrescu(a)gmail.com wrote: From: Alexandre Petrescu <alexandre.petrescu(a)gmail.com> That map does not show Texas, as far as I know America (USA) geography. ----------------------------------------------- <Texas is part of the US?> Being raised in Texas in a family that've been there for a buncha generations, I know that at least some folks there would challange that... :) https://en.wikipedia.org/wiki/Texas_secession_movements It was a nation unto itself for over decade: https://en.wikipedia.org/wiki/Republic_of_Texas Many old timers are a pretty independent type of people. scott ps. traffic is still normal here

2 1

backtracking forged packets?
by William Herrin 17 Mar '20

17 Mar '20

Howdy, Can anyone suggest tools, techniques and helpful contacts for backtracking spoofed packets? At the moment someone is forging TCP syns from my address block. I'm getting the syn/ack and icmp unreachable backscatter. Enough that my service provider briefly classified it a DDOS. I'd love to find the culprit. Thanks, Bill Herrin -- William Herrin bill(a)herrin.us https://bill.herrin.us/

10 15

sflow -> aggregated aspath visualization?
by Adam Thompson 16 Mar '20

16 Mar '20

I’m looking for product recommendations: We’ve noticed that about 20% of our traffic here lately has decamped from the free (or, at least, flat-rate) connection to CANARIE (our R&E network) and its various connected content-delivery networks, and onto our commercial provider. While this is presumptively a legitimate shift, we’d like to better understand these changes when they occur, in a way that our executive can understand at a glance. We do have sFlow (et al.) going to an Arbor PeakFlow box for analysis, but it’s lacklustre at best at understanding changes like this. I want: * Top #n ASNs by traffic volume, per router/interface, stacked chart * Some way to visualize large jumps in that dataset, e.g. if Cloudflare ditched their CANARIE connection and now that traffic all goes commercial, I don’t know what sort of graphic would be useful, maybe a stacked polar chart so you could see when an AS jumped from one sector to another? Even stacked bar charts could be useful. If anyone knows of tools capable of generating easy-to-understand reports, dashboards, including historical “what changed this week”-type data, please let me know. For that matter, if you have a technique of collecting this data and using Excel to do the reporting, that would work too. (Yes, I could theoretically build this off of existing open source tools… eventually) Thanks, -Adam Adam Thompson Consultant, Infrastructure Services [[MERLIN LOGO]]<https://www.merlin.mb.ca/> 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson(a)merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/>

5 4

Fwd: Your message to NANOG awaits moderator approval
by Alexandre Petrescu 16 Mar '20

16 Mar '20

fyi my figure of the map was cut by mailman software at nanog -------- Message transféré -------- Sujet : Your message to NANOG awaits moderator approval Date : Mon, 16 Mar 2020 20:37:58 +0000 De : nanog-owner(a)nanog.org Pour : alexandre.petrescu(a)gmail.com Your mail to 'NANOG' with the subject Re: COVID-19 vs. our Networks Is being held until the list moderator can review it for approval. The reason it is being held: Message body is too big: 288150 bytes with a limit of 100 KB Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL: https://mailman.nanog.org/mailman/confirm/nanog/546fda0d60c3e64480862eb9af6…

1 0

Re: COVID-19 vs. our Networks
by Clayton Zekelman 16 Mar '20

16 Mar '20

The access facility and the underlying long haul are telecommunications services. The application provided using that facility may or may not be. The congestion you were experiencing was not with the telecommunications facility itself, but with the application running on it, and was as you state, outside of your network - on a CDN hosted service. Your argument is with your third party hosted service. Their argument is with their CDN. Internet exchange points are not regulated. Interconnections between ISPs and CDNs are private agreements, and are always at risk of congestion and commercial dispute between the parties. There is a long history of this. If you have a direct layer 2 or 3 private line to your hosted service provider's CDN, and it was not performing as per the SLA, then you need to take that up with them. If the underlying telecommunications facility failed, and was classified as critical infrastructure, and not restored in a timely manner, then you need to take that up with the provider of that infrastructure. I'm not trying to be difficult, but the fact remains that there is a distinction between telecommunications services, and Internet services. The fact that Internet services (and I'm not talking about any one particular DIA circuit, but the rather the global network of networks) work so well most of the time, such that people tend to start treating it as a substitute for telecommunications services is pretty impressive. There are cost/benefit tradeoffs for using cloud hosted services and public Internet infrastructure. You save money by not operating the data centre yourself, but you pay for it in reliability. Your organization may have made that choice, but to say that because you chose to put critical applications on Internet infrastructure, other users of the Internet should take a back seat to your needs seems to be a bit of a stretch. Again, if your provider sold this to you as something that was NOT relying on public Internet, and was Layer 2/3 private managed services with dedicated bandwidth, then you need to have a conversation with them. At 02:01 PM 14/03/2020, Mike Bolitho wrote: >First of all, we use a mixture of layer 2/3 >private lines and DIA circuits. You don't know >our infrastructure, stop being condescending. It >goes against the spirit of this mailing list. > >Second, yes, the Internet is protected. Both >public and private lines. I know this because we >have TSP coded circuits and I spent four years >at a Tier I ISP servicing TSP coded circuits > >Third, the trouble we had was a third party >service having congestion issues. They are >hosted by the same CDN as Call of Duty. The >problem was both outside of our control and our >third party service'sÂ control. The chokepoint >was between ISPs/IXPs and the CDN. I've seen >this time and time again while working at the >aforementionedÂ ISP. Saturated links on >ISP/IXP/CDN networks. This is where the TSP code >comes in. In this day and age of cloud services, >it is financially unfeasible for every company >to have a private line to every single cloudÂ >provider. That's preposterousÂ to even suggest. > >- Mike Bolitho > > >On Sat, Mar 14, 2020 at 10:40 AM Clayton >Zekelman <<mailto:clayton@mnsi.net>clayton(a)mnsi.net> wrote: > > >The Internet is not a telecommunications >service, according to your FCC.Â If you want >predictability, buy WAN circuits, not Internet >circuits.Â Â If your provider is co-mingling >Internet and WAN traffic (i.e. circuits with >defined endpoints vs. public Internet or VPN), >then you need to talk to them about their prioritization. > >If you have mission critical applications, put >them on mission critical infrastructure, not the public Internet. > >Oh, that's right - Internet circuits are cheaper than WAN circuits > -- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409

4 3

access to sealandqualityfoods.com from Calgary
by David Charlebois 16 Mar '20

16 Mar '20

Hi Anyone in the Calgary area able to do a quick test? I'm troubleshooting access to sealandqualityfoods.com (resolving as 34.95.13.225 for me). Site is fine from where I am in Ontario, but reported down in Calgary. I do not know what, if any errror messages are being seen from a browser in Calgary. Just looking for more details is the server responding, is there are routing issue, etc. I'm reviewing some BGP looking-glass output now to determine if there is evidence of an issues Thanks in advance. Dave

1 0

Re: backtracking forged packets?
by konrad＠zemek.io 16 Mar '20

16 Mar '20

On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote: > I don't recommend filtering the SYN-ACK packets. That's what Octolus did, > and the result was leaving half-open SYN_RECV connections on all the nodes > used for reflection. That has two downsides: > > - the reflectors will retry the SYN-ACK (several times), which increases > your PPS load (amplifying the attack) > - the providers may notice the large number of SYN_RECV connections from > your network and put you on a blacklist I work at Path Network; we're providing DDoS mitigation services for Octolus. I wanted to follow up your message with a couple of points from our POV. The filter we have running for Octolus is a more generic stateful TCP filter, designed primarily to curb spoofed TCP floods. The bigger attacks we see are on the magnitude of 100s of Gbps/Mpps, and a simple fact is that outright dropping an out-of-state packet is multiple times less expensive for us than creating and responding with a RST. We see spoofed TCP attacks much more frequently than a SYN-ACK reflection, and in fact from our automation's point of view this looks exactly like a SYN-ACK flood from randomized source addresses. Aside from the resources cost, sending back RSTs for non-reflected attacks of respectable volume might also not be appreciated by some networks. Knowing the specifics of this long-running reflection attacks, we're considering how to reply with RSTs so to not leave lingering SYN_RECVs on reflectors' side. On 2020-03-15 16:36, Jean | ddostest.me <jean at ddostest.me> wrote: > I believe that Oculus blocked the RST and not the SYN/ACK. The SYN-ACKs are dropped; letting them reach the end servers and dropping outgoing RSTs instead would make for poor mitigation. :) On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote (cont.): > I don't want to leave you with the impression that it's hopeless... these > attacks aren't impossible to stop --- it just requires convincing the > transit providers to care. > > Damian > > On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me via NANOG < > nanog at nanog.org> wrote: > > Hi Bill, > > thanks for sharing the data. Indeed, I can't offer you a way to backtrack > the spoofed packets. > > Anyway, I'm not sure what could you do legally as there is a very high > chance that these people are not in the USA and the CFAA won't apply to > them. > > Here is what I would do if I was in your situation. > > Since these packets are spoof and malformed, I would block all SYN/ACK > based on the length. > > Depending on your hardware, it's very easy to inspect *only the SYN/ACK > by length* if you have modern hardware. On linux/unix, it's also very > straightforward. I'm not sure for windows though. > > Here is the details of the analysis: > > Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, > SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be > others, but let's use the basic one. > > In your case, there are none. There is only MSS and the SYN length is 44 > bytes. These are spoof packets maybe generated by either TCP-AMP like > reported earlier. > > I would try to block all SYN/ACK coming toward your network with a length > of 44 bytes or lower. But, this is weird because it should be 54 bytes. > Maybe there is some offloading of some sort in your gear. > > Now depending on your hardware, it could work or it could kill your router > as it will punt the cpu. I guess you have some modern gear. > > What I do when I am not sure about the length, I start to accept and log > at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins. > > Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower > than X bytes. It won't kill or block anything legitimate if you do it > properly. :) > > What will happen is that you will not reply to these spoof SYN/ACK with a > RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your > service providers will be happy and should not penalize you. > > I'm pretty sure that it will help you as it did for me in the past. > > Let me know if it's not clear and/or which part is foggy and I'll try to > give more details and better explanation. > > Regards, > > Jean St-Laurent > On 2020-03-14 11:46, William Herrin wrote: > > On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me via NANOG<nanog at nanog.org> <nanog at nanog.org> wrote: > > can you post some forged packets please? You can send them offlist if > you prefer. > > Hi Jean, > > Here are a couple examples (PDT this morning): > > 08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none], > proto ICMP (1), length 56) > 45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable - > admin prohibited filter, length 36 > IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP > (6), length 40) > 199.33.225.218.9851 > 45.89.93.26.443: [|tcp] > 0x0000: 4500 0038 277c 0000 3701 28da 2d59 5d1a > 0x0010: c721 e1da 030d 4b61 0000 0000 4500 0028 > 0x0020: 277c 4000 4506 dae4 c721 e1da 2d59 5d1a > 0x0030: 267b 01bb a057 e903 > > 08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto > TCP (6), length 44) > 104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a > (correct), seq 1216155085, ack 11765035, win 29200, options [mss > 1156], length 0 > 0x0000: 4500 002c 0000 4000 3606 e564 6857 4e5f > 0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b > 0x0020: 6012 7210 c97a 0000 0204 0484 > > I have observed no consistency in the remote IP addresses. I receive > no more than a few of each and they don't line up with particular > networks. Remote ports are heavily 80, 443, 22, 25, etc. but a > smattering of less common ports too. I'm not seeing any RSTs at all > nor any port-unreachables. Lots of syn/acks and a few time exceeded > and host unreachables. I don't know what to make of that. > > > On Sat, Mar 14, 2020 at 1:46 AM Andrew Smith<andrew.william.smith at gmail.com> <andrew.william.smith at gmail.com> wrote: > > Look inside the ICMP Unreachable backscatter at the truncated original packet that caused the unreachable message. > > Clever! I wouldn't have thought of that. Unfortunately as in the > example above, the TTLs in the packets encapsulated in ICMP are not > especially close to one of the common boundaries. > > Regards, > Bill Herrin > > -- > William Herrinbill at herrin.ushttps://bill.herrin.us/ > >

1 0

COVID-19 vs. our Networks
by g＠1337.io 16 Mar '20

16 Mar '20

With talk of there being an involuntary statewide (WA) and then national quarantines (house arrest) for multiple weeks, has anyone put thought into the impacts of this on your networks if/when this comes to fruition? We're already pushing the limits with telecommuters / those that are WFH, but I can only imagine what things will look like with everyone stuck at home for any duration of time.

27 33