Are you suggesting that ATT block all QUIC across their network? On Tue, Feb 18, 2020, 7:02 PM Ca By <cb.list6@gmail.com> wrote:

On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling <sterling.daniel@gmail.com> wrote:

...

I've AT&T fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic from google (esp. youtube) becomes very slow after a time.

This especially occurs with ipv4 connections. I'm not the only one to notice; a web search for e.g. "Extremely Poor Youtube TV Performance" notes the issue.

I assume traffic is being throttled on AT&T's side. And it's not done with their CPE; I'm bypassing their NAT box and connecting my laptop directly to the ONT.

A quick google search shows people are aware that QUIC can look like DoS traffic -- but how widely known is this problem? It may become worse if / when sites transition to HTTP/3

For now I reject UDP 443 on the client firewall. Applications using QUIC client libraries then fallback to TCP. This works well and I have no issues with latency or throughput when using TCP.

May I naively ask if Google staff are working with AT&T to address this?

Sounds like you found the answer, ATT just needs to scale up your rejection approach that is proven to work well.

Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they would be mixing with bad company in the pool of ipv4 udp traffic ... but they have reasons.

I am not a fan of quic or any udp traffic. My suggestion was that Google use an new L4 instead of UDP, but that was too hard for the Googlers.

So here we are.

Just say no to udp.

...

-- Dan

The members of the QUIC WG at IETF have not thought this was a problem as they don't observe it across the board. The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - this is also not considered an issue either. The explanation I got (which seems fair) from someone was that they only way to roll a new transport was to squat on some existing stuff that would make it through firewalls. We have an internet that is largely fixed around running NATs and TCP and UDP only these days. I for one find this sad and don't see a good way forward on either side. Sent from my iCar

On Feb 18, 2020, at 7:13 PM, Daniel Sterling <sterling.daniel@gmail.com> wrote:

9On Tue, Feb 18, 2020 at 7:07 PM Ross Tajvar <ross@tajvar.io> wrote:

...

Are you suggesting that ATT block all QUIC across their network?

One might argue they already *are* doing so; QUIC is essentially unusable on my AT&T ipv4 residential connection (and a web search suggests I'm not alone).

On Wed, Feb 19, 2020 at 1:28 AM Daniel Dent <nanog-list@contactdaniel.net> wrote:

On 2020-02-18 4:25 p.m., Jared Mauch wrote:

...

The members of the QUIC WG at IETF have not thought this was a problem as they don't observe it across the board.

The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - this is also not considered an issue either.

Jared, you mean: "Cost on the server" here not "cost on the router". I think, and I ask because at least some of Daniel's note implies, to my reading, that 'on the router' was your concern?

There's plenty of room for system call/interface improvements and hardware acceleration in UDP stacks, both of which should help with CPU concerns. Now that UDP may represent a significant portion of internet traffic, it will be easier for the necessary engineering expense to be justified.

...

The explanation I got (which seems fair) from someone was that they only way to roll a new transport was to squat on some existing stuff that would make it through firewalls. If there's clearly a two-way flow occurring, i.e. as is the case with a

2 way flow means something on your home host or home gateway. It means very little at internet scale... since, in many cases, you -> server and server -> you are not sharing many of the same links / routers / etc. -chris

On Wed, 19 Feb 2020 at 08:18, Masataka Ohta < mohta@necom830.hpcl.titech.ac.jp> wrote:

Christopher Morrow wrote:

...

2 way flow means something on your home host or home gateway. It means very little at internet scale... since, in many cases, you -> server and server -> you are not sharing many of the same links / routers / etc.

Subject suggests it's retail ISP to homes, which are unlikely to be multihomed.

My network has multiple ingress/egress points, and I do my filtering on the edge. The Chris's statement applies to my retail home customers too. Dave

Christopher Morrow wrote:

...

...

2 way flow means something on your home host or home gateway. It means very little at internet scale... since, in many cases, you -> server and server -> you are not sharing many of the same links / routers / etc.

Subject suggests it's retail ISP to homes, which are unlikely to be multihomed.

It probably depends on where you want to do such limiting, right? "At peering/transit edge" - save your core, dont' carry traffic you "know" you will throw away anyway. "At the customer edge" - scaling of state management could be problematic && you'll carry this 'bad' traffic across your network.

Limiting is not by edges but by ISPs. In transit ISPs, there are numerous flows generated by customers of other ISPs that customer wise rate limiting does not scale. In access ISPs, revenue increases proportional to the number of customers that you can prepare rate limiting devices proportional to the number of customers to make customer wise rate limiting scale. Masataka Ohta

On 2020-02-18 7:07 p.m., Ross Tajvar wrote:

Are you suggesting that ATT block all QUIC across their network? Blocking a (for you) undesirable option when an established fallback exists is a much better end user experience than introducing breakage into that option

When you throttle or subtly break things you get: On 2020-02-18 7:12 p.m., Daniel Sterling wrote:

One might argue they already *are* doing so; QUIC is essentially unusable on my AT&T ipv4 residential connection (and a web search suggests I'm not alone).

Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause terrible slowdowns due to packet loss when it broke Or: some AT&T customers cannot connect to our customers due to IPv6 HTTPS interception: https://meta.discourse.org/t/-/140769/3 Or (probably the same problem): https://tutanota.com/blog/posts/att-blocks-tutanota/ With blocking in these cases, QUIC falls back to TCP, Happy Eyeballs falls back to IPv4, everybody's happy.

On 2/18/20 18:40, nanog-list@contactdaniel.net wrote:

Growing prevalence of IPv6-only sites is probably the only thing that will get a lot of access networks to support v6.

I recall a similar idea called "The Great IPv6 Experiment" back in 2007. ;-) -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

No voice service on my line, or TV. Just gigabit internet. Also: I think ipv6 isn't working for me cuz it's being dropped by a switch I'm using! I will swap that out / remove that and try ipv6 again -- Dan On Thu, Feb 27, 2020, 9:10 AM Hiers, David <David.Hiers@cdk.com> wrote:

We find that they usually impose pretty harsh QOS on a link that has an ATT voice service.

David

-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Jay Hennigan Sent: Thursday, February 20, 2020 12:13 AM To: nanog@nanog.org Subject: Re: QUIC traffic throttled on AT&T residential

On 2/18/20 18:40, nanog-list@contactdaniel.net wrote:

...

Growing prevalence of IPv6-only sites is probably the only thing that will get a lot of access networks to support v6.

I recall a similar idea called "The Great IPv6 Experiment" back in 2007. ;-)

-- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

---------------------------------------------------------------------- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

On Tue, Feb 18, 2020 at 8:05 PM Michael Brown <michael@supermathie.net> wrote:

Blocking a (for you) undesirable option when an established fallback exists is a much better end user experience than introducing breakage into that option

Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause terrible slowdowns due to packet loss when it broke

All the +1s. I have one goal for my home internet: it should work better than my cell phone! In our house, "screen time" is "all the time". Everyone is on their phone non-stop. So you'd think getting fiber, installing UBNT APs and swapping out AT&T's CPE for a Core i5 linux box would provide a better internet experience than a tree-obstructed cell tower a mile or two down the road. But you'd be wrong. Everyone in the house was, on a daily basis, turning off wifi in favor of AT&T LTE! Why was everyone switching off wifi? I couldn't blame them -- I was toggling it on and off myself to get the occasional website or IM conversation to load. Why was my network broken?? How was it possible that a fairly high-latency mobile connection could provide a better experience than 802.11ac to an AP in the same room backed by a gigabit PON? I banged against this for *years*. I punted on using my own router and tried just AT&T's CPE (reset to factory defaults). That does work decently, but there are some maddening quirks, not least of which is insanely high jitter. I tried SoHo devices running vendor stock firmware driving hardware NAT. Those also work well -- until they inevitably crash. I tried ddwrt and openwrt. I tried AQM; I tried QoS. I tried NUCs running upstream kernels; downstream kernels; I tried custom patches. I tried HFSC and CoDel; I compiled iproute2 so I could have some tc_cake. On the link-layer (wifi) side I tried one AP; two APs; three APs. I tested any number of combinations of SSID name, channel frequency and width. I tried with ipv6; no ipv6; I put all 2ghz devices on their own AP; in desperation I even tried dedicating one AP and an entire 5ghz frequency range for just one phone. But nothing mattered until I finally figured it out: It was DNS. Of course it was DNS. It's always DNS. When DNS was solid and fast, everything else fell into place. Toggling wifi worked because it was the same as re-querying DNS! And the DNS service on mobile works well -- better than the average CPE. *** I cannot stress this enough. No manner of tuning or tweaking to my home network stopped users from fleeing it, until I had solid DNS. *** For fast DNS, you of course need fast UDP. And, as we've empirically discovered, well-behaved UDP is by no means guaranteed on residential connections. It turns out dnsmasq has a couple of tunables that can make a huge difference to home internet DNS performance. First, you need to be querying the DNS servers AT&T tells you to via DHCP. They're the least likely to be throttled, unfortunately. But I've found even that alone isn't enough. You need to set dnsmasq's "query-port" option. By default it's random to protect against CVE-2008-1447, but apparently sending a ton of random-source-port UDP traffic does not impress the AT&T network flow control systems, and your DNS traffic becomes unbearably slow (or is simply dropped entirely). AT&T gives you two DNS servers via DHCP. You can query more -- 8.8.8.8, 4.2.2.2, 2606:4700:4700::1111 -- but if you do, you'll want to enable dnsmasq's "all-servers" option. Packets are cheap -- send a query to every server on your list and use whatever comes back first. If you've angered the UDP flow restrictor, no matter -- with luck at least one of your packets is going to a server that's up and not throttled or overloaded! Of course DNS is just the beginning -- you still need a proper gateway device with a good NAT stack and/or firewall; you still need a strong wifi signal; you still need tc_cake so everyone can watch Netflix at the same time. But DNS is the *core*. Nothing works well until DNS works well. That means nothing works well unless UDP works well. And if I have learned anything about AS7018, it's that UDP -- especially its v4 UDP -- Does. Not. Work. Well. Enter QUIC. It may be the perfect transport-layer protocol; but by putting it on top of UDP it's hobbled. It breaks extant v4 internet in a way that nothing else we've yet seen does -- it takes what would be your TCP traffic and gives it inconsistent and intermittently poor performance. Maybe it's sometimes fast. Maybe it is. But I can tell you, it sometimes Is Very Much Not So. As much as I would on principle rather not stick to a legacy, TCP-only home network -- I can say that right now, my home internet, blocking UDP 443, and making tons of insecure DNS queries -- is the most stable, fastest, most usable and enjoyable home internet I've ever had. And my users agree -- they no longer turn off wifi. May I naively ask if Google staff have considered scrapping using UDP and instead proposing a new, first-class transport protocol that OSes can implement on top of IP? UDP certainly helped speed testing and iteration for QUIC in real-world scenarios, but I fear UDP is too brittle ground upon which to build the next generation of internet transport. Committing to UDP now with HTTP/3 may be a mistake. And if that doesn't convince you, consider that even I was smart enough to figure out how to block it :) -- Dan

On Tue, Feb 18, 2020 at 8:48 PM Daniel Sterling <sterling.daniel@gmail.com> wrote: [snip impressive debugging story] As much as I would on principle rather not stick to a legacy, TCP-only

home network --

I can say that right now, my home internet, blocking UDP 443, and making tons of insecure DNS queries -- is the most stable, fastest, most usable and enjoyable home internet I've ever had. And my users agree -- they no longer turn off wifi.

Rather than hobble your home network to work around a misbehaving ISP, have you considered simply getting a new ISP? Damian

Peace, On Wed, Feb 19, 2020 at 7:49 AM Daniel Sterling <sterling.daniel@gmail.com> wrote:

May I naively ask if Google staff have considered scrapping using UDP and instead proposing a new, first-class transport protocol that OSes can implement on top of IP?

The IETF WG did, at some point. The opinion overall I think is that this would probably bring in even more troubles both to the protocol (e.g. in plenty of networks which do not allow any IP proto except 1, 6 and 17) and to networks (we have things like RFC 8086 for a reason). There might've been other reasons. -- Töma

On 20 Feb 2020, at 11:29, Daniel Sterling <sterling.daniel@gmail.com> wrote:

On Tue, Feb 18, 2020 at 11:47 PM Daniel Sterling <sterling.daniel@gmail.com> wrote:

...

random-source-port UDP traffic does not impress the AT&T network flow control systems, and your DNS traffic becomes unbearably slow (or is

I received a comment that maybe the issue is not AT&T's "core" network, but rather to do with the NAT device in my house.

Oh, I wish this were the case!

Unfortunately, there is no NAT CPE from AT&T involved:

I've taken AT&T's RG CPE out completely. That device, which otherwise would indeed be my gateway, is unused on my home network, completely unpowered and unplugged.

Instead I've got a linux box hooked directly up to the ONT. This works fine; occasionally I *do* have do reconnect their RG and let it re-auth to the PON, but once the connection is live, I fully unplug their RG again, and plug my laptop (spoofing its MAC) back in directly to the ONT.

The laptop is running ubuntu 19.10, so there should be no artificial limits. It's running NAT directly from the v4 IP I get from DHCP.

You may have noticed v4 is a theme here. I have nothing against using v6 -- , I must admit the truth is I have no idea how to make ubuntu acquire a v6 -- address? block ? I don't even know the right term -- from uverse.

It should just be a DHCPv6 PREFIX DELEGATION (PD). See RFC 8415. If AT&T are still using 6RD (IPv6 Rapid Deployment) that is described in RFC 5969. There is a DHCPv4 option that you can request that gives you the details for configuring the IPv6 in IPv4 tunnel and how to compute the IPv6 prefix from the allocated IPv4 address. CPE’s just try both and use DHCPv6 PD if both exist.

I know v6 works cuz AT&T's device supports it, and openwrt does it out of the box-- but heck if I know how it works. At the risk of asking this list for tech support -- does anyone want to ping me off list and point me in the right direction?? Maybe somebody from Google -- you can make up for breaking my v4 internet!!

Thanks, Dan

-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

Daniel Sterling wrote:

I received a comment that maybe the issue is not AT&T's "core" network, but rather to do with the NAT device in my house.

A problem of QUIC with NAT is that existing NAT can not detect graceful shutdown of QUIC and must depends on timeout. So, port numbers may be used up before timeout. Masataka Ohta

I only wish I were insane; but from where I'm sitting, QUIC has broken my internet, and the resolution is blocking QUIC.

The QUIC protocol itself isn't breaking anything ; some middlebox is breaking QUIC. It's likely collateral damage from honest attempts to mitigate bad stuff. Blocking QUIC at your home edge surely helps to some degree, but the upstream issue still remains. I recall reading a draft from the WG about having things open a parallel TCP connection in case UDP got horked for seamless fallback, but I don't remember if it ever moved past that, or if anyone actually implemented it. On Wed, Feb 19, 2020 at 11:33 PM Daniel Sterling <sterling.daniel@gmail.com> wrote:

On Wed, Feb 19, 2020 at 8:27 PM Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote:

...

A problem of QUIC with NAT is that existing NAT can not detect graceful shutdown of QUIC and must depends on timeout.

So, port numbers may be used up before timeout.

Hmm, this is not what is happening.

I managed to (fairly easily!) reproduce the issue earlier tonight. I generated a fair bit of UDP traffic via xbox, a corporate VPN, and youtube over quic.

Sure enough, after about 45 minutes, the YouTube app on my iPad paused and then auto-reset the stream quality to "144p" resolution.

I was able to set it back to 720p60, but that only lasted about 2 minutes before the stream stopped playing. I waited several minutes for it to resume; it did not. So, I then blocked UDP 443 on my router. The video then *immediately* resumed at 720p60.

I didn't run tcpdump but I did grab some screenshots of iftop. It looks like my iPad connected to AS15169 with a single UDP connection. I see one consistent source and dest IP / port combo for those 10s of minutes, up until UDP is blocked. Local port 58053, remote port 443 on the same IP for the whole time.

At first the connection averages around 2mbps; when the issue occurs, I see it has dropped to a rate of under 200kbps.

I've no idea what to make of that. Surely Google isn't throttling their traffic to me? If so why allow a fallback to TCP?

When I originally discovered this issue, I was of course not trying to do anything odd with my internet connection. And I didn't immediately know QUIC was the issue.

Only after it happened several times did I dig into the traffic and then block QUIC, and I was shocked to see that both resolve the issue and prevent its recurrence.

So again -- I hit this issue repeatedly without trying to --

And just now, I was able to reproduce it simply by generating a bit of UDP traffic on purpose!

I only wish I were insane; but from where I'm sitting, QUIC has broken my internet, and the resolution is blocking QUIC.

-- Dan

On Thursday, 20 February, 2020 08:31, Ca By <cb.list6@gmail.com> wrote:

On Thu, Feb 20, 2020 at 8:34 AM Tom Beecher <beecher@beecher.cc> wrote:

I only wish I were insane; but from where I'm sitting, QUIC has broken my internet, and the resolution is blocking QUIC.

The QUIC protocol itself isn't breaking anything ; some middlebox is breaking QUIC. It's likely collateral damage from honest attempts to mitigate bad stuff. Blocking QUIC at your home edge surely helps to some degree, but the upstream issue still remains.

I recall reading a draft from the WG about having things open a parallel TCP connection in case UDP got horked for seamless fallback, but I don't remember if it ever moved past that, or if anyone actually implemented it.

UDP is broken

The Google devs said it would in fine in 2014

They said it would be “exciting”

UDP is not "broken". UDP is the UNRELIABLE DATAGRAM PROTOCOL and it is living up to its name! What is broken is something pretending that the "U" in UDP means something other than UNRELIABLE and then pretending their use of an unreliable delivery mechanism makes it reliable when that clearly is not the case. In other words, applying a coat of paint to a sows ear to make it look like a silk purse does not make the sows ear into a silk purse. It is still just a sows ear painted to look like a silk purse, and still behaves like a sows ear. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.

On Thu, 20 Feb 2020 at 15:57, Dave Bell <me@geordish.org> wrote:

On Thu, 20 Feb 2020 at 15:31, Ca By <cb.list6@gmail.com> wrote:

...

UDP is broken

I would argue that UDP isn't broken. Networks which drop it indiscriminately are broken.

Does this errant network behaviour not impact RTP applications like video streams? Aled

Not indiscriminate.

Indiscriminate - done at random or without careful judgement. Considering that Daniel is complaining that QUIC is broken, it certainly seems like some network operators are subjecting all UDP traffic on their network to the same policers. This feels pretty indiscriminate to me. I'm all for policing the known baddies, such as CHARGEN and NTP, but to discard UDP for fun is like taking a sledgehammer where a scalpel will do.

Access networks need controls to maintain uptime against the non-stop barrage of ddos attacks. I am sure you have seen the headlines and mails on this list, ddos is hard to deal with. Access network will use whatever tools are required to keep the pagers quiet and customers happy.

I operate an access network that does not blanket police UDP. Google give me a dashboard that tell me 45% of requests were served happily by QUIC, and I have no customers complaining about things not working, and our pagers are silent.

On 2/20/2020 10:41 AM, Dave Bell wrote:

Not indiscriminate.

Indiscriminate - done at random or without careful judgement.

Considering that Daniel is complaining that QUIC is broken, it certainly seems like some network operators are subjecting all UDP traffic on their network to the same policers. This feels pretty indiscriminate to me.

I'm all for policing the known baddies, such as CHARGEN and NTP, but to discard UDP for fun is like taking a sledgehammer where a scalpel will do.

Access networks need controls to maintain uptime against the non-stop barrage of ddos attacks. I am sure you have seen the headlines and mails on this list, ddos is hard to deal with. Access network will use whatever tools are required to keep the pagers quiet and customers happy.

I operate an access network that does not blanket police UDP. Google give me a dashboard that tell me 45% of requests  were served happily by QUIC, and I have no customers complaining about things not working, and our pagers are silent.

Dave, just wanted to say that I 100% agree with your comments. The bad actors are well known. I believe treating all UDP as bad is misguided. Like you, I assist in operation of several access networks that do not blanket police UDP and my pager remains relatively silent.

There are choices, such as making connection initiation, connection acceptance, and connection termination parsable by network elements on the path so state can be established, maintained, and cleared, DoS can be identified, and so on. The decision was to hide all that from network elements. -d

On Feb 20, 2020, at 7:54 PM, Matthew Kaufman <matthew@matthew.at> wrote:

On Thu, Feb 20, 2020 at 8:10 AM Ca By <cb.list6@gmail.com <mailto:cb.list6@gmail.com>> wrote:

Not indiscriminate.

As Google was informed by network operators all along since 2014, ipv4 UDP is a major uptime threat via DDoS to access networks. ...

Google choose not to be sensitive to that, they were told where the landmines were, they choose to go on anyhow.

It isn’t like they had a choice. You can’t build a transport protocol like QUIC on top of TCP (I know, I built one of its ancestors, which also uses UDP underneath). And if you think getting UDP through existing networks is hard, try using a novel IP protocol number.

Matthew Kaufman

On Fri, Feb 21, 2020 at 2:42 PM Jared Mauch <jared@puck.nether.net> wrote:

I can already hear the QUIC WG types blaming the network in abstentia, because well, why would an operator want to keep their network functioning? :-)

In fairness, it's not actually functioning. For one thing, it's passing some traffic at an abysmal rate. ;)

Hi Dan!

On 21 Feb 2020, at 20:22, Dan Wing <danwing@gmail.com> wrote:

There are choices, such as making connection initiation, connection acceptance, and connection termination parsable by network elements on the path so state can be established, maintained, and cleared, DoS can be identified, and so on. The decision was to hide all that from network elements.

Because monetization of content delivery should be constrained only for those, that are able to make new standards, while ignoring openness and cooperation. Google is the new AOL? With AMP, QUIC and other nice, shiny, closed and proprietary stuff? Oh, and BTW, please sync your life with our cloud. Now… once we are aware, the only question is — where we go from here? — ./

First we moved the entire internet to TCP/443. Now we propose moving it all to UDP/53. What’s next? Why not simply eliminate port numbers altogether in favor of a single 16-bit client-side unique session identifier. Owen

On Feb 21, 2020, at 15:20 , Matthew Petach <mpetach@netflight.com> wrote:

On Fri, Feb 21, 2020, 13:31 Łukasz Bromirski <lukasz@bromirski.net <mailto:lukasz@bromirski.net>> wrote:

[...]

Now… once we are aware, the only question is — where we go from here?

— ./

Well, it's clear the UDP 443 experiment wasn't entirely successful.

So clearly, it's time to use the one UDP port that is allowed through at the top of everyone's ACL rules, and update QUIC in the next iteration to use UDP/53.

*THAT* should solve the whole problem, once and for all.

;)

Matt

On 21/02/2020 23:37, Owen DeLong wrote:

What’s next? Why not simply eliminate port numbers altogether in favor of a single 16-bit client-side unique session identifier.

I see what you did there. -- Tom

Daniel Sterling wrote:

...

A problem of QUIC with NAT is that existing NAT can not detect graceful shutdown of QUIC and must depends on timeout.

So, port numbers may be used up before timeout.

Hmm, this is not what is happening.

I thought so. My point is that the problem can be another reason to avoid QUIC. Masataka Ohta

On 2020-02-19 1:06 a.m., Masataka Ohta wrote:

Are you saying AT&T should block UDP entirely?

No; while I don't presume to have all the answers they should at the minimum take into account how it affects the end-user (CUSTOMER!) experience when making decisions like this. (No they shouldn't block UDP entirely, but if they're having UDP/443 DDOS problems then blocking it while they get a proper scalable solution in place is better than throttling it).

On Tue, Feb 18, 2020 at 7:20 PM Dan Wing <danwing@gmail.com> wrote:

For all we know, you and the others noticing the issue have fallen into the pit of A/B testers checking for their current throttling, and others aren't being throttled.

Ouch, I hope not -- do A/B tests that result in extreme performance degradation continue indefinitely ?

Youtube/Google is hoping customers complain to AT&T

It seems like this is one area where AT&T has no reason to care (unless management decides to get involved). The default stance from support would be, I assume: "Google is slow for you? Well obviously it can't be Google's fault. Have a new NAT box" I would bet dollars to donuts that if Google took a proper look at their stats they would see this issue manifesting as users simply not using services that have been QUIC-holed. -- Dan

On Wed, Feb 19, 2020 at 2:55 PM Blake Hudson <blake@ispn.net> wrote:

I'm guessing ATT doesn't disclose this policy transparently either.

they disclose it pretty transparently to their customers in the form of very slow youtube traffic when using v4 QUIC ;)

On Wed, Feb 19, 2020 at 3:34 PM Blake Hudson <blake@ispn.net> wrote:

Yeah, that was a nice surprise to find that my tethered LTE connection was out performing my wired cable modem service. Of course, I had already signed up for a year of service and there were early termination fees for cancelling... that and there are no other wireline providers available at my home (not even ATT).

So we're left with some questions: 1. It's clear I'm not the only one experiencing this issue. How widespread is this problem, really? Has it gotten rather worse over the past ~year? 2. Are customers of larger ISPs much more impacted than customers of smaller ones that (assumedly) don't have to deprioritize UDP so much? 2a. If users *are* impacted, as Blake notes, they may not be able to switch ISPs to improve their lot.. will customers complain to their ISP or to Google? 3. How much worse is the problem when using v4 UDP QUIC vs v6? If QUIC only works on v6 (and if it in fact continues to actively BREAK v4-only users), then is this v6's "killer app" that will drive adoption? 3a. Or will this issue hinder HTTP/3 deployment (or cause mass blocking of UDP on clients)? 4. Will ISPs be willing to give UDP traffic higher priority to improve user experience? Will that only happen once HTTP/3 is widely deployed? 5. We can only assume Google is aware of this issue; will Google work to improve QUIC fallback to TCP, or will they work with ISPs to get QUIC (esp v4 QUIC) prioritized, or will they do nothing, or will they actively encourage QUIC to break v4 at the expensive of current user experience? 5a. Will another company that wants HTTP/3 to succeed take the mantle and work with ISPs to improve the situation? I'm reminded of when Microsoft worked with ISPs to ensure xbox UDP traffic would transit properly -- Dan

On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson <blake@ispn.net> wrote:

...

On Wed, Feb 19, 2020 at 3:34 PM Blake Hudson <blake@ispn.net> wrote:

...

Yeah, that was a nice surprise to find that my tethered LTE connection was out performing my wired cable modem service. Of course, I had already signed up for a year of service and there were early termination fees for cancelling... that and there are no other wireline providers available at my home (not even ATT). So we're left with some questions:

1. It's clear I'm not the only one experiencing this issue. How widespread is this problem, really? Has it gotten rather worse over the past ~year?

2. Are customers of larger ISPs much more impacted than customers of smaller ones that (assumedly) don't have to deprioritize UDP so much? 2a. If users *are* impacted, as Blake notes, they may not be able to switch ISPs to improve their lot.. will customers complain to their ISP or to Google?

3. How much worse is the problem when using v4 UDP QUIC vs v6? If QUIC only works on v6 (and if it in fact continues to actively BREAK v4-only users), then is this v6's "killer app" that will drive adoption? 3a. Or will this issue hinder HTTP/3 deployment (or cause mass blocking of UDP on clients)?

4. Will ISPs be willing to give UDP traffic higher priority to improve user experience? Will that only happen once HTTP/3 is widely deployed?

5. We can only assume Google is aware of this issue; will Google work to improve QUIC fallback to TCP, or will they work with ISPs to get QUIC (esp v4 QUIC) prioritized, or will they do nothing, or will they actively encourage QUIC to break v4 at the expensive of current user experience? 5a. Will another company that wants HTTP/3 to succeed take the mantle and work with ISPs to improve the situation? I'm reminded of when Microsoft worked with ISPs to ensure xbox UDP traffic would transit properly

-- Dan Dan, my experience with Cox is that their standard cable internet

On 2/19/2020 3:21 PM, Daniel Sterling wrote: package (advertised as 150Mbps) rate limits UDP to ~10Mbps. This appears to be controlled via the cable modem config file which is enforced by both the cable modem and the CMTS. I do not know if this is per flow or per circuit or affects IP4 differently than IP6. I suspect that someone at Cox decided that the only applications using UDP were VoIP and DNS and that those applications never needed more than 1Mbps so anything else must be "bad" and should be stopped. Whether "bad" means harmful to network operation, harmful to support costs, or harmful to profits, I do not know.

Your comments seem to differentiate IP4 vs IP6, but I don't believe that is relevant to the issue of an ISP throttling or breaking specific applications. If you have evidence that UDP on IP4 is treated differently than UDP on IP6 by your provider, without further information I would suspect that this is simply an unintentional over sight on their part.

This is your misunderstanding. The protections are to drop ipv4 udp because that is where the ddos / iot trash is , not v6.... for now

Perhaps the attention you've generated on this topic, along with the adoption of additional UDP based applications like QUIC, will encourage ISPs to treat UDP in a more neutral manner and not simply see UDP as something that is "bad".

Dropping udp is not from a “best practice” doc from a vendor, it is deployed by network ops folks that are trying to sleep at night.

--Blake

On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:

On 2/20/2020 10:34 AM, Ca By wrote:

...

On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson <blake@ispn.net <mailto:blake@ispn.net>> wrote: Dropping udp is not from a “best practice” doc from a vendor, it is deployed by network ops folks that are trying to sleep at night.

I get it Ca, I happen to be one of those network ops folks that likes to sleep at night. However, I've never thought it was a good practice to break applications in fun ways for my customers to discover on their own and I've never sold someone a 150Mbps package that actually only delivers 10Mbps for certain applications. Regardless of the intent, ATT and Cox's policies are not transparent, open, or neutral on this topic. This leaves us to speculate on what their intentions might have been and whether their actions are an appropriate response to any concerns they might have had.

I was responsible for deploying such policies in the past, going back as far as the UDP/1434 filters I was forced to deploy due to persistent network congestion. Rolling these back took some time. The same is true for UDP policers we ended up rolling out for NTP, chargen and other activities. Extending these to consumer side where the traffic often originated makes sense until the devices can be secured. You can blame the providers for deploying filters, or not disconnecting consumers that have devices that can be exploited or whatever other reason you believe. As a network operator my goal was always to ensure customers receive the traffic they expected, high rates of UDP were often not what they wanted. Adusting the limits may be useful but I still think the question of what rate of UDP traffic is acceptable is a practical one for the future. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.

and just to check one thing... On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling <sterling.daniel@gmail.com> wrote:

I don't particularly *want* to block or advocate blocking QUIC, but if I keep hitting the issue and can't help people troubleshoot, what other sane option have I?

i don't think you've addressed the "replace your broken ISP" action that is clearly sane and would fix this, right? i'm assuming that this is not an option to get a functional IP layer? t

-- Dan

On Thu, Feb 20, 2020 at 2:57 PM Jared Mauch <jared@puck.nether.net> wrote:

if the question is will the browser vendor (google) or the broadband provider (att) move first, i can already predict the answer. my experience (again) with the quic wg is they seem to think there's many options and bad providers will be replaced which seems disconnected from reality.

Agreed. Since I'm in RTP, I *am* lucky enough to live in an area where there are multiple ISPs. But not everyone is, and local-loop unbundling is no longer a thing :( The average user is at the mercy of their local ISP. So the major providers should (but may not have much incentive to) provide decent service. As has been continually noted, this issue goes away if you use v4 TCP or v6 UDP. And one may argue that most users *will* by default have v6 UDP connectivity from their ISP -- to Google at least. But HTTP/3 is coming -- and site operators may not realize deploying HTTP/3 on v4 only will break connectivity for their users. We're currently in a limbo where v4, which had been working fine, has been *broken* -- and not everyone uses v6. Site operators aren't used to deploying services that break when served over ipv4. HTTP/3 will be the first widely-deployed service that only properly works when served from v6. So something is going to give: * site operators will have to deploy v6 before they deploy HTTP/3, or * network operators will have to make v4 UDP as reliable as v4 TCP (or block HTTP/3) If neither happens, then as more sites start pushing v4 UDP, more and more users will start seeing issues, even if they *do* have v6 working well at their house. It may become common practice to block HTTP/3 on the client side to improve connectivity to v4-only services. Exciting times lay ahead, indeed -- Dan

Hello, On Thu, 20 Feb 2020 at 21:30, Daniel Sterling <sterling.daniel@gmail.com> wrote:

As has been continually noted, this issue goes away if you use v4 TCP or v6 UDP.

IPv6 UDP is currently not broken, that doesn't mean v6 is the solution to this problem. It's just means the particular ISP did not yet deploy the same policies or "mitigations" for v6 traffic. As v6 adoption increases, so will abuse/misuse, especially when attackers notice that their attack traffic is rate-limited on v4 but not on v6 and P2P gaming switches from v4 to v6. And at some point this will lead to "feature parity" in IPv6. So while v6 UDP currently works, I don't think we can assume that's permanent. I disagree this approach is necessary to keep the network running and the pagers from buzzing. In a much smaller eyeball environment (with much smaller chokepoints), we have mapped possibly amplificated packets (ip frag, dns, ntp, memcached, et all) to a specific queue. Unless the links are congested, this traffic passes just as any other traffic and during congestion it only uses whatever bandwidth the queue has - no static rate-limits. I'm not saying this will fix whatever the policies discussed here are supposed to fix, but there is always a way to improve and make the mitigations more nuanced. Of course ISPs will protect the network and the customers. But whether you apply a simple rate-limiting for some traffic or some AI-assisted auto-mitigation for traffic misuse, you gotta be prepared to monitor and update it, staying on top of at least the major false-positives, short-term but long-term as well. After all, things tend to change over time. lukas

i don't think you've addressed the "replace your broken ISP" action that is clearly sane and would fix this, right?

The sanity presumes two things: A: That he could do so without having to change addresses as well. (Something that is still all too true for much of the US.) B: The other provider is not doing anything on their network that might similarly be impacting this specific traffic. (But could very well be doing something to impact other traffic.) On Thu, Feb 20, 2020 at 2:53 PM Todd Underwood <toddunder@gmail.com> wrote:

and just to check one thing...

On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling <sterling.daniel@gmail.com> wrote:

...

I don't particularly *want* to block or advocate blocking QUIC, but if I keep hitting the issue and can't help people troubleshoot, what other sane option have I?

i don't think you've addressed the "replace your broken ISP" action that is clearly sane and would fix this, right?

i'm assuming that this is not an option to get a functional IP layer?

t

...

-- Dan

On Feb 20, 2020, at 4:42 PM, Blake Hudson <blake@ispn.net> wrote:

On 2/20/2020 1:10 PM, Jared Mauch wrote:

...

On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:

...

On 2/20/2020 10:34 AM, Ca By wrote:

...

On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson <blake@ispn.net <mailto:blake@ispn.net>> wrote: Dropping udp is not from a “best practice” doc from a vendor, it is deployed by network ops folks that are trying to sleep at night. I get it Ca, I happen to be one of those network ops folks that likes to sleep at night. However, I've never thought it was a good practice to break applications in fun ways for my customers to discover on their own and I've never sold someone a 150Mbps package that actually only delivers 10Mbps for certain applications. Regardless of the intent, ATT and Cox's policies are not transparent, open, or neutral on this topic. This leaves us to speculate on what their intentions might have been and whether their actions are an appropriate response to any concerns they might have had. I was responsible for deploying such policies in the past, going back as far as the UDP/1434 filters I was forced to deploy due to persistent network congestion. Rolling these back took some time.

The same is true for UDP policers we ended up rolling out for NTP, chargen and other activities.

Extending these to consumer side where the traffic often originated makes sense until the devices can be secured. You can blame the providers for deploying filters, or not disconnecting consumers that have devices that can be exploited or whatever other reason you believe.

As a network operator my goal was always to ensure customers receive the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of what rate of UDP traffic is acceptable is a practical one for the future.

- Jared

I think that's a fair statement Jared. How about this question: Would it be reasonable for one to presume that someone purchasing a 25Mbps internet connection might potentially want to send or receive 25Mbps of UDP traffic? I can think of a few (not uncommon) applications where this would be the case (VPNs, security cameras using RTP, teleconferencing, web browsers implementing QUIC, DNS servers, hosted PBX, etc).

I can think of many legitimate cases, but i think this is where you have internet for everyone and internet for the tech-savvy/business split that becomes interesting. I’ve generally been willing to pay more for a business class service for support and improved response SLA. The average user isn’t going to detect that 10% of their UDP has gone missing, nor should they be expected to. - Jared

On Feb 20, 2020, at 4:53 PM, Blake Hudson <blake@ispn.net> wrote:

...

...

...

As a network operator my goal was always to ensure customers receive the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of what rate of UDP traffic is acceptable is a practical one for the future.

- Jared I think that's a fair statement Jared. How about this question: Would it be reasonable for one to presume that someone purchasing a 25Mbps internet connection might potentially want to send or receive 25Mbps of UDP traffic? I can think of a few (not uncommon) applications where this would be the case (VPNs, security cameras using RTP, teleconferencing, web browsers implementing QUIC, DNS servers, hosted PBX, etc). I can think of many legitimate cases, but i think this is where you have internet for everyone and internet for the tech-savvy/business split that becomes interesting.

I’ve generally been willing to pay more for a business class service for support and improved response SLA. The average user isn’t going to detect that 10% of their UDP has gone missing, nor should they be expected to.

- Jared And here I think is where one crosses the threshold between providing an "internet connection" and providing a connection "that can be used to access specific applications or services" (as defined by your provider). This is one step away from your ISP selling you a connection to access Facebook, if you want to access Twitter that's available on their premium package. Oh, you want to access Slack, sorry we don't offer that as a package yet. Call back in a month. You need to esss-esss-achhh? I've never heard of that, why would you want to do that?

AT&T has rarely offered internet service, their required devices for their U-Verse often munged traffic. I recall when you could reboot their boxes by sending SIP packets to devices behind them and it would intercept them and think it was for itself for their POTS service. If you have any NAT/ALG in there, it’s not pure internet, but most people want access to the “web” and aren’t running ftp/finger/ytalk/uucp/sip etc.. This is why SSL VPNs on 443 became a thing over time. - jared

On Wed, 2020-02-19 at 13:54 -0600, Blake Hudson wrote:

Isn't this exactly why Net Neutrality is a thing:

Isn't it a "dead" thing in the USofA?

So that people (or companies) are free to develop new applications or enhance existing ones without running into a quagmire of different policies implemented by any number of different networks between the application developer and the application's users?

Yes, this is a very prominent reason for Net Neutrality. Too bad the FCC killed that out from under the people and companies that would utilize it to develop new applications. Cheers, b.

If by targeting DDoSes you mean rate limiting all UDP (because some UDP is bad), then I would expect that policy to be published on a provider's website so that folks using UDP would at least have the option to research and know this info before signing up or agreeing to a service commitment. Similarly, application developers could point to a provider's policy when designing their applications or assisting users. Even if Net Neutrality is dead, I believe the "Restoring Internet Freedom" order is still a thing in the US. On 2/19/2020 3:03 PM, Mike Hammett wrote:

Net Neutrality likely wouldn't have impacted this at all. AT&T isn't targeting QUIC, they're targeting DDoSes.

----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ------------------------------------------------------------------------ *From: *"Brian J. Murrell" <brian@interlinx.bc.ca> *To: *nanog@nanog.org *Sent: *Wednesday, February 19, 2020 3:01:20 PM *Subject: *Re: QUIC traffic throttled on AT&T residential

On Wed, 2020-02-19 at 13:54 -0600, Blake Hudson wrote:

...

Isn't this exactly why Net Neutrality is a thing:

Isn't it a "dead" thing in the USofA?

...

So that people (or companies) are free to develop new applications or enhance existing ones without running into a quagmire of different policies implemented by any number of different networks between the application developer and the application's users?

Yes, this is a very prominent reason for Net Neutrality.  Too bad the FCC killed that out from under the people and companies that would utilize it to develop new applications.

Cheers, b.

On 2/19/20 2:54 PM, Fred Baker wrote:

The argument I have heard is that residential firewalls often block anything that is*not* UDP or TCP. The question for the googlers was existential - can it work at all?

I'm not sure that they "block" it, per se, though some probably do have an explicit rule to that effect. I would think the bigger issue is that they don't know how to 1:N NAT arbitrary L4s (and how would they), so the absolute best you might get is that the first device behind the NAT to establish a mapping sees all the relevant L4 traffic and everybody else is locked out. I'd suspect the normal case is simply that they drop it on the floor unless there's a specified "DMZ" host. Perhaps this is just a semantic difference, but I think it's actually an even more difficult issue to resolve. If it were simply blocked, that's usually "easy" (either for the user, via a management interface, or for the vendor, via policy template) to fix. Writing an entirely new L4 NAT helper is a different matter entirely. IPv6 would of course render this moot, but we all know how well IPv6 traffic gets treated... -- Brandon Martin