What do you consider acceptable packet / session modification for a network operator?
I’m doing a bit of research on how consumer-focused ISPs are modifying their user’s traffic, and I think you guys would have useful insight into legitimate reasons to touch user packets. Almost all of these examples have been seen in the wild by at least one US ISP, so none of these are purely hypothetical. So, how do you feel about where to draw the line for what is acceptable from an ISP? Examples: - Using different source IP ranges in CGNat for ‘web’ traffic vs ’non-web’ (i.e. port 80/443 vs all other ports) - this can break local IP discovery for peer-to-peer stuff if it relies on a ‘web’ port for an API endpoint - Using any form of NAT / packet translation with IPv6 (not including nat64 / other v4 transition related) - Dropping non-TCP/UDP/ICMP protocols (outside of CGNat) - such as ‘raw’ IPSec ESP / AH without UDP encapsulation, or SCTP - TCP MSS - MSS Clamping all connections - TCP MSS - MSS Clamping, but you instead (accidentally?) set MSS to your desired value even if it was lower before - Other TCP options - Dropping syn packets with invalid/unknown options - TCP connection interception - Network operator terminates TCP session from user and then establishes a new one with the original destination. All TCP options, sequence numbers, .. are lost in this translation - Related to above - Network accepts TCP connection which it will intercept (sends SYN/ACK to user) before it confirms that the destination is reachable - Dropping/resetting port 80 sessions that don't ‘look like’ HTTP - Dropping/resetting port 443 sessions that don't ‘look like’ TLS - Redirecting port 53 DNS queries to ISP’s own servers, regardless of destination IP - HTTP header injection into port 80 HTTP traffic (i.e. for user tracking) - HTTP content injection into port 80 HTTP traffic (i.e. replacing ads, adding dialogs, …) (and not blanket redirection for non-payment) Thanks, Andrew ‘apalrd’ Palardy www.apalrd.net https://www.youtube.com/c/apalrdsadventures
If you're talking about consumer ISPs, I personally wouldn't want any of those things. If you're talking about IP TRANSPORT providers, I wouldn't tolerate any of those things. YMMV -- JF On Wed, Dec 24, 2025 at 8:09 PM Andrew via NANOG <nanog@lists.nanog.org> wrote:
I’m doing a bit of research on how consumer-focused ISPs are modifying their user’s traffic, and I think you guys would have useful insight into legitimate reasons to touch user packets.
Almost all of these examples have been seen in the wild by at least one US ISP, so none of these are purely hypothetical.
So, how do you feel about where to draw the line for what is acceptable from an ISP?
Examples:
- Using different source IP ranges in CGNat for ‘web’ traffic vs ’non-web’ (i.e. port 80/443 vs all other ports) - this can break local IP discovery for peer-to-peer stuff if it relies on a ‘web’ port for an API endpoint
- Using any form of NAT / packet translation with IPv6 (not including nat64 / other v4 transition related)
- Dropping non-TCP/UDP/ICMP protocols (outside of CGNat) - such as ‘raw’ IPSec ESP / AH without UDP encapsulation, or SCTP
- TCP MSS - MSS Clamping all connections
- TCP MSS - MSS Clamping, but you instead (accidentally?) set MSS to your desired value even if it was lower before
- Other TCP options - Dropping syn packets with invalid/unknown options
- TCP connection interception - Network operator terminates TCP session from user and then establishes a new one with the original destination. All TCP options, sequence numbers, .. are lost in this translation
- Related to above - Network accepts TCP connection which it will intercept (sends SYN/ACK to user) before it confirms that the destination is reachable
- Dropping/resetting port 80 sessions that don't ‘look like’ HTTP
- Dropping/resetting port 443 sessions that don't ‘look like’ TLS
- Redirecting port 53 DNS queries to ISP’s own servers, regardless of destination IP
- HTTP header injection into port 80 HTTP traffic (i.e. for user tracking)
- HTTP content injection into port 80 HTTP traffic (i.e. replacing ads, adding dialogs, …) (and not blanket redirection for non-payment)
Thanks,
Andrew ‘apalrd’ Palardy www.apalrd.net https://www.youtube.com/c/apalrdsadventures _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/JCNJISMB...
On Thu, Dec 25, 2025 at 01:08:05AM +0000, Andrew via NANOG wrote:
So, how do you feel about where to draw the line for what is acceptable from an ISP?
Some of these may be double-edged (on how a person may feel, depending on their perspective). As an example, some virtual private server operators will drop outgoing SMTP traffic by default. Someone who's the target of spammers may cheer this. Someone who wants to use it to run a mail server (non-spamming) will not. Some operators can be contacted through a form to remove the default filter.
- Redirecting port 53 DNS queries to ISP’s own servers, regardless of destination IP
As a DNS programmer, this particularly bites me. My ISP does this and I bypass it by SSH'ing into machines in a datacenter to do my tests elsewhere. Another option is to use an IP tunnel such as Wireguard. I guess that some ISPs can't avoid intercepting DNS due to government relgulation that asks that certain qnames be blocked (for public benefit?). Some VPN operators (if you can call them internet service providers as they become the default route) also intercept all port 53 traffic and redirect to their own resolvers. This is explained as for the improvement of privacy for the customer. So it depends on the perspective of the person how they feel about this. Mukund
After Federal network neutrality failed, courts delegated those regulatory powers to the states. California’s 2018 SB822 law, for example, prohibits ISPs from blocking, throttling, or engaging in paid prioritization, creating a significant check on some of the ISP practices you suggest. Other states have followed suit, creating a patchwork of regulations. So is there any legally safe way for any ISP to engage in traffic-infringing actions like the ones you discuss? -mel via cell
On Dec 24, 2025, at 6:37 PM, Mukund Sivaraman via NANOG <nanog@lists.nanog.org> wrote:
On Thu, Dec 25, 2025 at 01:08:05AM +0000, Andrew via NANOG wrote:
So, how do you feel about where to draw the line for what is acceptable from an ISP?
Some of these may be double-edged (on how a person may feel, depending on their perspective).
As an example, some virtual private server operators will drop outgoing SMTP traffic by default. Someone who's the target of spammers may cheer this. Someone who wants to use it to run a mail server (non-spamming) will not. Some operators can be contacted through a form to remove the default filter.
- Redirecting port 53 DNS queries to ISP’s own servers, regardless of destination IP
As a DNS programmer, this particularly bites me. My ISP does this and I bypass it by SSH'ing into machines in a datacenter to do my tests elsewhere. Another option is to use an IP tunnel such as Wireguard. I guess that some ISPs can't avoid intercepting DNS due to government relgulation that asks that certain qnames be blocked (for public benefit?).
Some VPN operators (if you can call them internet service providers as they become the default route) also intercept all port 53 traffic and redirect to their own resolvers. This is explained as for the improvement of privacy for the customer. So it depends on the perspective of the person how they feel about this.
Mukund _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/CXB37NBD... <signature.asc>
On Thu, 25 Dec 2025, Mukund Sivaraman via NANOG wrote:
Some of these may be double-edged (on how a person may feel, depending on their perspective).
As an example, some virtual private server operators will drop outgoing SMTP traffic by default. Someone who's the target of spammers may cheer this. Someone who wants to use it to run a mail server (non-spamming) will not. Some operators can be contacted through a form to remove the default filter.
IMO, this has been best practice for cloud/VPS providers for >10 years. Having been on both edges of that sword, I can tell you from first hand experience, there are just too many PoS spammers who will sign up for service with stolen credit card numbers just to spam (if you let them). Defaulting to outgoing 25/TCP blocked puts a stop to that. Legitimate customers can contact the provider to get the block removed. I'm not a fan of CGNAT, but it's become a fact of life for many people and most customers subjected to it will never notice it. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Blue Stream Fiber, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Many of these things happen. None of these things are acceptable. On Wed, Dec 24, 2025 at 8:08 PM Andrew via NANOG <nanog@lists.nanog.org> wrote:
I’m doing a bit of research on how consumer-focused ISPs are modifying their user’s traffic, and I think you guys would have useful insight into legitimate reasons to touch user packets.
Almost all of these examples have been seen in the wild by at least one US ISP, so none of these are purely hypothetical.
So, how do you feel about where to draw the line for what is acceptable from an ISP?
Examples:
- Using different source IP ranges in CGNat for ‘web’ traffic vs ’non-web’ (i.e. port 80/443 vs all other ports) - this can break local IP discovery for peer-to-peer stuff if it relies on a ‘web’ port for an API endpoint
- Using any form of NAT / packet translation with IPv6 (not including nat64 / other v4 transition related)
- Dropping non-TCP/UDP/ICMP protocols (outside of CGNat) - such as ‘raw’ IPSec ESP / AH without UDP encapsulation, or SCTP
- TCP MSS - MSS Clamping all connections
- TCP MSS - MSS Clamping, but you instead (accidentally?) set MSS to your desired value even if it was lower before
- Other TCP options - Dropping syn packets with invalid/unknown options
- TCP connection interception - Network operator terminates TCP session from user and then establishes a new one with the original destination. All TCP options, sequence numbers, .. are lost in this translation
- Related to above - Network accepts TCP connection which it will intercept (sends SYN/ACK to user) before it confirms that the destination is reachable
- Dropping/resetting port 80 sessions that don't ‘look like’ HTTP
- Dropping/resetting port 443 sessions that don't ‘look like’ TLS
- Redirecting port 53 DNS queries to ISP’s own servers, regardless of destination IP
- HTTP header injection into port 80 HTTP traffic (i.e. for user tracking)
- HTTP content injection into port 80 HTTP traffic (i.e. replacing ads, adding dialogs, …) (and not blanket redirection for non-payment)
Thanks,
Andrew ‘apalrd’ Palardy www.apalrd.net https://www.youtube.com/c/apalrdsadventures _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/JCNJISMB...
Am 25.12.2025 um 01:08:05 Uhr schrieb Andrew via NANOG:
- Using any form of NAT / packet translation with IPv6 (not including nat64 / other v4 transition related)
Don't do that, there is enough address space for the customers.
- Dropping non-TCP/UDP/ICMP protocols (outside of CGNat) - such as ‘raw’ IPSec ESP / AH without UDP encapsulation, or SCTP
Don't do that, it's the customers data and not yours, so do not interrupt other people's connections.
- TCP MSS - MSS Clamping all connections
- TCP MSS - MSS Clamping, but you instead (accidentally?) set MSS to your desired value even if it was lower before
This is crap. ICMP exists for this and also works for UDP.
- Other TCP options - Dropping syn packets with invalid/unknown options
Not your task, this is being done at the customer's machines.
- TCP connection interception - Network operator terminates TCP session from user and then establishes a new one with the original destination. All TCP options, sequence numbers, .. are lost in this translation
- Related to above - Network accepts TCP connection which it will intercept (sends SYN/ACK to user) before it confirms that the destination is reachable
Are you a crappy ISP that really needs to do this?
- Dropping/resetting port 80 sessions that don't ‘look like’ HTTP
- Dropping/resetting port 443 sessions that don't ‘look like’ TLS
Can you please stop interfering connections? You are an ISP and people pay your for transferring the data they requested.
- Redirecting port 53 DNS queries to ISP’s own servers, regardless of destination IP
Do you want to attack it? Only nasty ISPs are doing this.
- HTTP header injection into port 80 HTTP traffic (i.e. for user tracking)
- HTTP content injection into port 80 HTTP traffic (i.e. replacing ads, adding dialogs, …) (and not blanket redirection for non-payment)
Ask in darknet crime forums for that. There is the right place for you if you want to do that. -- Gruß Marco Send unsolicited bulk mail to 1766621285muell@cartoonies.org
On Wed, Dec 24, 2025 at 8:59 PM Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
Am 25.12.2025 um 01:08:05 Uhr schrieb Andrew via NANOG:
- Using any form of NAT / packet translation with IPv6 (not including nat64 / other v4 transition related)
Don't do that, there is enough address space for the customers.
Hi Marco, It depends on the price. When you're trying to minimize the price of your service, IPv4 addresses have become one of the expenses you can tweak.
- TCP MSS - MSS Clamping all connections
- TCP MSS - MSS Clamping, but you instead (accidentally?) set MSS to your desired value even if it was lower before
This is crap. ICMP exists for this and also works for UDP.
With due respect, it's no secret that PMTUD on the Internet is broken. There are just too many ways for that ICMP packet from the middle box to get lost and not all of them are a result of ignorant configuration. PMTUD is one of the very few places that IPv4's designers broke with the end-to-end principle and it shows. If you know you're transiting a link with an MTU below 1500, reliable use means clamping the MSS. Sorry, but that's how it is these days.
- Related to above - Network accepts TCP connection which it will intercept (sends SYN/ACK to user) before it confirms that the destination is reachable
Are you a crappy ISP that really needs to do this?
Geostationary satellite. You HAVE to do things to speed up TCP or the customer feels the pain. And before you say Startlink is the answer... it turns out they drop a burst of packets every 15 seconds when they adjust the antenna. Every. 15. Seconds.
- Dropping/resetting port 80 sessions that don't ‘look like’ HTTP
- Dropping/resetting port 443 sessions that don't ‘look like’ TLS
Can you please stop interfering connections? You are an ISP and people pay your for transferring the data they requested.
This is usually done by enterprises rather than ISPs. Except when the DDOS mitigation service is active. Then they're quite pointedly filtering out non-standard traffic. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
On 25.12.2025 10:28 William Herrin <bill@herrin.us> wrote:
It depends on the price. When you're trying to minimize the price of your service, IPv4 addresses have become one of the expenses you can tweak.
I agree on CGNAT (or other forms of NAT) for IPv4, but not IPv6.
- TCP MSS - MSS Clamping all connections
- TCP MSS - MSS Clamping, but you instead (accidentally?) set MSS to your desired value even if it was lower before
This is crap. ICMP exists for this and also works for UDP.
With due respect, it's no secret that PMTUD on the Internet is broken. There are just too many ways for that ICMP packet from the middle box to get lost and not all of them are a result of ignorant configuration. PMTUD is one of the very few places that IPv4's designers broke with the end-to-end principle and it shows.
IPv4 is indeed nasty because if the DF bit is not set, a router might fragment and the receiver might not handle that properly. Everything else is handled by ICMP. If people are blocking that, it is their fault.
If you know you're transiting a link with an MTU below 1500, reliable use means clamping the MSS. Sorry, but that's how it is these days.
If that fixed the problem, it is still broken and everything else (like UDP) is broken.
- Related to above - Network accepts TCP connection which it will intercept (sends SYN/ACK to user) before it confirms that the destination is reachable
Are you a crappy ISP that really needs to do this?
Geostationary satellite. You HAVE to do things to speed up TCP or the customer feels the pain.
If the customer agrees to that - fine. But as a customer I want to know what interception is being done.
- Dropping/resetting port 80 sessions that don't ‘look like’ HTTP
- Dropping/resetting port 443 sessions that don't ‘look like’ TLS
Can you please stop interfering connections? You are an ISP and people pay your for transferring the data they requested.
This is usually done by enterprises rather than ISPs. Except when the DDOS mitigation service is active. Then they're quite pointedly filtering out non-standard traffic.
Enterprises are not ISPs for normal situations. I do filter stuff too in certain parts of my network, but I can decide myself what to filter, rather than my ISP. -- kind regards Marco Send spam to abfall1766654886@stinkedores.dorfdsl.de
So the tl;dr I'm getting from this is that - I don't think anyone considers this 'acceptable' for an ISP - Everyone knows that AT&T does it anyway I haven't found any documentation on how or why AT&T is actually doing this, so it's all either very old or they've scrubbed it from the internet (I know they deleted their whole forums, and it seems like some of the discussion was formerly there)
On Fri, Dec 26, 2025 at 8:44 AM andrew--- via NANOG <nanog@lists.nanog.org> wrote:
So the tl;dr I'm getting from this is that
- I don't think anyone considers this 'acceptable' for an ISP
- Everyone knows that AT&T does it anyway
I haven't found any documentation on how or why AT&T is actually doing this, so it's all either very old or they've scrubbed it from the internet (I know they deleted their whole forums, and it seems like some of the discussion was formerly there)
If you google the term “middlebox”, you will see a lot of research on it from 10 years ago. It was a hot topic of academic and commercial pursuit, but time has proven it is boring for both
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/NFUAW4XI...
I think the general stance, at least in my opinion, is that it may be acceptable for a non-fixed broadband service (e.g., cell phone service), while it is not for a fixed broadband (FTTP or otherwise). I fully expect CGNAT, a basic level of packet mangling in an effort to maximize performance on the cell network, etc., while I am hopping between several different cell towers in an area along with thousands of other users, in an effort to improve the experience of services commonly used on a cell phone… but not so much if I am sitting in my house, connected via my own network, to a wired broadband connection.
On Dec 25, 2025, at 5:44 PM, andrew--- via NANOG <nanog@lists.nanog.org> wrote:
So the tl;dr I'm getting from this is that
- I don't think anyone considers this 'acceptable' for an ISP
- Everyone knows that AT&T does it anyway
I haven't found any documentation on how or why AT&T is actually doing this, so it's all either very old or they've scrubbed it from the internet (I know they deleted their whole forums, and it seems like some of the discussion was formerly there) _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/NFUAW4XI...
Is QUIC also affected in any way? Sent from my iPhone
On 26 Dec 2025, at 22:31, Tim Burke via NANOG <nanog@lists.nanog.org> wrote:
I think the general stance, at least in my opinion, is that it may be acceptable for a non-fixed broadband service (e.g., cell phone service), while it is not for a fixed broadband (FTTP or otherwise). I fully expect CGNAT, a basic level of packet mangling in an effort to maximize performance on the cell network, etc., while I am hopping between several different cell towers in an area along with thousands of other users, in an effort to improve the experience of services commonly used on a cell phone… but not so much if I am sitting in my house, connected via my own network, to a wired broadband connection.
On Dec 25, 2025, at 5:44 PM, andrew--- via NANOG <nanog@lists.nanog.org> wrote:
So the tl;dr I'm getting from this is that
- I don't think anyone considers this 'acceptable' for an ISP
- Everyone knows that AT&T does it anyway
I haven't found any documentation on how or why AT&T is actually doing this, so it's all either very old or they've scrubbed it from the internet (I know they deleted their whole forums, and it seems like some of the discussion was formerly there) _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/NFUAW4XI...
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/J3ZFZOA4...
On 26.12.2025 22:47 Pedro Prado via NANOG <nanog@lists.nanog.org> wrote:
Is QUIC also affected in any way?
It uses UDP, so it is affected by NAT, but not by MSS-clamping. If PMTUD fails and people use MSS-Clamping, QUIC will fail. -- kind regards Marco Send spam to abfall1766785668@stinkedores.dorfdsl.de
Yup, sure it’s UDP; I was just curious wether those providers were also trying to find a way to affect QUIC… Sent from my iPhone
On 27 Dec 2025, at 06:44, Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
On 26.12.2025 22:47 Pedro Prado via NANOG <nanog@lists.nanog.org> wrote:
Is QUIC also affected in any way?
It uses UDP, so it is affected by NAT, but not by MSS-clamping. If PMTUD fails and people use MSS-Clamping, QUIC will fail.
-- kind regards Marco
Send spam to abfall1766785668@stinkedores.dorfdsl.de _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/QV6VJ5PT... <mime-attachment>
On Sat, Dec 27, 2025 at 12:50 AM Pedro Prado via NANOG < nanog@lists.nanog.org> wrote:
Yup, sure it’s UDP; I was just curious wether those providers were also trying to find a way to affect QUIC…
Quic, being modern, has many of these ideas built-in. For example, it has a spin bit and active pmtu discovery along with general lower mss built in.
Sent from my iPhone
On 27 Dec 2025, at 06:44, Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
On 26.12.2025 22:47 Pedro Prado via NANOG <nanog@lists.nanog.org> wrote:
Is QUIC also affected in any way?
It uses UDP, so it is affected by NAT, but not by MSS-clamping. If PMTUD fails and people use MSS-Clamping, QUIC will fail.
-- kind regards Marco
Send spam to abfall1766785668@stinkedores.dorfdsl.de _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/QV6VJ5PT...
<mime-attachment>
NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/7I4Z2HVC...
On Thu, Dec 25, 2025 at 11:04 AM Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
On 25.12.2025 10:28 William Herrin <bill@herrin.us> wrote:
- Using any form of NAT / packet translation with IPv6 (not including nat64 / other v4 transition related)
Don't do that, there is enough address space for the customers.
It depends on the price. When you're trying to minimize the price of your service, IPv4 addresses have become one of the expenses you can tweak.
I agree on CGNAT (or other forms of NAT) for IPv4, but not IPv6.
My error; I misread the question.
- TCP MSS - MSS Clamping all connections
- TCP MSS - MSS Clamping, but you instead (accidentally?) set MSS to your desired value even if it was lower before
This is crap. ICMP exists for this and also works for UDP.
With due respect, it's no secret that PMTUD on the Internet is broken. There are just too many ways for that ICMP packet from the middle box to get lost and not all of them are a result of ignorant configuration. PMTUD is one of the very few places that IPv4's designers broke with the end-to-end principle and it shows.
IPv4 is indeed nasty because if the DF bit is not set, a router might fragment and the receiver might not handle that properly. Everything else is handled by ICMP. If people are blocking that, it is their fault.
That's not really on the list of Internet problems with PMTUD. Not a lot of packets without the DF bit set any more. No, the problem is there's lots of reasons for that ICMP packet to get dropped. * No valid route from the complaining router to the packet origin. IP is end-to-end. You're only supposed to have to guarantee routes between the endpoints, not between the midpoints and endpoints. * Complaining router's interface is numbered with RFC1918. Your firewall drops Internet packets FROM RFC1918 space, right? Most folks' do. * Or even 240/4 for large ISPs who've run out of IPv4 addresses. After all, nobody's supposed to be talking TO the router except ISP staff, right? And I haven't even touched the stupid firewall admins who erroneously block all ICMP "because it's ping." There are a lot of them. No, if you don't want the headache of having to deal with every goofy little situation where PMTUD doesn't work and you _know_ you have a link with an MTU under 1500 (common with ISPs using PPPOE to the customer premise equipment) then you clamp the TCP MSS. You don't like it. But you do it anyway because tech support hours are expensive and that results in fewer of them.
- Related to above - Network accepts TCP connection which it will intercept (sends SYN/ACK to user) before it confirms that the destination is reachable
Are you a crappy ISP that really needs to do this?
Geostationary satellite. You HAVE to do things to speed up TCP or the customer feels the pain.
If the customer agrees to that - fine. But as a customer I want to know what interception is being done.
If you're on a geostationary satellite and your TCP connections don't crawl along at kilobits per second then you know something about your TCP connections is being tampered with. TCP doesn't like half-second RTTs. Really doesn't like it. If you're paying the big bucks for a frequency band you can do whatever you want, but if you're on commodity shared satellite you get the service they provide and if you "do not agree" then you can pound sand. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
Am 26.12.2025 um 06:08:34 Uhr schrieb William Herrin:
That's not really on the list of Internet problems with PMTUD. Not a lot of packets without the DF bit set any more.
No, the problem is there's lots of reasons for that ICMP packet to get dropped.
* No valid route from the complaining router to the packet origin.
IP is end-to-end. You're only supposed to have to guarantee routes between the endpoints, not between the midpoints and endpoints.
I do not understand that. If the router has a public routable address and either a default route to a router with full table, the packet should arrive. Otherwise a general routing problem exist. I am aware of such situations, but PMTU issues are just one of the many issues that are caused by this.
* Complaining router's interface is numbered with RFC1918.
Then the NAT mechanism is failing, as there must not be non-global addresses traveling AS borders. The NAT ACL must include all used addresses that are non-global.
And I haven't even touched the stupid firewall admins who erroneously block all ICMP "because it's ping." There are a lot of them.
I know, but they create there own problems and there is no need that ISPs circumvent their self-made problems.
No, if you don't want the headache of having to deal with every goofy little situation where PMTUD doesn't work and you _know_ you have a link with an MTU under 1500 (common with ISPs using PPPOE to the customer premise equipment) then you clamp the TCP MSS. You don't like it. But you do it anyway because tech support hours are expensive and that results in fewer of them.
I've never seen that yet at the ISPs I use. -- Gruß Marco Send unsolicited bulk mail to 1766725714muell@cartoonies.org
On Fri, 26 Dec 2025 at 16:17, Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
No, if you don't want the headache of having to deal with every goofy little situation where PMTUD doesn't work and you _know_ you have a link with an MTU under 1500 (common with ISPs using PPPOE to the customer premise equipment) then you clamp the TCP MSS. You don't like it. But you do it anyway because tech support hours are expensive and that results in fewer of them.
I've never seen that yet at the ISPs I use.
Maybe I am misreading, I'm reading that physical MTU is 1500B out of which PPPoE headers eat. So the 1500B user packet wont fit. You're saying you've never seen an ISP adjust TCP MSS here? I must have misread, because I've never seen an ISP not adjust here. Funnily enough, there is absolutely no need. My ISP bought gear which can do >1500B, they control both ends of the link, there is a PPP option to negotiate MTU. So my ISP could have just simply configured physical MTU above 1500B, even potentially only when it is their own CPE specifically asking >1500B. And never have to clamp. Yet, they clamp, because it is so ingrained in the industry, people are not even asking why we are doing this. -- ++ytti
Am 26.12.2025 um 17:47:14 Uhr schrieb Saku Ytti:
You're saying you've never seen an ISP adjust TCP MSS here? I must have misread, because I've never seen an ISP not adjust here.
If that fixes the problem, PMTU discovery (mandatory for IPv6 and IPv4 with DF bit) is broken and that means UDP, IPsec, GRE etc. all fail. The ISPs I used emit ICMP packet too big messages. -- Gruß Marco Send unsolicited bulk mail to 1766767634muell@cartoonies.org
On Fri, 26 Dec 2025 at 18:08, Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
If that fixes the problem, PMTU discovery (mandatory for IPv6 and IPv4 with DF bit) is broken and that means UDP, IPsec, GRE etc. all fail.
The ISPs I used emit ICMP packet too big messages.
While it would be terrific if this would work, reality is that content networks all have tried, and all have resorted to fixing lower MSS. Internet is just too broken. Idealistically I agree. -- ++ytti
On Fri, Dec 26, 2025 at 8:07 AM Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
Am 26.12.2025 um 17:47:14 Uhr schrieb Saku Ytti:
You're saying you've never seen an ISP adjust TCP MSS here? I must have misread, because I've never seen an ISP not adjust here.
If that fixes the problem, PMTU discovery (mandatory for IPv6 and IPv4 with DF bit) is broken and that means UDP, IPsec, GRE etc. all fail.
Correct: PMTUD on the Internet is broken. ISPs work around this by engineering a clean 1500 byte path everywhere they can and clamping the MSS the few places that they can't. There's a reason we haven't moved up to 9kb ethernet frames on the server and eyeball LANs. This is that reason.
The ISPs I used emit ICMP packet too big messages.
Everybody emits them. Too many don't make it to the destination. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
I thought 9000-byte MTU wasn't used on LANs due to the headache of ensuring every single device on the LAN has the same MTU. You don't need PMTUD to work on the internet to use longer packets in your LAN. The "packet too big" reply only has to make it from *your* edge router back to *your* server through *your* network. But every host and switch in an Ethernet must agree on MTU because there's no Ethernet-layer PMTUD. On 27 December 2025 17:14:19 CET, William Herrin via NANOG <nanog@lists.nanog.org> wrote:
On Fri, Dec 26, 2025 at 8:07 AM Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
Am 26.12.2025 um 17:47:14 Uhr schrieb Saku Ytti:
You're saying you've never seen an ISP adjust TCP MSS here? I must have misread, because I've never seen an ISP not adjust here.
If that fixes the problem, PMTU discovery (mandatory for IPv6 and IPv4 with DF bit) is broken and that means UDP, IPsec, GRE etc. all fail.
Correct: PMTUD on the Internet is broken. ISPs work around this by engineering a clean 1500 byte path everywhere they can and clamping the MSS the few places that they can't.
There's a reason we haven't moved up to 9kb ethernet frames on the server and eyeball LANs. This is that reason.
The ISPs I used emit ICMP packet too big messages.
Everybody emits them. Too many don't make it to the destination.
Regards, Bill Herrin
-- For hire. https://bill.herrin.us/resume/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/IOTN4VSU...
On Tue, Dec 30, 2025 at 9:24 AM nanog--- via NANOG <nanog@lists.nanog.org> wrote:
I thought 9000-byte MTU wasn't used on LANs due to the headache of ensuring every single device on the LAN has the same MTU.
You don't need PMTUD to work on the internet to use longer packets in your LAN. The "packet too big" reply only has to make it from *your* edge router back to *your* server through *your* network.
But every host and switch in an Ethernet must agree on MTU because there's no Ethernet-layer PMTUD.
Actually, they only have to agree on the MRU and the upper level protocols just about always provide mechanisms to assure the packets they emit won't exceed the recipient's MRU. It's not -quite- that simple but it's simple enough that but for PMTUD being broken on the Internet we could have moved to 9k MTUs by now. Interestingly, AWS VPCs mostly have moved to 9k MTUs. Check your EC2 instance: $ ifconfig -a ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001 The Blackfoot firewalls that implement NAT between the VPCs and the Internet do MSS clamping so that they don't have to rely on PMTUD for TCP to work. Do a tcpdump on both sides. You'll see the MSS leave your EC2 instances in the upper 8000's but arrive at the other end clamped below 1500. Inside the VPC of course, they work at 9k. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
On Sat, 27 Dec 2025 at 17:14, William Herrin via NANOG <nanog@lists.nanog.org> wrote:
The ISPs I used emit ICMP packet too big messages.
Everybody emits them. Too many don't make it to the destination.
In some cases the last router before the MTU bottleneck is not emitting the ICMP Type 3 Code 4 response. In Cisco land for example many configs and templates contain the "no ip unreachables" interface configuration, stopping the router from emitting all ICMP Type 3 messages, including Code 4 Frag needed. Non routable source IPs discarded by uRPF have been mentioned, which is a common problem. Then there is the issue of rate limiting. Rate limiting packets punted to the CPU for ICMP response emission. Rate limiting ICMP response emission itself. And sometimes even ICMP rate limiting on interfaces as a poor mans DDoS mitigation attempt. Or ICMP QoS mapping in worse than best effort queues that overload. Lukas
I do not understand that. If the router has a public routable address and either a default route to a router with full table, the packet should arrive. Otherwise a general routing problem exist. I am aware of such situations, but PMTU issues are just one of the many issues that are caused by this.
Let's say all my physical interfaces have public addresses on them, but my router loopback is numbered RFC1918. Perfectly acceptable and common configuration. Packet comes in with DF set. Egress interface MTU is too small. ICMP Frag Needed generated, source address is RFC1918 loopback from the router control plane. On the return trip, packet crosses network that (correctly) drops all RFC1918 sourced traffic. This is not a routing problem at all. This is very common. On Fri, Dec 26, 2025 at 9:17 AM Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
Am 26.12.2025 um 06:08:34 Uhr schrieb William Herrin:
That's not really on the list of Internet problems with PMTUD. Not a lot of packets without the DF bit set any more.
No, the problem is there's lots of reasons for that ICMP packet to get dropped.
* No valid route from the complaining router to the packet origin.
IP is end-to-end. You're only supposed to have to guarantee routes between the endpoints, not between the midpoints and endpoints.
I do not understand that. If the router has a public routable address and either a default route to a router with full table, the packet should arrive. Otherwise a general routing problem exist. I am aware of such situations, but PMTU issues are just one of the many issues that are caused by this.
* Complaining router's interface is numbered with RFC1918.
Then the NAT mechanism is failing, as there must not be non-global addresses traveling AS borders. The NAT ACL must include all used addresses that are non-global.
And I haven't even touched the stupid firewall admins who erroneously block all ICMP "because it's ping." There are a lot of them.
I know, but they create there own problems and there is no need that ISPs circumvent their self-made problems.
No, if you don't want the headache of having to deal with every goofy little situation where PMTUD doesn't work and you _know_ you have a link with an MTU under 1500 (common with ISPs using PPPOE to the customer premise equipment) then you clamp the TCP MSS. You don't like it. But you do it anyway because tech support hours are expensive and that results in fewer of them.
I've never seen that yet at the ISPs I use.
-- Gruß Marco
Send unsolicited bulk mail to 1766725714muell@cartoonies.org _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/NZLMD3NC...
Am 26.12.2025 um 10:52:48 Uhr schrieb Tom Beecher:
Packet comes in with DF set. Egress interface MTU is too small. ICMP Frag Needed generated, source address is RFC1918 loopback from the router control plane. On the return trip, packet crosses network that (correctly) drops all RFC1918 sourced traffic.
Can't you use NAT (the Cisco ISR devices support nat inside on lo too IIRC) or configure the source address for outgoing router traffic?
This is not a routing problem at all. This is very common.
Indeed, but a filtering problem if routers on the path filter non-public address ranges. -- Gruß Marco Send unsolicited bulk mail to 1766742768muell@cartoonies.org
On Dec 26, 2025, at 11:05 AM, Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
Am 26.12.2025 um 10:52:48 Uhr schrieb Tom Beecher:
Packet comes in with DF set. Egress interface MTU is too small. ICMP Frag Needed generated, source address is RFC1918 loopback from the router control plane. On the return trip, packet crosses network that (correctly) drops all RFC1918 sourced traffic.
Can't you use NAT (the Cisco ISR devices support nat inside on lo too IIRC) or configure the source address for outgoing router traffic?
Some devices you can configure this, but it varies and gets even more interesting when you have a device that may not even have an IPv4 address but is processing IPv4 packets. Eg: https://www.juniper.net/documentation/us/en/software/junos/cli-reference/top... I think as the scope of IPv4 public continues to narrow we will see more of this as time goes on. This was already a challenge for IPv6 over a 6PE network, but this is just the flip side of that. - Jared
[snip]
Examples:
- Using different source IP ranges in CGNat for ‘web’ traffic vs ’non-web’ (i.e. port 80/443 vs all other ports) - this can break local IP discovery for peer-to-peer stuff if it relies on a ‘web’ port for an API endpoint
Even more annoying than basic CGNAT, and doesn’t really benefit the ISP.
- Using any form of NAT / packet translation with IPv6 (not including nat64 / other v4 transition related)
Pointless, annoying, unacceptable.
- Dropping non-TCP/UDP/ICMP protocols (outside of CGNat) - such as ‘raw’ IPSec ESP / AH without UDP encapsulation, or SCTP
Completely unacceptable.
- TCP MSS - MSS Clamping all connections
May be necessary in limited circumstances. Best avoided if possible.
- TCP MSS - MSS Clamping, but you instead (accidentally?) set MSS to your desired value even if it was lower before
That’s just dumb.
- Other TCP options - Dropping syn packets with invalid/unknown options
Annoying and probably ill-advised.
- TCP connection interception - Network operator terminates TCP session from user and then establishes a new one with the original destination. All TCP options, sequence numbers, .. are lost in this translation
I don’t know what you would call this form of proxy, but it’s not internet service.
- Related to above - Network accepts TCP connection which it will intercept (sends SYN/ACK to user) before it confirms that the destination is reachable
A particularly ill-advised version of the above.
- Dropping/resetting port 80 sessions that don't ‘look like’ HTTP
Unacceptable.
- Dropping/resetting port 443 sessions that don't ‘look like’ TLS
Unacceptable
- Redirecting port 53 DNS queries to ISP’s own servers, regardless of destination IP
Unacceptable
- HTTP header injection into port 80 HTTP traffic (i.e. for user tracking)
Unacceptable
- HTTP content injection into port 80 HTTP traffic (i.e. replacing ads, adding dialogs, …) (and not blanket redirection for non-payment)
Unacceptable Owen
Thanks,
Andrew ‘apalrd’ Palardy www.apalrd.net https://www.youtube.com/c/apalrdsadventures _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/JCNJISMB...
participants (17)
-
Andrew -
andrew＠apalrd.net
-
Ca By -
Jared Mauch -
John Fraizer -
Jon Lewis -
Lukas Tribus -
Marco Moock -
Mel Beckman -
Mukund Sivaraman -
nanog＠immibis.com -
Owen DeLong -
Pedro Prado -
Saku Ytti -
Tim Burke -
Tom Beecher -
William Herrin