WIndows Updates Fail Via IPv6
Hi all. Anyone ever figured out why Windows updates fail when the computer has an IPv6 connection? Google has tickets and tickets of this to and outside of Microsoft since 2013, with no real solution or answer as to what the problem actually is. In essence, many of the solutions out there point toward making sure the updates do not occur over IPv6, which, in effect, is the same as disabling it. I have a family PC at home running Windows 10 Pro, and noticed updates would fail in recent months. It took me a moment to realize that this started happening only after I enabled IPv6 in the TCP/IP stack. Disabling it immediately solves the issue. Quite odd that this is happening in 2018... Mark.
at 4:29 AM, Mark Tinka <mark.tinka@seacom.mu> wrote:
Hi all.
Anyone ever figured out why Windows updates fail when the computer has an IPv6 connection?
Google has tickets and tickets of this to and outside of Microsoft since 2013, with no real solution or answer as to what the problem actually is. In essence, many of the solutions out there point toward making sure the updates do not occur over IPv6, which, in effect, is the same as disabling it.
I have a family PC at home running Windows 10 Pro, and noticed updates would fail in recent months. It took me a moment to realize that this started happening only after I enabled IPv6 in the TCP/IP stack. Disabling it immediately solves the issue.
Quite odd that this is happening in 2018...
Mark.
I’ve had IPv6 enabled for a while and I don’t have the same issue. We also peer directly with Microsoft. Are you sure it’s an IPv6 issue and not a general reachability issue? -Daniel
Also no problems here with IPv6 and Windows Updates... On Sun, Nov 11, 2018 at 1:42 PM Daniel Corbe <dcorbe@hammerfiber.com> wrote:
at 4:29 AM, Mark Tinka <mark.tinka@seacom.mu> wrote:
Hi all.
Anyone ever figured out why Windows updates fail when the computer has an IPv6 connection?
Google has tickets and tickets of this to and outside of Microsoft since 2013, with no real solution or answer as to what the problem actually is. In essence, many of the solutions out there point toward making sure the updates do not occur over IPv6, which, in effect, is the same as disabling it.
I have a family PC at home running Windows 10 Pro, and noticed updates would fail in recent months. It took me a moment to realize that this started happening only after I enabled IPv6 in the TCP/IP stack. Disabling it immediately solves the issue.
Quite odd that this is happening in 2018...
Mark.
I’ve had IPv6 enabled for a while and I don’t have the same issue. We also peer directly with Microsoft. Are you sure it’s an IPv6 issue and not a general reachability issue?
-Daniel
-- Regards, Chris Knipe
On 11/Nov/18 14:02, Chris Knipe wrote:
Also no problems here with IPv6 and Windows Updates...
The issue is affecting (and has affected) quite a few folk: https://social.technet.microsoft.com/Forums/windowsserver/en-US/a9b1b537-ad2... https://social.technet.microsoft.com/Forums/office/en-US/16e7aa06-9b90-48f8-... https://answers.microsoft.com/en-us/windows/forum/all/windows-8-pro-windows-... It occurs to me that this, could, perhaps be CDN specific. I'm currently not in Johannesburg, but last time I checked, the majority of Windows updates were being handled by Akamai. Perhaps this is where the issue could be, although we have got local IPv6 peering with Akamai and don't generally have issues with it. I'll dig deeper into how Akamai may be involved when I get back home. Thanks for the pointers. Mark.
On Nov 11, 2018, at 8:45 AM, Mark Tinka <mark.tinka@seacom.mu> wrote:
On 11/Nov/18 14:02, Chris Knipe wrote:
Also no problems here with IPv6 and Windows Updates...
The issue is affecting (and has affected) quite a few folk:
https://social.technet.microsoft.com/Forums/windowsserver/en-US/a9b1b537-ad2... https://social.technet.microsoft.com/Forums/office/en-US/16e7aa06-9b90-48f8-... https://answers.microsoft.com/en-us/windows/forum/all/windows-8-pro-windows-...
It occurs to me that this, could, perhaps be CDN specific.
I'm currently not in Johannesburg, but last time I checked, the majority of Windows updates were being handled by Akamai. Perhaps this is where the issue could be, although we have got local IPv6 peering with Akamai and don't generally have issues with it.
I'll dig deeper into how Akamai may be involved when I get back home.
Let me know if you see anything related to Akamai. Looking at these threads I don’t see anything really obvious and some are much older posts. - Jared
Not to beat a dead horse, but could the problem be so simple? I have tons of dual-stacked machines that have updated forever without issue, so I assume they update via IPV4. That being said, I've not packet sniffed any of the update stuff in a while but if the DNS is any indication then dual-stacked machines can update via IPV4 while IPV6 ONLY machines will likely fail since the DNS shows IPV4 only.... host windowsupdate.microsoft.com windowsupdate.microsoft.com is an alias for windowsupdate.redir.update.microsoft.com.nsatc.net. windowsupdate.redir.update.microsoft.com.nsatc.net is an alias for redir.update.microsoft.com.nsatc.net. redir.update.microsoft.com.nsatc.net has address 157.56.77.153 On 11/11/2018 01:35 PM, Jared Mauch wrote:
On Nov 11, 2018, at 8:45 AM, Mark Tinka <mark.tinka@seacom.mu> wrote:
On 11/Nov/18 14:02, Chris Knipe wrote:
Also no problems here with IPv6 and Windows Updates...
The issue is affecting (and has affected) quite a few folk:
https://social.technet.microsoft.com/Forums/windowsserver/en-US/a9b1b537-ad2... https://social.technet.microsoft.com/Forums/office/en-US/16e7aa06-9b90-48f8-... https://answers.microsoft.com/en-us/windows/forum/all/windows-8-pro-windows-...
It occurs to me that this, could, perhaps be CDN specific.
I'm currently not in Johannesburg, but last time I checked, the majority of Windows updates were being handled by Akamai. Perhaps this is where the issue could be, although we have got local IPv6 peering with Akamai and don't generally have issues with it.
I'll dig deeper into how Akamai may be involved when I get back home.
Let me know if you see anything related to Akamai. Looking at these threads I don’t see anything really obvious and some are much older posts.
- Jared
-- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------------- The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ----------------------------------------------------------------------------
On 11/Nov/18 20:35, Jared Mauch wrote:
Let me know if you see anything related to Akamai.
Will do.
Looking at these threads I don’t see anything really obvious and some are much older posts.
Agreed - but those posts were never really solved, and the issue description and behaviour mirrors my own. Mark.
I’m on native IPv6 via Spectrum and have no problems with Windows Updates. Could this be a tunneling issue? Tony
On Nov 10, 2018, at 23:29, Mark Tinka <mark.tinka@seacom.mu> wrote:
Hi all.
Anyone ever figured out why Windows updates fail when the computer has an IPv6 connection?
Google has tickets and tickets of this to and outside of Microsoft since 2013, with no real solution or answer as to what the problem actually is. In essence, many of the solutions out there point toward making sure the updates do not occur over IPv6, which, in effect, is the same as disabling it.
I have a family PC at home running Windows 10 Pro, and noticed updates would fail in recent months. It took me a moment to realize that this started happening only after I enabled IPv6 in the TCP/IP stack. Disabling it immediately solves the issue.
Quite odd that this is happening in 2018...
Mark.
On 11/Nov/18 18:51, Lavanauts wrote:
I’m on native IPv6 via Spectrum and have no problems with Windows Updates. Could this be a tunneling issue?
I do run 6-in-4 from my backbone to my house as my FTTH provider does not do IPv6. I can't imagine this to specifically be the issue, as all other IPv6 traffic is fine, but at this point, I'm open to suggestion. Mark.
On Mon, 12 Nov 2018, Mark Tinka wrote:
I do run 6-in-4 from my backbone to my house as my FTTH provider does not do IPv6.
I can't imagine this to specifically be the issue, as all other IPv6 traffic is fine, but at this point, I'm open to suggestion.
Are you doing TCP MSS adjust/clamping? If you don't, try that and see if it helps. This might be a PMTUD issue. Otherwise if possible, try lowering the MTU sent in RA to the one you have on your tunnel (this depends on if this is available to you in your RA sending device). -- Mikael Abrahamsson email: swmike@swm.pp.se
On 12/Nov/18 20:34, Mikael Abrahamsson wrote:
Are you doing TCP MSS adjust/clamping? If you don't, try that and see if it helps. This might be a PMTUD issue.
Otherwise if possible, try lowering the MTU sent in RA to the one you have on your tunnel (this depends on if this is available to you in your RA sending device).
Thanks, Mikael. I'll have a sniff and see of this helps. Mark.
Hi all. Just an update on this... it did turn out to be an MTU issue which I've been working on since last year, November. The trick was finding the right combination of settings between my Mikrotik home router and one of our Cisco ASR1006 edge routers in my backbone that terminates the 6-in-4 tunnel. After testing this at the office, I was, then, sure that the problem was MTU-related as it only occurred at my house. My FTTH service is delivered over GPON, and after a bit of testing, concluded that my MTU for IPv4 is 1,452 bytes. Across the 6-in-4 tunnel, the tested MTU is 1,232 for IPv6. The Mikrotik will not pass any IPv6 traffic if the 6-in-4 tunnel does not have the minimum default MTU for IPv6, i.e., 1,280 bytes (even if the tunnel cannot actually transport 1,280 byte-sized packets). This is not documented anywhere, so it took a while to figure out. On the Cisco, you can't configure an IPv6 interface MTU lower than 1,280 bytes... but that is within the standard IPv6 spec., so no major drama. So the right combination of settings is to have 1,280 bytes on the Mikrotik and enable "ipv6 tcp adjust-mss 1232" on the Cisco (the latter for my specific case - yours may vary depending on your IPv4 conditions). Just wanted to update this thread in case someone else runs into this issue. Thanks for the clue, Mikael and all. Mark. On 13/Nov/18 12:38, Mark Tinka wrote:
On 12/Nov/18 20:34, Mikael Abrahamsson wrote:
Are you doing TCP MSS adjust/clamping? If you don't, try that and see if it helps. This might be a PMTUD issue.
Otherwise if possible, try lowering the MTU sent in RA to the one you have on your tunnel (this depends on if this is available to you in your RA sending device). Thanks, Mikael.
I'll have a sniff and see of this helps.
Mark.
On 2019-03-03 11:31, Mark Tinka wrote: [..]
Across the 6-in-4 tunnel, the tested MTU is 1,232 for IPv6.
IPv6 requires a minimum MTU of 1280. If you cannot transport it, then the transport (the tunnel in this case) needs to handle the fragmentation of packets of 1280 down to whatever does fit in the tunnel. [..]
So the right combination of settings is to have 1,280 bytes on the Mikrotik and enable "ipv6 tcp adjust-mss 1232" on the Cisco (the latter for my specific case - yours may vary depending on your IPv4 conditions).
Have fun with all your UDP traffic that does not care about your TCP MSS adjustment. You just hid the problem... And a correctly configured MTU is especially going to be fun with "HTTP/3" that is being pushed through, even though the predecessor QUIC does not care about MTU at all... good that it is all in the hands of a company that can fix it themselves ;) Greets, Jeroen
On 3/Mar/19 18:05, Jeroen Massar wrote:
IPv6 requires a minimum MTU of 1280.
If you cannot transport it, then the transport (the tunnel in this case) needs to handle the fragmentation of packets of 1280 down to whatever does fit in the tunnel.
As you know, IPv6 does not support fragmentation in transit. So that's not an option. Host fragmentation is per standard, but signaling of that was not so successful in IPv4. Real world scenarios for IPv6 (reasonably) apply here.
Have fun with all your UDP traffic that does not care about your TCP MSS adjustment. You just hid the problem...
I considered this issue, but as with all things UDP re: fragmentation, it depends. Testing I've been doing all day shows previously (mostly-TCP) issues have resolved, and I've not run into any major problems that are impacting UDP. Nonetheless, I'm keeping an eye out.
And a correctly configured MTU is especially going to be fun with "HTTP/3" that is being pushed through, even though the predecessor QUIC does not care about MTU at all... good that it is all in the hands of a company that can fix it themselves ;)
Is it an ideal situation? About as ideal as flying in the cargo bay. But my reality is that until my FTTH provider can deliver native IPv6, this is what I have. Mark.
On 2019-03-03 20:13, Mark Tinka wrote:
On 3/Mar/19 18:05, Jeroen Massar wrote:
IPv6 requires a minimum MTU of 1280.
If you cannot transport it, then the transport (the tunnel in this case) needs to handle the fragmentation of packets of 1280 down to whatever does fit in the tunnel.
As you know, IPv6 does not support fragmentation in transit. So that's not an option.
The transport (tunnel) CAN support that kind of fragmentation. (e.g. the tunnel could chop up a 1280 byte packet into two packets and the remote then join them together; that is a "ethernet" level thing). Indeed, IPv6 won't do it: get a better transport.
Host fragmentation is per standard, but signaling of that was not so successful in IPv4. Real world scenarios for IPv6 (reasonably) apply here.
Real world for IPv6 is: do not try to transport it over a medium that does not support packets of at least 1280 bytes.
Have fun with all your UDP traffic that does not care about your TCP MSS adjustment. You just hid the problem...
I considered this issue, but as with all things UDP re: fragmentation, it depends.
Testing I've been doing all day shows previously (mostly-TCP) issues have resolved, and I've not run into any major problems that are impacting UDP. Nonetheless, I'm keeping an eye out.
And a correctly configured MTU is especially going to be fun with "HTTP/3" that is being pushed through, even though the predecessor QUIC does not care about MTU at all... good that it is all in the hands of a company that can fix it themselves ;)
Is it an ideal situation? About as ideal as flying in the cargo bay. But my reality is that until my FTTH provider can deliver native IPv6, this is what I have.
Maybe you should ask this "FTTH" provider to deliver a decent MTU size? (next to native IPv6, something something, 20+ years old protocol...) Greets, Jeroen
On 3/Mar/19 21:57, Jeroen Massar wrote:
The transport (tunnel) CAN support that kind of fragmentation.
(e.g. the tunnel could chop up a 1280 byte packet into two packets and the remote then join them together; that is a "ethernet" level thing).
If you have a working example between a Cisco IOS XE device and a Mikrotik router, I am all ears.
Real world for IPv6 is: do not try to transport it over a medium that does not support packets of at least 1280 bytes.
Note I am not recommending this as a best practice. I believe the subscribers on this list are clued enough to discern that for themselves. But in the interest of posterity, let me make that explicit.
Maybe you should ask this "FTTH" provider to deliver a decent MTU size? (next to native IPv6, something something, 20+ years old protocol...)
:-), you're a funny guy... maybe my provider and I will just get off the Internet altogether. Mark.
On 3/3/19 16:57, Jeroen Massar wrote:
On 2019-03-03 20:13, Mark Tinka wrote:
On 3/Mar/19 18:05, Jeroen Massar wrote:
IPv6 requires a minimum MTU of 1280.
If you cannot transport it, then the transport (the tunnel in this case) needs to handle the fragmentation of packets of 1280 down to whatever does fit in the tunnel.
As you know, IPv6 does not support fragmentation in transit. So that's not an option.
The transport (tunnel) CAN support that kind of fragmentation.
Still, that's certainly not panacea. See: https://tools.ietf.org/html/rfc7872 -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers. There are also IDIOTS that deploy load balancers that DO NOT LOOK INSIDE ICMP messages for redirecting ICMP messages to the correct back end. There are also IDOITS that rate limit PTB generation to ridiculously low rates. One should be able to generate PTB at line rate. Everyone that has configured mss-fix-up has contributed to misunderstanding that you can block ICMP. It is time we had a flag day to REMOVE mss-fix-up from all the boxes you control. We need to get PTB working and unfortunately that means that we need to stop pandering to admins who don’t know how IP is supposed to work. ICMP is NOT optional. If you don’t want to do PMTUD then DO NOT SEND packet bigger than the network MTU. For IPv6 set IPV6_USE_MIN_MTU 1 on the socket. On a properly written IP stack this will result in TCP MSS negotiation to the same value. Yes, it is a requirement of TCP to pay attention to this as it becomes the effective MTU of the outgoing interface even if it wasn’t explicitly written into the RFC that defined IPV6_USE_MIN_MTU. Mark
On 4 Mar 2019, at 6:13 am, Mark Tinka <mark.tinka@seacom.mu> wrote:
On 3/Mar/19 18:05, Jeroen Massar wrote:
IPv6 requires a minimum MTU of 1280.
If you cannot transport it, then the transport (the tunnel in this case) needs to handle the fragmentation of packets of 1280 down to whatever does fit in the tunnel.
As you know, IPv6 does not support fragmentation in transit. So that's not an option.
Host fragmentation is per standard, but signaling of that was not so successful in IPv4. Real world scenarios for IPv6 (reasonably) apply here.
Have fun with all your UDP traffic that does not care about your TCP MSS adjustment. You just hid the problem...
I considered this issue, but as with all things UDP re: fragmentation, it depends.
Testing I've been doing all day shows previously (mostly-TCP) issues have resolved, and I've not run into any major problems that are impacting UDP. Nonetheless, I'm keeping an eye out.
And a correctly configured MTU is especially going to be fun with "HTTP/3" that is being pushed through, even though the predecessor QUIC does not care about MTU at all... good that it is all in the hands of a company that can fix it themselves ;)
Is it an ideal situation? About as ideal as flying in the cargo bay. But my reality is that until my FTTH provider can deliver native IPv6, this is what I have.
Mark.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 3/Mar/19 23:04, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers. There are also IDIOTS that deploy load balancers that DO NOT LOOK INSIDE ICMP messages for redirecting ICMP messages to the correct back end. There are also IDOITS that rate limit PTB generation to ridiculously low rates. One should be able to generate PTB at line rate.
Everyone that has configured mss-fix-up has contributed to misunderstanding that you can block ICMP. It is time we had a flag day to REMOVE mss-fix-up from all the boxes you control. We need to get PTB working and unfortunately that means that we need to stop pandering to admins who don’t know how IP is supposed to work. ICMP is NOT optional.
If you don’t want to do PMTUD then DO NOT SEND packet bigger than the network MTU. For IPv6 set IPV6_USE_MIN_MTU 1 on the socket. On a properly written IP stack this will result in TCP MSS negotiation to the same value. Yes, it is a requirement of TCP to pay attention to this as it becomes the effective MTU of the outgoing interface even if it wasn’t explicitly written into the RFC that defined IPV6_USE_MIN_MTU.
You're most welcome to my choir group, good sir. Mark.
On 3/3/19 1:04 PM, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers.
For those of us who are in the dark, "PTB" appears to refer to "Packet Too Big" responses in ICMPv6. Yes, some admins don't have fine-enough grain tools to block or throttle specific types of ICMP, but that's the fault of the vendors, not the admins.
On 4 Mar 2019, at 9:33 am, Stephen Satchell <list@satchell.net> wrote:
On 3/3/19 1:04 PM, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers.
For those of us who are in the dark, "PTB" appears to refer to "Packet Too Big" responses in ICMPv6.
Yes, some admins don't have fine-enough grain tools to block or throttle specific types of ICMP, but that's the fault of the vendors, not the admins.
No, it is the fault of the admins. They should be making it part of the purchasing decision if they want to filter ICMP. It’s not like selective filtering is a new idea. It is well over 20 years old at this stage. The amount of +20 year old equipment on the net is minimal. That said modern OS’s don’t need other equipment to “protect" them from ICMP of any form. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 3/3/19 20:16, Mark Andrews wrote:
On 4 Mar 2019, at 9:33 am, Stephen Satchell <list@satchell.net> wrote:
On 3/3/19 1:04 PM, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers.
For those of us who are in the dark, "PTB" appears to refer to "Packet Too Big" responses in ICMPv6.
Yes, some admins don't have fine-enough grain tools to block or throttle specific types of ICMP, but that's the fault of the vendors, not the admins.
No, it is the fault of the admins. They should be making it part of the purchasing decision if they want to filter ICMP. It’s not like selective filtering is a new idea. It is well over 20 years old at this stage. The amount of +20 year old equipment on the net is minimal.
That said modern OS’s don’t need other equipment to “protect" them from ICMP of any form.
These news don't help in that direction: https://www.theregister.co.uk/2016/06/02/cisco_warns_of_ipv6_dos_vulnerabili... (I'm not complaining about the news, but about the bugs, if you wish) -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
On Sun, Mar 3, 2019, at 17:35, Stephen Satchell wrote:
Yes, some admins don't have fine-enough grain tools to block or throttle specific types of ICMP, but that's the fault of the vendors, not the admins.
We call these tunable parameters "nerd knobs". I used to create those knobs for firewalls. My experience then (and now, with my current employer) is that admins turn every knob you give them up to eleven; there is no finesse. The only answer was, and is, to remove the knobs altogether. (Can I join the choir too? :) -- Harald Koch chk@pobox.com
On Sun, Mar 3, 2019, at 22:05, Mark Andrews wrote:
admins who don’t know how IP is supposed to work.
You do realise that in "corporate world" that's more than 80% of network admins ? Some of them even make it to "audit" companies, so they can screw a company with clueful admins with their "mandatory reccomandations".
ICMP is NOT optional.
Can we make a short rule that says: For ICMP, *ALLOW* *ALL* unless you do have a very specific and motivated reason to block some types. I would even go as far as "allow all icmp from any to any" (and if possible as the first firewall rule), but I do understand that may make some people have hives.
On 4/Mar/19 09:12, Radu-Adrian Feurdean wrote:
Can we make a short rule that says: For ICMP, *ALLOW* *ALL* unless you do have a very specific and motivated reason to block some types. I would even go as far as "allow all icmp from any to any" (and if possible as the first firewall rule), but I do understand that may make some people have hives.
Not to be the wet blanket, but we've be crying about this since before I knew what CLI meant, and it either didn't work or has gotten even worse. That is how we ended up with all manner of hacks to work around failure to reliably deliver PTB messages. We've been crying about the same during the IPv6 era, and we appear to be running the same hacks for it too. Is there any reason to expect things to change given the continued "crying about it" approach? Just look at what I had to (unhappily) do over the weekend :-(. I don't have the answers yet, but just because it now ends with a "6", doesn't mean we shall necessarily drop our IPv4 bad habits. Perhaps it's time to consider a different approach, if we don't want to resign ourselves to the death of ICMP as we know it, and simply talking about what could have been had its full potential been realized. Mark.
On Mon, Mar 4, 2019 at 10:02 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
Can we make a short rule that says: For ICMP, *ALLOW* *ALL* unless you do have a very specific and motivated reason to block some types. I would even go as far as "allow all icmp from any to any" (and if possible as the first firewall rule), but I do understand that may make some people have hives.
Not to be the wet blanket, but we've be crying about this since before I knew what CLI meant, and it either didn't work or has gotten even worse. That is how we ended up with all manner of hacks to work around failure to reliably deliver PTB messages.
Not just ICMP but everything. We've designed these nice extendible protocols, but we've configured network so that we can't extend them. Like why is QUIC riding on UDP, instead of having its own L4 protocol number. Because of HTTP/3 majority of Internet traffic will be UDP, and due to its reflection potential in other applications that is not obvious net win. We should just retire UDP with status of 'trusted network only L4' and use something like QUIC for all untrusted L4 applications, where we've thought about issues like reflection. -- ++ytti
On Mon, Mar 04, 2019 at 08:04:12AM +1100, Mark Andrews wrote:
ICMP is NOT optional.
I've pointed folks at this for years: ICMP Packet Filtering v1.2 http://www.cymru.com/Documents/icmp-messages.html ---rsk
Hey Rich,
I've pointed folks at this for years: ICMP Packet Filtering v1.2 http://www.cymru.com/Documents/icmp-messages.html
To me this seems anti-pattern. It seems it was written on basis of 'what we know we allow, what we don't know we deny'. With assumption that ICMP dangerous. Similarly we break IP extensibility by not allowing IP protocols we don't know about. There are many, hopefully obvious reasons that just because we don't know about it, doesn't mean it's dangerous. One more obvious is, that it may not exist yet. To me, the correct pattern is here is to deny things you know to be harmful and can justify it reasonably and test that justification over time for its validity. One particular example springs to mind, ICMP Timestamp, this allows you to measure unidirectional latency to millisecond precision, unless we specifically break it. It's been useful troubleshooting tool to me in the past, saving time and money. -- ++ytti
From: NANOG <nanog-bounces@nanog.org> On Behalf Of Saku Ytti
Hey Rich,
I've pointed folks at this for years: ICMP Packet Filtering v1.2 http://www.cymru.com/Documents/icmp-messages.html
To me, the correct pattern is here is to deny things you know to be harmful and can justify it reasonably and test that justification over time for its validity.
Let me play a devil's advocate here, the above statement begs a question then, how do you know all that is harmful would you test for every possible extension and hw/sw permutation? So there would be 3 sets (though lines might be blurred) known safe, known harmful and the biggest of them unknown unknowns. Now as an operator of a commercial network (i.e. your customers like it mostly up) wouldn't you do a calculated risk evaluation and opt for the known safe -which you know 99% of your customers use and block the rest while pissing off the remaining 1%? I know it sounds awful (like a calculations for vehicle safety recalls), but ... adam
On Tue, Mar 5, 2019 at 4:54 PM <adamv0025@netconsultings.com> wrote:
Let me play a devil's advocate here, the above statement begs a question then, how do you know all that is harmful would you test for every possible extension and hw/sw permutation? So there would be 3 sets (though lines might be blurred) known safe, known harmful and the biggest of them unknown unknowns. Now as an operator of a commercial network (i.e. your customers like it mostly up) wouldn't you do a calculated risk evaluation and opt for the known safe -which you know 99% of your customers use and block the rest while pissing off the remaining 1%? I know it sounds awful (like a calculations for vehicle safety recalls), but ...
You don't know. Everything is horribly broken anyhow and if you are not pwned, the main reason is that you're not attractive target. If you are being targeted, you will be pwned by zero to modest budget. Attacker budget leverage to defender is ridiculous. And ICMP won't be the vector. Fear is excellent marketing tool, as we can see in politics, works every time. But I rather fix realised problems, rather than make unprovable assumptions of actions yielding to net benefit. The assumption here is, if we just allow ICMP types A, B and C we are gaining in security, can we substantiate that claim at all? We can substantiate easily that the proposed ICMP filter breaks real useful ICMP tooling. -- ++ytti
From: Saku Ytti <saku@ytti.fi> Sent: Tuesday, March 5, 2019 3:00 PM
On Tue, Mar 5, 2019 at 4:54 PM <adamv0025@netconsultings.com> wrote:
Let me play a devil's advocate here, the above statement begs a question then, how do you know all that is harmful would you test for every possible extension and hw/sw permutation? So there would be 3 sets (though lines might be blurred) known safe, known harmful and the biggest of them unknown unknowns. Now as an operator of a commercial network (i.e. your customers like it mostly up) wouldn't you do a calculated risk evaluation and opt for the known safe -which you know 99% of your customers use and block the rest while pissing off the remaining 1%? I know it sounds awful (like a calculations for vehicle safety recalls), but ...
Fear is excellent marketing tool, as we can see in politics, works every time. But I rather fix realised problems, rather than make unprovable assumptions of actions yielding to net benefit. The assumption here is, if we just allow ICMP types A, B and C we are gaining in security, can we substantiate that claim at all? We can substantiate easily that the proposed ICMP filter breaks real useful ICMP tooling.
From past experience my assumptions would be more along the lines of if it's not mainstream there's a higher likelihood that it might trigger exceptions in code.
adam
On Thu, Mar 7, 2019 at 5:19 PM <adamv0025@netconsultings.com> wrote:
From past experience my assumptions would be more along the lines of if it's not mainstream there's a higher likelihood that it might trigger exceptions in code.
My point is, let it break. Don't pre-emptively drop things that you don't know to be harmful, but where dropping definitely is harmful. After risk has realised you have more data about the risk. If it has never realised you have no idea. If we extrapolate this culture of fear, we will only have HTTPS open, nothing else, and we have to build then next-gen stuff as an overlay using HTTPS transport. -- ++ytti
Hey Saku,
From: Saku Ytti <saku@ytti.fi> Sent: Thursday, March 7, 2019 3:29 PM
On Thu, Mar 7, 2019 at 5:19 PM <adamv0025@netconsultings.com> wrote:
From past experience my assumptions would be more along the lines of if it's not mainstream there's a higher likelihood that it might trigger exceptions in code.
My point is, let it break. Don't pre-emptively drop things that you don't know to be harmful, but where dropping definitely is harmful.
After risk has realised you have more data about the risk. If it has never realised you have no idea. If we extrapolate this culture of fear, we will only have HTTPS open, nothing else, and we have to build then next-gen stuff as an overlay using HTTPS transport.
Sure I get it it's a very valid and a noble point, But what you're asking is let it break (yes potentially -it's just probability until it happens) for 1000s of subs just so that one kiddo has a working niche feature, I can already see what board has to say about that -screw that kid we have money to make there's our brand at stake (yes again just potentially -it's just probability until it actually happens) -but you'll already know what they're gonna say. So yes on a technical level I agree with you, but on a commercial level it's a tough case to make. And the same logic applies to the other thread around BGP communities filtering... adam
On Thu, Mar 7, 2019 at 6:06 PM <adamv0025@netconsultings.com> wrote:
Sure I get it it's a very valid and a noble point, But what you're asking is let it break (yes potentially -it's just probability until it happens) for 1000s of subs just so that one kiddo has a working niche feature, I can already see what board has to say about that -screw that kid we have money to make there's our brand at stake (yes again just potentially -it's just probability until it actually happens) -but you'll already know what they're gonna say. So yes on a technical level I agree with you, but on a commercial level it's a tough case to make.
So why not disable ICMP Echo and UDP traceroute, those kids using network diagnostics don't need them. For clue constrained audience fear will always be the most compelling argument. -- ++ytti
On 3/7/19 8:10 AM, Saku Ytti wrote:
So why not disable ICMP Echo and UDP traceroute, those kids using network diagnostics don't need them.
For clue constrained audience fear will always be the most compelling argument.
OK, OK, so I will continue to rate-limit both, to reasonably high limits on the order of 250/second. Absent a DoS, it allows network operators to use these tools as they should. My logs show no harm except to attack traffic. Everything in moderation.
On Thu, Mar 7, 2019 at 8:33 PM Stephen Satchell <list@satchell.net> wrote:
OK, OK, so I will continue to rate-limit both, to reasonably high limits on the order of 250/second. Absent a DoS, it allows network operators to use these tools as they should.
My logs show no harm except to attack traffic.
Everything in moderation.
Yes this is fair, all untrusted traffic to control-plane should be limited. But the question was, what about ICMP types you don't know about, like timestamps, is it fair to drop those just because we didn't know about them? What are the risks? Essentially timestamp it's just echo which happens to have timestamp from each side, hard to imagine a threat if ICMP echo is not also considered a threat. And massive benefit in diagnostics if you are able tell that issue is unidirectional and which direction is the culprit. Similarly, what about the ICMP type which doesn't exist yet? Should that be victim of our protection mechanism? This is essentially the reason why we can't introduce new L4 protocols, because Internet is full of networks on autopilot with legacy policies where we drop things we don't know about, without having any specific realised threat or way to test if that threat is still valid. And even if there are threats, remember ICMP echo f00fc7c8, +++ or plethora of crash bugs? We still run ICMP echo, because the diagnostic value exceeds the cost of the risk. What tools and protocols are we missing, because we never specify them as we know they would never work in practice? We have beautiful, expressive, extendible protocols and then operational policies which nullify that. We recently had crash bug in pseudowire LSP ping, but it gives us operational benefits so we'll just address the bug and we will continue using the tool. Threat was realised, but it was lower cost than the cost of losing the tool. -- ++ytti
On Tue, Mar 5, 2019 at 07:15 Rich Kulawiec <rsk@gsp.org> wrote:
On Mon, Mar 04, 2019 at 08:04:12AM +1100, Mark Andrews wrote:
ICMP is NOT optional.
I've pointed folks at this for years:
ICMP Packet Filtering v1.2 http://www.cymru.com/Documents/icmp-messages.html
---rsk
Some of the networks that completely block ICMP are shocking. Best, -M<
On 3/3/19 18:04, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers. There are also IDIOTS that deploy load balancers that DO NOT LOOK INSIDE ICMP messages for redirecting ICMP messages to the correct back end. There are also IDOITS that rate limit PTB generation to ridiculously low rates. One should be able to generate PTB at line rate.
Everyone that has configured mss-fix-up has contributed to misunderstanding that you can block ICMP. It is time we had a flag day to REMOVE mss-fix-up from all the boxes you control. We need to get PTB working and unfortunately that means that we need to stop pandering to admins who don’t know how IP is supposed to work. ICMP is NOT optional.
It would seem IETF's intention is to actually move away from ICMPv6-based PMTUD, to the extent that is possible. (RFC4821). Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
On 6 Mar 2019, at 1:30 pm, Fernando Gont <fgont@si6networks.com> wrote:
On 3/3/19 18:04, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers. There are also IDIOTS that deploy load balancers that DO NOT LOOK INSIDE ICMP messages for redirecting ICMP messages to the correct back end. There are also IDOITS that rate limit PTB generation to ridiculously low rates. One should be able to generate PTB at line rate.
Everyone that has configured mss-fix-up has contributed to misunderstanding that you can block ICMP. It is time we had a flag day to REMOVE mss-fix-up from all the boxes you control. We need to get PTB working and unfortunately that means that we need to stop pandering to admins who don’t know how IP is supposed to work. ICMP is NOT optional.
It would seem IETF's intention is to actually move away from ICMPv6-based PMTUD, to the extent that is possible. (RFC4821).
Which is not a reason to not fix broken equipment and misconfigured firewalls. The workarounds are basically there because people deploy broken equipment. Additionally everything isn’t TCP.
Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 6/3/19 01:09, Mark Andrews wrote:
On 6 Mar 2019, at 1:30 pm, Fernando Gont <fgont@si6networks.com> wrote:
On 3/3/19 18:04, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers. There are also IDIOTS that deploy load balancers that DO NOT LOOK INSIDE ICMP messages for redirecting ICMP messages to the correct back end. There are also IDOITS that rate limit PTB generation to ridiculously low rates. One should be able to generate PTB at line rate.
Everyone that has configured mss-fix-up has contributed to misunderstanding that you can block ICMP. It is time we had a flag day to REMOVE mss-fix-up from all the boxes you control. We need to get PTB working and unfortunately that means that we need to stop pandering to admins who don’t know how IP is supposed to work. ICMP is NOT optional.
It would seem IETF's intention is to actually move away from ICMPv6-based PMTUD, to the extent that is possible. (RFC4821).
Which is not a reason to not fix broken equipment and misconfigured firewalls. The workarounds are basically there because people deploy broken equipment.
Agreed. That said, it wasn't solved in 30+ years of IPv4. Do you have hopes it will be different with IPv6? Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
On 6 Mar 2019, at 3:37 pm, Fernando Gont <fgont@si6networks.com> wrote:
On 6/3/19 01:09, Mark Andrews wrote:
On 6 Mar 2019, at 1:30 pm, Fernando Gont <fgont@si6networks.com> wrote:
On 3/3/19 18:04, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers. There are also IDIOTS that deploy load balancers that DO NOT LOOK INSIDE ICMP messages for redirecting ICMP messages to the correct back end. There are also IDOITS that rate limit PTB generation to ridiculously low rates. One should be able to generate PTB at line rate.
Everyone that has configured mss-fix-up has contributed to misunderstanding that you can block ICMP. It is time we had a flag day to REMOVE mss-fix-up from all the boxes you control. We need to get PTB working and unfortunately that means that we need to stop pandering to admins who don’t know how IP is supposed to work. ICMP is NOT optional.
It would seem IETF's intention is to actually move away from ICMPv6-based PMTUD, to the extent that is possible. (RFC4821).
Which is not a reason to not fix broken equipment and misconfigured firewalls. The workarounds are basically there because people deploy broken equipment.
Agreed. That said, it wasn't solved in 30+ years of IPv4. Do you have hopes it will be different with IPv6?
Make a big enough stink and it will get fixed. People just don’t make enough of a stink. Use social media. None of the companies with broken firewalls really want their idiotic practices pointed out in public. Start doing so every time you see it #STUPIDFIREWALL.
Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 6/3/19 03:29, Mark Andrews wrote:
On 6 Mar 2019, at 3:37 pm, Fernando Gont <fgont@si6networks.com> wrote:
On 6/3/19 01:09, Mark Andrews wrote:
On 6 Mar 2019, at 1:30 pm, Fernando Gont <fgont@si6networks.com> wrote:
On 3/3/19 18:04, Mark Andrews wrote:
There are lots of IDIOTS out there that BLOCK ALL ICMP. That blocks PTB getting back to the TCP servers. There are also IDIOTS that deploy load balancers that DO NOT LOOK INSIDE ICMP messages for redirecting ICMP messages to the correct back end. There are also IDOITS that rate limit PTB generation to ridiculously low rates. One should be able to generate PTB at line rate.
Everyone that has configured mss-fix-up has contributed to misunderstanding that you can block ICMP. It is time we had a flag day to REMOVE mss-fix-up from all the boxes you control. We need to get PTB working and unfortunately that means that we need to stop pandering to admins who don’t know how IP is supposed to work. ICMP is NOT optional.
It would seem IETF's intention is to actually move away from ICMPv6-based PMTUD, to the extent that is possible. (RFC4821).
Which is not a reason to not fix broken equipment and misconfigured firewalls. The workarounds are basically there because people deploy broken equipment.
Agreed. That said, it wasn't solved in 30+ years of IPv4. Do you have hopes it will be different with IPv6?
Make a big enough stink and it will get fixed. People just don’t make enough of a stink. Use social media. None of the companies with broken firewalls really want their idiotic practices pointed out in public. Start doing so every time you see it #STUPIDFIREWALL.
At times, it's fw defaults. Other times, it is admin policies. RFC4821 seems to signal that the IETF has given up in making ICMP-based PMTUD work, and aims at a (mostly) ICMP-free PMTUD. Essentially, when brokenness is widespread, you have to come-up with workarounds. And when workarounds are sufficiently widespread, there's less of an incentive to fix the original issue. Other times, there's a disconnect between with protocol standards, products, and operational requirements. That's the case of IPv6 EHs. This is their usability on the public Internet: RFC 7872. And these are some of the reasons why you get the numbers in RFC 7872: https://tools.ietf.org/html/draft-gont-v6ops-ipv6-ehs-packet-drops Cheers, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
On 6/Mar/19 08:38, Fernando Gont wrote:
RFC4821 seems to signal that the IETF has given up in making ICMP-based PMTUD work, and aims at a (mostly) ICMP-free PMTUD.
As much as I hate to admit it, I think this is a more realistic approach. Mark.
On 6/Mar/19 08:29, Mark Andrews wrote:
Make a big enough stink and it will get fixed. People just don’t make enough of a stink. Use social media. None of the companies with broken firewalls really want their idiotic practices pointed out in public. Start doing so every time you see it #STUPIDFIREWALL.
I think the (Inter)network is growing a lot faster than we can shame folk into fixing things. Mark.
I recently go a Linksys home wifi router, by default it enables ipv6 on the LAN. If there is no native IPv6 on the WAN side (which is my case since FiOS doesnt do v6 yet) the Linksys defaults to a v6 tunnel. For the first few weeks of using the router, I had no idea alot of my traffic was going out via the v6 tunnel. Then I started getting random reachability and availability issues. Google would not load, but Bing and Yahoo would, and so on. I thought it was a FiOS issue, but after digging, I discovered the v6 tunnel, disabled it and all my issues went away. I dont know what Linksys uses for the v6 tunnel because its buried in the firmware, but any tunnel service is vulnerable to a variety of issues that could effect access. Its odd that it always effects Windows update all the time, but who knows. -John On 11/12/18 1:18 PM, Mark Tinka wrote:
On 11/Nov/18 18:51, Lavanauts wrote:
I’m on native IPv6 via Spectrum and have no problems with Windows Updates. Could this be a tunneling issue?
I do run 6-in-4 from my backbone to my house as my FTTH provider does not do IPv6.
I can't imagine this to specifically be the issue, as all other IPv6 traffic is fine, but at this point, I'm open to suggestion.
Mark.
Which just shows content providers and tunnel end point problems. * load balancers that don’t properly handle ICMP{v6} * stupid firewalls that block PTB * tunnel end points that don’t generate PTB for EVERY oversize packet (you wouldn’t drop TCP ACKS and PTBs are just as important) PMTD requires PTBs to be generated. Report the problems so they can get fixed. Mark
On 13 Nov 2018, at 6:08 am, John Von Essen <john@essenz.com> wrote:
I recently go a Linksys home wifi router, by default it enables ipv6 on the LAN. If there is no native IPv6 on the WAN side (which is my case since FiOS doesnt do v6 yet) the Linksys defaults to a v6 tunnel.
For the first few weeks of using the router, I had no idea alot of my traffic was going out via the v6 tunnel.
Then I started getting random reachability and availability issues. Google would not load, but Bing and Yahoo would, and so on. I thought it was a FiOS issue, but after digging, I discovered the v6 tunnel, disabled it and all my issues went away. I dont know what Linksys uses for the v6 tunnel because its buried in the firmware, but any tunnel service is vulnerable to a variety of issues that could effect access. Its odd that it always effects Windows update all the time, but who knows.
-John
On 11/12/18 1:18 PM, Mark Tinka wrote:
On 11/Nov/18 18:51, Lavanauts wrote:
I’m on native IPv6 via Spectrum and have no problems with Windows Updates. Could this be a tunneling issue?
I do run 6-in-4 from my backbone to my house as my FTTH provider does not do IPv6.
I can't imagine this to specifically be the issue, as all other IPv6 traffic is fine, but at this point, I'm open to suggestion.
Mark.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
John Von Essen <john@essenz.com> writes:
I recently go a Linksys home wifi router, by default it enables ipv6 on the LAN. If there is no native IPv6 on the WAN side (which is my case since FiOS doesnt do v6 yet) the Linksys defaults to a v6 tunnel.
Could this be a 6RD tunnel requested by your ISP using DHCP with OPTION_6RD? Ref RFC5969 Setting up any tunnel to some pre-configured endpoint by default does not sound like a good idea.... But DHCP on the WAN side is "trusted", so configuring a DHCP requested tunnel by default is reasonable.
For the first few weeks of using the router, I had no idea alot of my traffic was going out via the v6 tunnel.
Then I started getting random reachability and availability issues. Google would not load, but Bing and Yahoo would, and so on. I thought it was a FiOS issue, but after digging, I discovered the v6 tunnel, disabled it and all my issues went away.
I dont know what Linksys uses for the v6 tunnel because its buried in the firmware, but any tunnel service is vulnerable to a variety of issues that could effect access. Its odd that it always effects Windows update all the time, but who knows.
It would be great to have more details about this default tunnel setup. Can't you sniff the traffic? Anyway: Thanks for yet another argument for native dual-stack. Avoiding such unwanted tunnels is really simple: If you're an ISP: Offer native dual-stack to your Internet access customers. If you're an Internet access customer: Request native dual-stack from your ISP Problem solved. Bjørn
I saw this issue randomly on Windows PCs due to IPV6 TCP checksum offloading. Try the following on the problem Windows machine: - Open the Device Manager -> Network Adapters -> Network Interface (Ethernet NIC):- Under the Network Adapter -> Advanced Tab, disable these options if present:TCP Checksum Offloading (IPV6) -> Disabled UDP Checksum Offloading (IPV6) -> Disabled - Try Windows update again -- Clinton Work Airdrie, AB On Sun, Nov 11, 2018, at 2:29 AM, Mark Tinka wrote:
Hi all.
Anyone ever figured out why Windows updates fail when the computer has an IPv6 connection?
Google has tickets and tickets of this to and outside of Microsoft since 2013, with no real solution or answer as to what the problem actually is. In essence, many of the solutions out there point toward making sure the updates do not occur over IPv6, which, in effect, is the same as disabling it.
I have a family PC at home running Windows 10 Pro, and noticed updates would fail in recent months. It took me a moment to realize that this started happening only after I enabled IPv6 in the TCP/IP stack. Disabling it immediately solves the issue.
Quite odd that this is happening in 2018...
Mark.
On 12/Nov/18 18:18, Clinton Work wrote:
I saw this issue randomly on Windows PCs due to IPV6 TCP checksum offloading.
Try the following on the problem Windows machine: - Open the Device Manager -> Network Adapters -> Network Interface (Ethernet NIC): - Under the Network Adapter -> Advanced Tab, disable these options if present: TCP Checksum Offloading (IPV6) -> Disabled UDP Checksum Offloading (IPV6) -> Disabled - Try Windows update again
Thanks, Airdrie, but I don't have those options on the network adapter. Mark.
participants (20)
-
adamv0025＠netconsultings.com -
Bjørn Mork -
Chris Knipe -
Clinton Work -
Daniel Corbe -
Fernando Gont -
Harald Koch -
Jared Mauch -
Jeroen Massar -
John Von Essen -
Lavanauts -
Mark Andrews -
Mark Tinka -
Martin Hannigan -
Mikael Abrahamsson -
Morgan A. Miskell -
Radu-Adrian Feurdean -
Rich Kulawiec -
Saku Ytti -
Stephen Satchell