Hopefully, my pain will help someone else. I've had sporadic Internet slowdowns and stuck networking since IPv6 was enabled on my FIOS ONT a few months ago. After too much troubleshooting, I found out some older Intel GbE ethernet cards have a IPv6 Checksum Offload incompatibility with certain fiber ONT terminals. As Verizon is enabling IPv6 on its FIOS network, you might find intermittent network problems. Intermittent are the worst kind of problems. In some situations where a client machine is connected via some specific Optical Network Terminals (ONTs), and data is appended after the packet checksum, the network adapter can drop receive packets when using TCP-IPv6 Checksum Offload for receive traffic. Intel published an alert in 2017, but I didn't have IPv6 on FIOS then. https://www.intel.com/content/www/us/en/download/19174/disabling-tcp-ipv6-ch... TLDR; turn off TCP IPv6 Checksum Offload Affects all operating systems (Windows, BSD, Linux, etc) using the affected wired Intel ethernet controllers. Not a problem with Intel WiFi.
On 8/27/22 12:00 PM, Sean Donelan wrote:
Hopefully, my pain will help someone else.
I've had sporadic Internet slowdowns and stuck networking since IPv6 was enabled on my FIOS ONT a few months ago.
After too much troubleshooting, I found out some older Intel GbE ethernet cards have a IPv6 Checksum Offload incompatibility with certain fiber ONT terminals. As Verizon is enabling IPv6 on its FIOS network, you might find intermittent network problems.
Intermittent are the worst kind of problems.
In some situations where a client machine is connected via some specific Optical Network Terminals (ONTs), and data is appended after the packet checksum, the network adapter can drop receive packets when using TCP-IPv6 Checksum Offload for receive traffic.
Intel published an alert in 2017, but I didn't have IPv6 on FIOS then.
https://www.intel.com/content/www/us/en/download/19174/disabling-tcp-ipv6-ch...
TLDR; turn off TCP IPv6 Checksum Offload
Affects all operating systems (Windows, BSD, Linux, etc) using the affected wired Intel ethernet controllers. Not a problem with Intel WiFi.
My reaction is "offload from what"? Isn't this all done in silicon? Mike
No. In fact, a lot of low-end Ethernet interfaces are completely implemented in interrupt-driven driver software that runs in the host OS (such as Windows). The only thing the hardware provides is the magnetically to transduce binary bit streams. Even MAC-address decode is in software, and as a result, broadcast storms can slow these hosts to a crawl as the CPU had to check and discard every broadcast packet as “not mine”. When these tasks are offloaded from the CPU to the Ethernet hardware, the CPU doesn’t need to perform these tasks, reducing CPU workload. These also offloading resources provide parallel computing and validation of checksums, which is otherwise computationally expensive. I don’t know how this particular ONT bug works, but I’m guessing that it results in checksum failures under certain conditions, leading to retransmissions. -mel via cell
On Aug 27, 2022, at 3:08 PM, Michael Thomas <mike@mtcc.com> wrote:
On 8/27/22 12:00 PM, Sean Donelan wrote: Hopefully, my pain will help someone else.
I've had sporadic Internet slowdowns and stuck networking since IPv6 was enabled on my FIOS ONT a few months ago.
After too much troubleshooting, I found out some older Intel GbE ethernet cards have a IPv6 Checksum Offload incompatibility with certain fiber ONT terminals. As Verizon is enabling IPv6 on its FIOS network, you might find intermittent network problems.
Intermittent are the worst kind of problems.
In some situations where a client machine is connected via some specific Optical Network Terminals (ONTs), and data is appended after the packet checksum, the network adapter can drop receive packets when using TCP-IPv6 Checksum Offload for receive traffic.
Intel published an alert in 2017, but I didn't have IPv6 on FIOS then.
https://www.intel.com/content/www/us/en/download/19174/disabling-tcp-ipv6-ch...
TLDR; turn off TCP IPv6 Checksum Offload
Affects all operating systems (Windows, BSD, Linux, etc) using the affected wired Intel ethernet controllers. Not a problem with Intel WiFi.
My reaction is "offload from what"? Isn't this all done in silicon?
Mike
On 8/27/22 3:36 PM, Mel Beckman wrote:
No. In fact, a lot of low-end Ethernet interfaces are completely implemented in interrupt-driven driver software that runs in the host OS (such as Windows). The only thing the hardware provides is the magnetically to transduce binary bit streams.
Even MAC-address decode is in software, and as a result, broadcast storms can slow these hosts to a crawl as the CPU had to check and discard every broadcast packet as “not mine”.
When these tasks are offloaded from the CPU to the Ethernet hardware, the CPU doesn’t need to perform these tasks, reducing CPU workload. These also offloading resources provide parallel computing and validation of checksums, which is otherwise computationally expensive.
I don’t know how this particular ONT bug works, but I’m guessing that it results in checksum failures under certain conditions, leading to retransmissions.
Yeah, sorry brain fart. I'd be surprised if that were a big issue on home networks, but who knows. Mike
-mel via cell
On Aug 27, 2022, at 3:08 PM, Michael Thomas <mike@mtcc.com> wrote:
On 8/27/22 12:00 PM, Sean Donelan wrote: Hopefully, my pain will help someone else.
I've had sporadic Internet slowdowns and stuck networking since IPv6 was enabled on my FIOS ONT a few months ago.
After too much troubleshooting, I found out some older Intel GbE ethernet cards have a IPv6 Checksum Offload incompatibility with certain fiber ONT terminals. As Verizon is enabling IPv6 on its FIOS network, you might find intermittent network problems.
Intermittent are the worst kind of problems.
In some situations where a client machine is connected via some specific Optical Network Terminals (ONTs), and data is appended after the packet checksum, the network adapter can drop receive packets when using TCP-IPv6 Checksum Offload for receive traffic.
Intel published an alert in 2017, but I didn't have IPv6 on FIOS then.
https://www.intel.com/content/www/us/en/download/19174/disabling-tcp-ipv6-ch...
TLDR; turn off TCP IPv6 Checksum Offload
Affects all operating systems (Windows, BSD, Linux, etc) using the affected wired Intel ethernet controllers. Not a problem with Intel WiFi. My reaction is "offload from what"? Isn't this all done in silicon?
Mike
On Sat, 27 Aug 2022, Michael Thomas wrote:
In some situations where a client machine is connected via some specific Optical Network Terminals (ONTs), and data is appended after the packet checksum, the network adapter can drop receive packets when using TCP-IPv6 Checksum Offload for receive traffic.
My reaction is "offload from what"? Isn't this all done in silicon?
Because the interoperability flaw is in silicon, it can't be easily fixed in either the legacy wired Intel ethernet controller or fiber ONT. Would need to replace the hardware to fix the silicon. Need to disable the hardware IPv6 TCP checksum offload, so its not mangled or dropped at the silicon layers anymore. Its annoyingly intermittent and not visible with client-based Wireshark because the corruption occurs in the hardware controller.
On Mon, Aug 29, 2022 at 12:00 PM Sean Donelan <sean@donelan.com> wrote:
On Sat, 27 Aug 2022, Michael Thomas wrote:
In some situations where a client machine is connected via some specific Optical Network Terminals (ONTs), and data is appended after the packet checksum, the network adapter can drop receive packets when using TCP-IPv6 Checksum Offload for receive traffic.
My reaction is "offload from what"? Isn't this all done in silicon?
Because the interoperability flaw is in silicon, it can't be easily fixed in either the legacy wired Intel ethernet controller or fiber ONT. Would need to replace the hardware to fix the silicon.
Need to disable the hardware IPv6 TCP checksum offload, so its not mangled or dropped at the silicon layers anymore.
Its annoyingly intermittent and not visible with client-based Wireshark because the corruption occurs in the hardware controller.
Uhm, this includes various versions of the intel pro 1000 card... so that's a TON of gear, to include like lenovo laptops, for instance. I'd wager that this is super common in the field. The PDF in the download says; "Products Affected: All 1gbe and 10gbe intel ethernet controllers...." One wonders if this is a case of the 'mac addresses that start with 4 or 6 fail' problem? (the pdf has zero words about what the actual problem is)
On 8/29/22 10:59 AM, Christopher Morrow wrote:
Uhm, this includes various versions of the intel pro 1000 card... so that's a TON of gear, to include like lenovo laptops, for instance. I'd wager that this is super common in the field. The PDF in the download says; "Products Affected: All 1gbe and 10gbe intel ethernet controllers...."
So I keep seeing this being pushed as a problem with clients... but isn't it the ONT that is bugged out and appending the extra data after the checksum is already there (as Bill Herrin points out)? I know it's asking a lot to expect networking equipment vendors to fix their gear, but... Unless I'm totally not understanding the bug, which is entirely (and likely). -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
* bruns@2mbit.com (Brie) [Mon 29 Aug 2022, 19:38 CEST]:
On 8/29/22 10:59 AM, Christopher Morrow wrote:
Uhm, this includes various versions of the intel pro 1000 card... so that's a TON of gear, to include like lenovo laptops, for instance. I'd wager that this is super common in the field. The PDF in the download says; "Products Affected: All 1gbe and 10gbe intel ethernet controllers...."
So I keep seeing this being pushed as a problem with clients... but isn't it the ONT that is bugged out and appending the extra data after the checksum is already there (as Bill Herrin points out)?
I know it's asking a lot to expect networking equipment vendors to fix their gear, but...
Unless I'm totally not understanding the bug, which is entirely (and likely).
Here's my speculation on what was happening at Casa Sean. The Ethernet frame has a length header. The IP frame has a length header. Ideally, the IP frame fits completely into the Ethernet frame, leaving room for the other required Ethernet bits but nothing more. I vaguely recall there being some equipment that interpreted the minimum MTU requirement in IPv6 as meaning that there was a minimum packet size, not a minimum for the *maximum* packet size. Perhaps the fiber NTU padded the Ethernet frame up to the minimum MTU, sending along a bunch of junk bytes, without otherwise touching the IP packet. The NIC would then perhaps forget that there were a bunch of junk bytes attached to the end of the frame beyond where the IP packet would end, and calculate the Ethernet checksum based on the IP packet length header, discarding otherwise valid frames as a result. I've had my own run-ins over the years with supposed checksum offloading absolutely not happening on other brands, so implementation errors appear to be relatively common. -- Niels.
On Sat, Aug 27, 2022 at 12:00 PM Sean Donelan <sean@donelan.com> wrote:
In some situations where a client machine is connected via some specific Optical Network Terminals (ONTs), and data is appended after the packet checksum, the network adapter can drop receive packets when using TCP-IPv6 Checksum Offload for receive traffic.
Hi Sean, Do you happen to have any details on the bug? I note that the IPv6 header DOES NOT HAVE a checksum; it relies on the checksum in the layer-2 frame. I'm not clear how you "append" bytes to the ethernet frame "after" calculating the checksum, have the hardware checksum fail but have the software checksum succeed. Regards, Bill Herrin -- For hire. https://bill.herrin.us/resume/
participants (7)
-
Brie -
Christopher Morrow -
Mel Beckman -
Michael Thomas -
Niels Bakker -
Sean Donelan -
William Herrin