First of all, kudos to Comcast for trying to roll out IPv6 across their entire network. Static IPv6 netblocks seem to be available for Comcast business users, and IPv6 is enabled unconditionally in the CPE routers used by Comcast business class internet. Unfortunately, the software in the two available CPE routers (SMC & Cisco) is horribly broken when it comes to IPv6. The TL;DR summary: even when IPv6 firewalling is disabled in the configuration, the router still tracks every IPv6 "connection", which causes every single DNS lookup to fill up a slot in its connection tracking table. The router's logs say it blocks tens of thousands of IPv6 connections every day, despite firewalling being "disabled" on the router. Once the connection tracking table fills up, both IPv6 and IPv4 start having trouble, with packet loss on ICMP, high ping times to the local router (and the internet), and new connections not establishing. The router randomly crashes and reboots too, sometimes multiple times a day. This ends up breaking both IPv6 and IPv4. It only takes about 300kbit/s of DNS traffic to trigger the bug, in both the SMC and the Cisco routers. Are there any Comcast NOC or other technical people present who could help? I am interested both in helping resolve the firmware issues in the routers (there will no doubt be other customers who hit this in the future, as IPv6 becomes ore common) or, if that is not an option, finding some way to avoid the issue. http://forums.businesshelp.comcast.com/t5/Equipment-Modems-Gateways/Cis co-DPC3941B-slows-to-a-crawl-and-crashes-several-times-a-day/td-p/30807 -- All Rights Reversed.
I can send it along to folks here at Comcast. - Jason On 11/28/16, 1:46 PM, "NANOG on behalf of Rik van Riel" <nanog-bounces@nanog.org on behalf of riel@surriel.com> wrote: First of all, kudos to Comcast for trying to roll out IPv6 across their entire network. Static IPv6 netblocks seem to be available for Comcast business users, and IPv6 is enabled unconditionally in the CPE routers used by Comcast business class internet. Unfortunately, the software in the two available CPE routers (SMC & Cisco) is horribly broken when it comes to IPv6. The TL;DR summary: even when IPv6 firewalling is disabled in the configuration, the router still tracks every IPv6 "connection", which causes every single DNS lookup to fill up a slot in its connection tracking table. The router's logs say it blocks tens of thousands of IPv6 connections every day, despite firewalling being "disabled" on the router. Once the connection tracking table fills up, both IPv6 and IPv4 start having trouble, with packet loss on ICMP, high ping times to the local router (and the internet), and new connections not establishing. The router randomly crashes and reboots too, sometimes multiple times a day. This ends up breaking both IPv6 and IPv4. It only takes about 300kbit/s of DNS traffic to trigger the bug, in both the SMC and the Cisco routers. Are there any Comcast NOC or other technical people present who could help? I am interested both in helping resolve the firmware issues in the routers (there will no doubt be other customers who hit this in the future, as IPv6 becomes ore common) or, if that is not an option, finding some way to avoid the issue. http://forums.businesshelp.comcast.com/t5/Equipment-Modems-Gateways/Cis co-DPC3941B-slows-to-a-crawl-and-crashes-several-times-a-day/td-p/30807 -- All Rights Reversed.
I concur with the kudos bit, but I'll also concur that the CPE support appears to be limited. Another example: IPv6 prefix delegation is broken on the SMCD3G-CCR, and according to the following threads: http://www.gossamer-threads.com/lists/nsp/ipv6/54761 (scroll down to the IPv6 OPERATIONS - BUSINESS section) http://forums.businesshelp.comcast.com/t5/IPV6/Dual-Stack-on-SMC-D3GCCR-and-... ... others have the same issue and there isn't much of an incentive to fix it. When I asked if I could use my own CPE, I was told no, because I'm a "business customer", which is a requirement if you want static v4 IPs. Anyone have any success with a different model CPE and Comcast v6? I love that they hand out a /56 by default, but it's not of much use if I can only use a single /64. - bryan On 11/29/16 11:45 AM, Livingood, Jason wrote:
I can send it along to folks here at Comcast.
- Jason
On 11/28/16, 1:46 PM, "NANOG on behalf of Rik van Riel" <nanog-bounces@nanog.org on behalf of riel@surriel.com> wrote:
First of all, kudos to Comcast for trying to roll out IPv6 across their entire network. Static IPv6 netblocks seem to be available for Comcast business users, and IPv6 is enabled unconditionally in the CPE routers used by Comcast business class internet.
Unfortunately, the software in the two available CPE routers (SMC & Cisco) is horribly broken when it comes to IPv6.
The TL;DR summary: even when IPv6 firewalling is disabled in the configuration, the router still tracks every IPv6 "connection", which causes every single DNS lookup to fill up a slot in its connection tracking table.
The router's logs say it blocks tens of thousands of IPv6 connections every day, despite firewalling being "disabled" on the router.
Once the connection tracking table fills up, both IPv6 and IPv4 start having trouble, with packet loss on ICMP, high ping times to the local router (and the internet), and new connections not establishing. The router randomly crashes and reboots too, sometimes multiple times a day.
This ends up breaking both IPv6 and IPv4.
It only takes about 300kbit/s of DNS traffic to trigger the bug, in both the SMC and the Cisco routers.
Are there any Comcast NOC or other technical people present who could help?
I am interested both in helping resolve the firmware issues in the routers (there will no doubt be other customers who hit this in the future, as IPv6 becomes ore common) or, if that is not an option, finding some way to avoid the issue.
http://forums.businesshelp.comcast.com/t5/Equipment-Modems-Gateways/Cis co-DPC3941B-slows-to-a-crawl-and-crashes-several-times-a-day/td-p/30807
-- All Rights Reversed.
Folks at Comcast have told me to ask for the SMC gateway to be replaced with either the netgear or Cisco to solve that issue. Jared Mauch
On Nov 29, 2016, at 1:28 PM, Bryan Holloway <bryan@shout.net> wrote:
I concur with the kudos bit, but I'll also concur that the CPE support appears to be limited. Another example: IPv6 prefix delegation is broken on the SMCD3G-CCR, and according to the following threads:
http://www.gossamer-threads.com/lists/nsp/ipv6/54761 (scroll down to the IPv6 OPERATIONS - BUSINESS section)
http://forums.businesshelp.comcast.com/t5/IPV6/Dual-Stack-on-SMC-D3GCCR-and-...
... others have the same issue and there isn't much of an incentive to fix it.
When I asked if I could use my own CPE, I was told no, because I'm a "business customer", which is a requirement if you want static v4 IPs.
Anyone have any success with a different model CPE and Comcast v6? I love that they hand out a /56 by default, but it's not of much use if I can only use a single /64.
- bryan
On 11/29/16 11:45 AM, Livingood, Jason wrote: I can send it along to folks here at Comcast.
- Jason
On 11/28/16, 1:46 PM, "NANOG on behalf of Rik van Riel" <nanog-bounces@nanog.org on behalf of riel@surriel.com> wrote:
First of all, kudos to Comcast for trying to roll out IPv6 across their entire network. Static IPv6 netblocks seem to be available for Comcast business users, and IPv6 is enabled unconditionally in the CPE routers used by Comcast business class internet.
Unfortunately, the software in the two available CPE routers (SMC & Cisco) is horribly broken when it comes to IPv6.
The TL;DR summary: even when IPv6 firewalling is disabled in the configuration, the router still tracks every IPv6 "connection", which causes every single DNS lookup to fill up a slot in its connection tracking table.
The router's logs say it blocks tens of thousands of IPv6 connections every day, despite firewalling being "disabled" on the router.
Once the connection tracking table fills up, both IPv6 and IPv4 start having trouble, with packet loss on ICMP, high ping times to the local router (and the internet), and new connections not establishing. The router randomly crashes and reboots too, sometimes multiple times a day.
This ends up breaking both IPv6 and IPv4.
It only takes about 300kbit/s of DNS traffic to trigger the bug, in both the SMC and the Cisco routers.
Are there any Comcast NOC or other technical people present who could help?
I am interested both in helping resolve the firmware issues in the routers (there will no doubt be other customers who hit this in the future, as IPv6 becomes ore common) or, if that is not an option, finding some way to avoid the issue.
http://forums.businesshelp.comcast.com/t5/Equipment-Modems-Gateways/Cis co-DPC3941B-slows-to-a-crawl-and-crashes-several-times-a-day/td-p/30807
-- All Rights Reversed.
On Tue, 2016-11-29 at 13:34 -0500, Jared Mauch wrote:
Folks at Comcast have told me to ask for the SMC gateway to be replaced with either the netgear or Cisco to solve that issue.
Over the past year and a bit, I have had all three of the Comcast business routers in my network. The Netgear only stayed for one day - after about 10-15 minutes of "heavy" (~300kbit/s) DNS lookups coming in from the outside, it was almost impossible to make new TCP connections across the router, either IPv4 or IPv6. The SMC D3G-CCR mostly worked, except at some point during the year, the fraction of traffic going over IPv6 went high enough to wreck the D3G, causing it to crash and reboot several times a day, without having enough diagnostics for me to figure out what was going on. The Cisco DPC3941B seems to fail in pretty much the same way as the SMC D3G-CCR, but it has enough diagnostics that I could finally figure out what was happening. With "Gateway Smart Packet Detection" disabled, and the "Firewall completely disabled", the logs are still showing tens of thousands of dropped IPv6 connections every day. In other words, the config options that supposedly disable the firewall completely, do not in fact disable the firewall code, and I am still hitting connection tracking limits. DNS lookups coming from randomized port numbers (to avoid spoofing issues) mean every DNS query takes up another slot in the connection tracking table. Once the table is full, the router will search for a re-usable slot before routing a packet. This can cause ping times to 10.1.10.1 (the router) to go as high as 800ms. This is from a system sitting 5ft from the router. If the router does not find any re-usable slot in the connection tracking table, packets can get lost. This leads to the "fun" scenario where pinging the router from a system directly connected to it shows 30% packet loss, while streaming video over an already established TCP stream continues at full speed! Not a symptom I ever expected to see... -- All rights reversed
Because if you want static IPs from them you must rent one of the following. Cisco DPC3939B or DPC3941B Netgear CG3000DCR SMC Networks SMCD3G Luke Guillory Network Operations Manager Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com Reserve Telecommunications 100 RTC Dr Reserve, LA 70084 _________________________________________________________________________________________________ Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. . -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Randy Bush Sent: Tuesday, November 29, 2016 1:41 PM To: Rik van Riel Cc: North American Network Operators' Group Subject: Re: Comcast business IPv6 vs rbldnsd & PSBL i am running my own (why rent at silly costs) dpc3008 and wfm. randy
Not to mention that they "raised my rent" a few months ago by $5/mo, which is pretty ludicrous considering that a) it doesn't actually work as advertised, and b) it probably cost them $20-30 to purchase those SMCs wholesale in the first place. They've made their money on my CPE many many times over. But that's just the way it is. On 11/29/16 1:48 PM, Luke Guillory wrote:
Because if you want static IPs from them you must rent one of the following.
Cisco DPC3939B or DPC3941B Netgear CG3000DCR SMC Networks SMCD3G
Luke Guillory Network Operations Manager
Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com
Reserve Telecommunications 100 RTC Dr Reserve, LA 70084
_________________________________________________________________________________________________
Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. .
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Randy Bush Sent: Tuesday, November 29, 2016 1:41 PM To: Rik van Riel Cc: North American Network Operators' Group Subject: Re: Comcast business IPv6 vs rbldnsd & PSBL
i am running my own (why rent at silly costs) dpc3008 and wfm.
randy
To clarify, you cannot rent AND have static IP's. You can rent your own modem ofr business service when using dynamic IP's. Robert Webb On Tue, 29 Nov 2016 15:07:52 -0500 Jared Mauch <jared@puck.nether.net> wrote:
Can't do that with the business service. Oh well, to have choices.
Jared Mauch
On Nov 29, 2016, at 2:40 PM, Randy Bush <randy@psg.com> wrote:
i am running my own (why rent at silly costs) dpc3008 and wfm.
randy
That's true - I had one of the SMC routers for many years when I had static Business HSI service, and switched earlier this year to using a off the shelf Arris (ex Motorola) Surfboard modems and dynamic IP on my BHSI service... my IPv6 service has never been better. :) Unless you have a static IP configuration - As long as it's on Comcast's approved modem list they don't care what modem you use even if it's on their business class service. Best Wishes - Peter On Tue, Nov 29, 2016 at 1:18 PM, <rwebb@ropeguru.com> wrote:
To clarify, you cannot rent AND have static IP's.
You can rent your own modem ofr business service when using dynamic IP's.
Robert Webb
On Tue, 29 Nov 2016 15:07:52 -0500 Jared Mauch <jared@puck.nether.net> wrote:
Can't do that with the business service. Oh well, to have choices. Jared Mauch
On Nov 29, 2016, at 2:40 PM, Randy Bush <randy@psg.com> wrote:
i am running my own (why rent at silly costs) dpc3008 and wfm.
randy
-- [ http://blog.plosh.net ] - "Earth Halted: Please reboot to continue"
On Tue, 29 Nov 2016, Rik van Riel wrote:
Not a symptom I ever expected to see...
It's pretty obvious that the CPEs being sold for this "business service" isn't meant for the kind of service you run. They're probably doing connection tracking for ACK optimization, this should not be done for UDP but it's still being done. They probably have a connection limit of a few thousand connections (not uncommon for these kinds of devices) and it's not possible to turn off what you need to turn off to make them work correctly. Do you have any other options in your area for other ISPs that can offer a better service for you? Otherwise you might hack around it by running an IPSEC/UDP tunnel to somewhere else where there isn't this kind of connection limit. -- Mikael Abrahamsson email: swmike@swm.pp.se
participants (9)
-
Bryan Holloway -
Jared Mauch -
Livingood, Jason -
Luke Guillory -
Mikael Abrahamsson -
Peter Losher -
Randy Bush -
Rik van Riel -
rwebb＠ropeguru.com