Hey all, I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done. In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year. -- Tom
We’ve been having IPv6 issues for weeks in Texas on Spectrum, we have an Enterprise support ticket open on our IP transit circuits but I’ve seen no real movement. Our issue is being tracked internally as INC000026440632, but I believe it’s specific to the DFW area. Daniel Marks d@nielmarks.com
On Apr 26, 2023, at 11:52, Tom Rini <trini@konsulko.com> wrote: Hey all,
I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done.
In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year.
-- Tom
If it is helpful, I am on Spectrum in Ohio and I can reach that address from my IPv6 test system. This is on a commercial account with a provider assigned network. My system is at ip6echo.net. You are welcome to ping and traceroute to it if that is helpful to you. A friend CANNOT get to that address from a Spectrum residential account also in my area. Brandon -----Original Message----- From: NANOG <nanog-bounces+saundeb1=ohio.edu@nanog.org> On Behalf Of Tom Rini Sent: Tuesday, April 25, 2023 2:13 PM To: nanog@nanog.org Subject: [External] Spectrum networks IPv6 access issue Use caution with links and attachments. Hey all, I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done. In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year. -- Tom
For what it's worth, on Spectrum Residential here in LAX and I can reach ip6echo.net and Google's public ipv6.google.com sites via residential service... I have my own modems and routers, which may make the difference in my case... Spectrum does indicate that IPv6 my need to be enabled on their routers... https://www.spectrum.net/support/internet/ipv6-faq jeffp@jeffp-Linux:~$ ping -6 ip6echo.net PING ip6echo.net(ip6echo.net (2600:5c01:f876:10::53)) 56 data bytes 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=1 ttl=51 time=76.1 ms 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=2 ttl=51 time=78.4 ms 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=3 ttl=51 time=76.5 ms jeffp@jeffp-Linux:~$ ping -6 ipv6.google.com PING ipv6.google.com(lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e)) 56 data bytes 64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=1 ttl=116 time=12.6 ms 64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=2 ttl=116 time=14.4 ms I normally use the Google IPv6 site for such tests to avoid irritating private hosts, so a big thanks to Brandon for offering that alternative! Jeffp On 4/26/23 10:46, Saunders, Brandon wrote:
If it is helpful, I am on Spectrum in Ohio and I can reach that address from my IPv6 test system. This is on a commercial account with a provider assigned network. My system is at ip6echo.net. You are welcome to ping and traceroute to it if that is helpful to you.
A friend CANNOT get to that address from a Spectrum residential account also in my area.
Brandon
-----Original Message----- From: NANOG<nanog-bounces+saundeb1=ohio.edu@nanog.org> On Behalf Of Tom Rini Sent: Tuesday, April 25, 2023 2:13 PM To:nanog@nanog.org Subject: [External] Spectrum networks IPv6 access issue
Use caution with links and attachments.
Hey all,
I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done.
In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year.
-- Tom
-- JeffP "It is not for me to place expectations on someone and fail them for not meeting them. It's my goal to encourage others to raise their expectations of themselves, to educate them and help them gain the skills needed to meet their goals and encourage them to succeed in attaining them."
I’m in the northeast and I can confirm on a Spectrum Enterprise connection I have issues tracerouting to 2604:1380:4641:c500::1. Gets to e0-33.core1.nyc7.he.net and then packets just disappear: Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 108 1.0 1.7 0.8 26.3 3.3 2. 2602:XXX:XXXX:X::X 0.0% 108 4.4 6.6 4.2 27.9 4.0 3. 2602:XXX:XXXX:X::XXX 0.0% 108 3.1 3.0 3.0 3.5 0.0 4. 2604:XXXX:X:X::X:XXXX 0.0% 108 5.2 4.8 3.3 37.1 5.0 5. lag-15.srspny0101h.netops.charter.com 0.0% 108 3.9 20.8 3.6 371.6 45.4 6. lag-32.albynyyf02r.netops.charter.com 44.4% 108 4.9 4.9 4.8 5.5 0.1 7. lag-26.rcr01albynyyf.netops.charter.com 80.4% 108 5.1 5.3 5.0 7.7 0.6 8. lag-416.nycmny837aw-bcr00.netops.charter.com 57.9% 108 37.4 14.5 12.7 54.4 7.1 9. lag-1.pr2.nyc20.netops.charter.com 0.0% 108 49.8 17.1 12.3 65.7 11.0 10. e0-33.core1.nyc7.he.net 0.0% 108 12.9 13.2 12.8 19.8 0.7 11. (waiting for reply) Trying from another network and I was able to get through: Host Loss% Snt Last Avg Best Wrst StDev 1. ca.tor.ix.caramelfox.net 0.0% 25 45.7 45.9 43.9 50.9 2.0 2. (waiting for reply) 3. (waiting for reply) 4. ve955.core2.stl1.he.net 56.0% 25 62.0 63.9 62.0 67.6 2.0 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. equinix-ix.dfw2.packet.net 0.0% 25 81.2 86.0 77.8 103.3 8.4 9. (waiting for reply) 10. (waiting for reply) 11. dfw.source.kernel.org 0.0% 24 77.6 78.1 77.0 82.5 1.1 Maybe something in Hurricane Electric’s network potentially not getting these packets where they need to go?
On Apr 26, 2023, at 5:27 PM, Jeff P <jeffp@jeffp.us> wrote:
For what it's worth, on Spectrum Residential here in LAX and I can reach ip6echo.net and Google's public ipv6.google.com sites via residential service... I have my own modems and routers, which may make the difference in my case... Spectrum does indicate that IPv6 my need to be enabled on their routers...
https://www.spectrum.net/support/internet/ipv6-faq
jeffp@jeffp-Linux:~$ ping -6 ip6echo.net PING ip6echo.net(ip6echo.net (2600:5c01:f876:10::53)) 56 data bytes 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=1 ttl=51 time=76.1 ms 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=2 ttl=51 time=78.4 ms 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=3 ttl=51 time=76.5 ms
jeffp@jeffp-Linux:~$ ping -6 ipv6.google.com PING ipv6.google.com(lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e)) 56 data bytes 64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=1 ttl=116 time=12.6 ms 64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=2 ttl=116 time=14.4 ms
I normally use the Google IPv6 site for such tests to avoid irritating private hosts, so a big thanks to Brandon for offering that alternative!
Jeffp
On 4/26/23 10:46, Saunders, Brandon wrote:
If it is helpful, I am on Spectrum in Ohio and I can reach that address from my IPv6 test system. This is on a commercial account with a provider assigned network. My system is at ip6echo.net. You are welcome to ping and traceroute to it if that is helpful to you.
A friend CANNOT get to that address from a Spectrum residential account also in my area.
Brandon
-----Original Message----- From: NANOG <nanog-bounces+saundeb1=ohio.edu@nanog.org> <mailto:nanog-bounces+saundeb1=ohio.edu@nanog.org> On Behalf Of Tom Rini Sent: Tuesday, April 25, 2023 2:13 PM To: nanog@nanog.org <mailto:nanog@nanog.org> Subject: [External] Spectrum networks IPv6 access issue
Use caution with links and attachments.
Hey all,
I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done.
In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year.
-- Tom -- JeffP
"It is not for me to place expectations on someone and fail them for not meeting them. It's my goal to encourage others to raise their expectations of themselves, to educate them and help them gain the skills needed to meet their goals and encourage them to succeed in attaining them."
Just adding another data point of a failure from Spectrum IPv6 to that host in question.
From here in Milledgeville Georgia that host seems to be unreachable, can I get to hurricane electrics network and seems to die with them.
Apologies for this output but I'm just using an app that I already had on my phone. Didn't have a PC available.. #1 - RTT [ms]: 7.8 - Probe Send Time: 21:26:22 - IP Address: 2600:6c5a:47f:xxxx:9e53:22ff:fe95:d858 - TTL: 64 - AS Number: AS20115 - AS Name: CHARTER-20115 - Country Name: United States - Country Code: US - Time Zone: America/New_York - Region: GA - City: Milledgeville - Latitude: 33.08 - Longitude: -83.24 #2 - Probe Send Time: 21:26:23 #3 - RTT [ms]: 16.5 - Probe Send Time: 21:26:26 - IP Address: 2001:506:100:548f::4 - Hostname: lag-22-10.dtr01mdvlga.netops.charter.com - TTL: 62 - Country Name: United States - Country Code: US - Time Zone: America/New_York - Region: GA - City: Columbus - Latitude: 32.47 - Longitude: -84.98 #4 - Probe Send Time: 21:26:26 #5 - RTT [ms]: 21.1 - Probe Send Time: 21:26:29 - IP Address: 2001:506:100:5507::4 - Hostname: lag-20.rcr01sghlgaao.netops.charter.com - TTL: 59 - Country Name: United States - Country Code: US - Time Zone: America/New_York - Region: GA - City: Sugar Hill - Latitude: 34.11 - Longitude: -84.00 #6 - Probe Send Time: 21:26:29 #7 - Probe Send Time: 21:26:32 #8 - Probe Send Time: 21:26:35 #9 - RTT [ms]: 35.3 - Probe Send Time: 21:26:38 - IP Address: 2001:470:0:42a::2 - Hostname: e0-35.core2.bna1.he.net - TTL: 56 - AS Number: AS6939 - AS Name: HURRICANE - Country Name: United States - Country Code: US - Time Zone: America/Chicago #10 - Probe Send Time: 21:26:38 #11 - Probe Send Time: 21:26:41 #12 - Probe Send Time: 21:26:44 #13 - Probe Send Time: 21:26:47 #14 - Probe Send Time: 21:26:50 #15 - Probe Send Time: 21:26:53 #16 - Probe Send Time: 21:26:56 #17 - Probe Send Time: 21:27:00 #18 - Probe Send Time: 21:27:03 On Thu, Apr 27, 2023, 09:48 Francis via NANOG <nanog@nanog.org> wrote:
I’m in the northeast and I can confirm on a Spectrum Enterprise connection I have issues tracerouting to 2604:1380:4641:c500::1.
Gets to e0-33.core1.nyc7.he.net and then packets just disappear:
Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 108 1.0 1.7 0.8 26.3 3.3 2. 2602:XXX:XXXX:X::X 0.0% 108 4.4 6.6 4.2 27.9 4.0 3. 2602:XXX:XXXX:X::XXX 0.0% 108 3.1 3.0 3.0 3.5 0.0 4. 2604:XXXX:X:X::X:XXXX 0.0% 108 5.2 4.8 3.3 37.1 5.0 5. lag-15.srspny0101h.netops.charter.com 0.0% 108 3.9 20.8 3.6 371.6 45.4 6. lag-32.albynyyf02r.netops.charter.com 44.4% 108 4.9 4.9 4.8 5.5 0.1 7. lag-26.rcr01albynyyf.netops.charter.com 80.4% 108 5.1 5.3 5.0 7.7 0.6 8. lag-416.nycmny837aw-bcr00.netops.charter.com 57.9% 108 37.4 14.5 12.7 54.4 7.1 9. lag-1.pr2.nyc20.netops.charter.com 0.0% 108 49.8 17.1 12.3 65.7 11.0 10. e0-33.core1.nyc7.he.net 0.0% 108 12.9 13.2 12.8 19.8 0.7 11. (waiting for reply)
Trying from another network and I was able to get through:
Host Loss% Snt Last Avg Best Wrst StDev 1. ca.tor.ix.caramelfox.net 0.0% 25 45.7 45.9 43.9 50.9 2.0 2. (waiting for reply) 3. (waiting for reply) 4. ve955.core2.stl1.he.net 56.0% 25 62.0 63.9 62.0 67.6 2.0 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. equinix-ix.dfw2.packet.net 0.0% 25 81.2 86.0 77.8 103.3 8.4 9. (waiting for reply) 10. (waiting for reply) 11. dfw.source.kernel.org 0.0% 24 77.6 78.1 77.0 82.5 1.1
Maybe something in Hurricane Electric’s network potentially not getting these packets where they need to go?
On Apr 26, 2023, at 5:27 PM, Jeff P <jeffp@jeffp.us> wrote:
For what it's worth, on Spectrum Residential here in LAX and I can reach ip6echo.net and Google's public ipv6.google.com sites via residential service... I have my own modems and routers, which may make the difference in my case... Spectrum does indicate that IPv6 my need to be enabled on their routers...
https://www.spectrum.net/support/internet/ipv6-faq
jeffp@jeffp-Linux:~$ ping -6 ip6echo.net PING ip6echo.net(ip6echo.net (2600:5c01:f876:10::53)) 56 data bytes 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=1 ttl=51 time=76.1 ms 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=2 ttl=51 time=78.4 ms 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=3 ttl=51 time=76.5 ms
jeffp@jeffp-Linux:~$ ping -6 ipv6.google.com PING ipv6.google.com(lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e)) 56 data bytes 64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=1 ttl=116 time=12.6 ms 64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=2 ttl=116 time=14.4 ms
I normally use the Google IPv6 site for such tests to avoid irritating private hosts, so a big thanks to Brandon for offering that alternative!
Jeffp On 4/26/23 10:46, Saunders, Brandon wrote:
If it is helpful, I am on Spectrum in Ohio and I can reach that address from my IPv6 test system. This is on a commercial account with a provider assigned network. My system is at ip6echo.net. You are welcome to ping and traceroute to it if that is helpful to you.
A friend CANNOT get to that address from a Spectrum residential account also in my area.
Brandon
-----Original Message----- From: NANOG <nanog-bounces+saundeb1=ohio.edu@nanog.org> <nanog-bounces+saundeb1=ohio.edu@nanog.org> On Behalf Of Tom Rini Sent: Tuesday, April 25, 2023 2:13 PM To: nanog@nanog.org Subject: [External] Spectrum networks IPv6 access issue
Use caution with links and attachments.
Hey all,
I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done.
In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year.
-- Tom
-- JeffP
"It is not for me to place expectations on someone and fail them for not meeting them. It's my goal to encourage others to raise their expectations of themselves, to educate them and help them gain the skills needed to meet their goals and encourage them to succeed in attaining them."
you have permission to traverse this host but not talk to it. On Thu, Apr 27, 2023 at 09:31:01PM -0400, Brandon Jackson wrote:
Just adding another data point of a failure from Spectrum IPv6 to that host in question. From here in Milledgeville Georgia that host seems to be unreachable, can I get to hurricane electrics network and seems to die with them. Apologies for this output but I'm just using an app that I already had on my phone. Didn't have a PC available.. #1 - RTT [ms]: 7.8 - Probe Send Time: 21:26:22 - IP Address: 2600:6c5a:47f:xxxx:9e53:22ff:fe95:d858 - TTL: 64 - AS Number: AS20115 - AS Name: CHARTER-20115 - Country Name: United States - Country Code: US - Time Zone: America/New_York - Region: GA - City: Milledgeville - Latitude: 33.08 - Longitude: -83.24 #2 - Probe Send Time: 21:26:23 #3 - RTT [ms]: 16.5 - Probe Send Time: 21:26:26 - IP Address: 2001:506:100:548f::4 - Hostname: lag-22-10.dtr01mdvlga.netops.charter.com - TTL: 62 - Country Name: United States - Country Code: US - Time Zone: America/New_York - Region: GA - City: Columbus - Latitude: 32.47 - Longitude: -84.98 #4 - Probe Send Time: 21:26:26 #5 - RTT [ms]: 21.1 - Probe Send Time: 21:26:29 - IP Address: 2001:506:100:5507::4 - Hostname: lag-20.rcr01sghlgaao.netops.charter.com - TTL: 59 - Country Name: United States - Country Code: US - Time Zone: America/New_York - Region: GA - City: Sugar Hill - Latitude: 34.11 - Longitude: -84.00 #6 - Probe Send Time: 21:26:29 #7 - Probe Send Time: 21:26:32 #8 - Probe Send Time: 21:26:35 #9 - RTT [ms]: 35.3 - Probe Send Time: 21:26:38 - IP Address: 2001:470:0:42a::2 - Hostname: e0-35.core2.bna1.he.net - TTL: 56 - AS Number: AS6939 - AS Name: HURRICANE - Country Name: United States - Country Code: US - Time Zone: America/Chicago #10 - Probe Send Time: 21:26:38 #11 - Probe Send Time: 21:26:41 #12 - Probe Send Time: 21:26:44 #13 - Probe Send Time: 21:26:47 #14 - Probe Send Time: 21:26:50 #15 - Probe Send Time: 21:26:53 #16 - Probe Send Time: 21:26:56 #17 - Probe Send Time: 21:27:00 #18 - Probe Send Time: 21:27:03 On Thu, Apr 27, 2023, 09:48 Francis via NANOG <nanog@nanog.org> wrote:
I’m in the northeast and I can confirm on a Spectrum Enterprise connection I have issues tracerouting to 2604:1380:4641:c500::1. Gets to e0-33.core1.nyc7.he.net and then packets just disappear: Host Loss% Snt Last Avg Best Wrst StDev 1. _gateway 0.0% 108 1.0 1.7 0.8 26.3 3.3 2. 2602:XXX:XXXX:X::X 0.0% 108 4.4 6.6 4.2 27.9 4.0 3. 2602:XXX:XXXX:X::XXX 0.0% 108 3.1 3.0 3.0 3.5 0.0 4. 2604:XXXX:X:X::X:XXXX 0.0% 108 5.2 4.8 3.3 37.1 5.0 5. lag-15.srspny0101h.netops.charter.com 0.0% 108 3.9 20.8 3.6 371.6 45.4 6. lag-32.albynyyf02r.netops.charter.com 44.4% 108 4.9 4.9 4.8 5.5 0.1 7. lag-26.rcr01albynyyf.netops.charter.com 80.4% 108 5.1 5.3 5.0 7.7 0.6 8. lag-416.nycmny837aw-bcr00.netops.charter.com 57.9% 108 37.4 14.5 12.7 54.4 7.1 9. lag-1.pr2.nyc20.netops.charter.com 0.0% 108 49.8 17.1 12.3 65.7 11.0 10. e0-33.core1.nyc7.he.net 0.0% 108 12.9 13.2 12.8 19.8 0.7 11. (waiting for reply) Trying from another network and I was able to get through: Host Loss% Snt Last Avg Best Wrst StDev 1. ca.tor.ix.caramelfox.net 0.0% 25 45.7 45.9 43.9 50.9 2.0 2. (waiting for reply) 3. (waiting for reply) 4. ve955.core2.stl1.he.net 56.0% 25 62.0 63.9 62.0 67.6 2.0 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. equinix-ix.dfw2.packet.net 0.0% 25 81.2 86.0 77.8 103.3 8.4 9. (waiting for reply) 10. (waiting for reply) 11. dfw.source.kernel.org 0.0% 24 77.6 78.1 77.0 82.5 1.1 Maybe something in Hurricane Electric’s network potentially not getting these packets where they need to go?
On Apr 26, 2023, at 5:27 PM, Jeff P <jeffp@jeffp.us> wrote:
For what it's worth, on Spectrum Residential here in LAX and I can reach ip6echo.net and Google's public ipv6.google.com sites via residential service... I have my own modems and routers, which may make the difference in my case... Spectrum does indicate that IPv6 my need to be enabled on their routers...
https://www.spectrum.net/support/internet/ipv6-faq
jeffp@jeffp-Linux:~$ ping -6 ip6echo.net PING ip6echo.net(ip6echo.net (2600:5c01:f876:10::53)) 56 data bytes 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=1 ttl=51 time=76.1 ms 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=2 ttl=51 time=78.4 ms 64 bytes from ip6echo.net (2600:5c01:f876:10::53): icmp_seq=3 ttl=51 time=76.5 ms
jeffp@jeffp-Linux:~$ ping -6 ipv6.google.com PING ipv6.google.com(lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e)) 56 data bytes 64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=1 ttl=116 time=12.6 ms 64 bytes from lax31s06-in-x0e.1e100.net (2607:f8b0:4007:811::200e): icmp_seq=2 ttl=116 time=14.4 ms
I normally use the Google IPv6 site for such tests to avoid irritating private hosts, so a big thanks to Brandon for offering that alternative!
Jeffp
On 4/26/23 10:46, Saunders, Brandon wrote:
If it is helpful, I am on Spectrum in Ohio and I can reach that address from my IPv6 test system. This is on a commercial account with a provider assigned network. My system is at ip6echo.net. You are welcome to ping and traceroute to it if that is helpful to you.
A friend CANNOT get to that address from a Spectrum residential account also in my area.
Brandon
-----Original Message----- From: NANOG <nanog-bounces+saundeb1=ohio.edu@nanog.org> On Behalf Of Tom Rini Sent: Tuesday, April 25, 2023 2:13 PM To: nanog@nanog.org Subject: [External] Spectrum networks IPv6 access issue
Use caution with links and attachments.
Hey all,
I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done.
In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year.
-- Tom
-- JeffP
"It is not for me to place expectations on someone and fail them for not meeting them. It's my goal to encourage others to raise their expectations of themselves, to educate them and help them gain the skills needed to meet their goals and encourage them to succeed in attaining them."
-- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.
Actual data from a Spectrum residential customer in DFW. First, IPv4: <ns:~> trace dfw.source.kernel.org traceroute to dfw.source.kernel.org (139.178.84.217), 64 hops max, 40 byte packets 1 my.router 0.389 ms 0.350 ms 0.292 ms 2 142-254-130-077.inf.spectrum.com (142.254.130.77) 8.423 ms 8.408 ms 8.080 ms 3 lag-63.artrtx2801h.netops.charter.com (24.28.88.17) 27.167 ms 25.065 ms 21.977 ms 4 lag-22.artntxaf01r.netops.charter.com (24.175.49.233) 10.718 ms 10.083 ms 15.886 ms 5 lag-23.mcr11crtntxjt.netops.charter.com (24.175.36.224) 13.386 ms 11.560 ms 11.297 ms 6 lag-21.rcr01dllatx37.netops.charter.com (24.175.49.0) 11.339 ms lag-28.rcr01dllatx37.netops.charter.com (24.175.33.246) 11.904 ms 128.186 ms 7 lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.603 ms lag-14.dllstx976iw-bcr00.netops.charter.com (66.109.6.88) 12.172 ms lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.299 ms 8 lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 21.570 ms lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 11.763 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 12.182 ms 9 dls-b23-link.ip.twelve99.net (62.115.156.208) 11.515 ms * 11.706 ms 10 packethost-ic-369414.ip.twelve99-cust.net (213.248.72.3) 11.870 ms 30.246 ms 18.199 ms 11 * * * 12 * * * 13 dfw.source.kernel.org (139.178.84.217) 12.021 ms 12.076 ms 11.922 ms ping dfw.source.kernel.org PING dfw.source.kernel.org (139.178.84.217): 56 data bytes 64 bytes from 139.178.84.217: icmp_seq=0 ttl=50 time=11.590 ms 64 bytes from 139.178.84.217: icmp_seq=1 ttl=50 time=11.785 ms IPv6: <ns:~> trace6 dfw.source.kernel.org traceroute6 to dfw.source.kernel.org (2604:1380:4641:c500::1) from 2603:8080:REDACTED, 64 hops max, 20 byte packets 1 2603-8080-REDACTED.res6.spectrum.com 0.404 ms 0.340 ms 0.322 ms 2 2603-90c5-0003-000e-0000-0000-0000-0001.inf6.spectrum.com 10.308 ms 7.901 ms 9.902 ms 3 lag-63.artrtx2801h.netops.charter.com 17.008 ms 10.523 ms 11.077 ms 4 lag-22.artntxaf01r.netops.charter.com 14.638 ms * * 5 lag-23.mcr11crtntxjt.netops.charter.com 11.090 ms 11.612 ms 12.234 ms 6 * * * 7 lag-414.dllstx976iw-bcr00.netops.charter.com 12.572 ms * lag-24.dllstx976iw-bcr00.netops.charter.com 12.160 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 *^C ping6 dfw.source.kernel.org PING6(56=40+8+8 bytes) 2603:8080:REDACTED --> 2604:1380:4641:c500::1 ^C --- dfw.source.kernel.org ping6 statistics --- 5 packets transmitted, 0 packets received, 100.0% packet loss I have a Linode VM in Dallas that I also can't get to via IPv6. Traffic appears to take the same path for IPv4. On Wed, Apr 26, 2023 at 10:50 AM Tom Rini <trini@konsulko.com> wrote:
Hey all,
I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done.
In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year.
-- Tom
This has been “resolved", I finally got through to some awesome engineer at Spectrum who has rerouted traffic while they work with their hardware vendor (thanks Jake): 1 2605:6000:0:8::f:7acc (2605:6000:0:8::f:7acc) 0.372 ms 0.323 ms 0.282 ms 2 ae15.ARTNTXAF02H.chtrse.com (2605:6000:0:4::f:87e9) 4.002 ms 3.977 ms 10.884 ms 3 * * * 4 lag-23.mcr11dllbtxlb.netops.charter.com (2605:6000:0:4::2:1e) 1.464 ms 1.574 ms 1.560 ms 5 * * * 6 * * lag-26.hstqtx0209w-bcr00.netops.charter.com (2001:1998:0:4::528) 6.351 ms 7 * * * 8 lag-0.pr2.atl20.netops.charter.com (2001:1998:0:4::51f) 18.577 ms 18.572 ms 18.428 ms 9 * * * 10 e0-36.core2.bna1.he.net (2001:470:0:429::2) 32.553 ms 32.720 ms 32.563 ms 11 * * * 12 * * * 13 equinix-ix.dfw2.packet.net (2001:504:0:5:0:5:4825:1) 24.591 ms 24.553 ms 24.604 ms 14 * * * 15 * * * 16 dfw.source.kernel.org (2604:1380:4641:c500::1) 24.423 ms 26.020 ms 24.415 ms
On Apr 28, 2023, at 11:35 AM, Sam Thomas <sthomas@lart.net> wrote:
Actual data from a Spectrum residential customer in DFW.
First, IPv4:
<ns:~> trace dfw.source.kernel.org traceroute to dfw.source.kernel.org (139.178.84.217), 64 hops max, 40 byte packets 1 my.router 0.389 ms 0.350 ms 0.292 ms 2 142-254-130-077.inf.spectrum.com (142.254.130.77) 8.423 ms 8.408 ms 8.080 ms 3 lag-63.artrtx2801h.netops.charter.com (24.28.88.17) 27.167 ms 25.065 ms 21.977 ms 4 lag-22.artntxaf01r.netops.charter.com (24.175.49.233) 10.718 ms 10.083 ms 15.886 ms 5 lag-23.mcr11crtntxjt.netops.charter.com (24.175.36.224) 13.386 ms 11.560 ms 11.297 ms 6 lag-21.rcr01dllatx37.netops.charter.com (24.175.49.0) 11.339 ms lag-28.rcr01dllatx37.netops.charter.com (24.175.33.246) 11.904 ms 128.186 ms 7 lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.603 ms lag-14.dllstx976iw-bcr00.netops.charter.com (66.109.6.88) 12.172 ms lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.299 ms 8 lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 21.570 ms lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 11.763 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 12.182 ms 9 dls-b23-link.ip.twelve99.net (62.115.156.208) 11.515 ms * 11.706 ms 10 packethost-ic-369414.ip.twelve99-cust.net (213.248.72.3) 11.870 ms 30.246 ms 18.199 ms 11 * * * 12 * * * 13 dfw.source.kernel.org (139.178.84.217) 12.021 ms 12.076 ms 11.922 ms
ping dfw.source.kernel.org PING dfw.source.kernel.org (139.178.84.217): 56 data bytes 64 bytes from 139.178.84.217: icmp_seq=0 ttl=50 time=11.590 ms 64 bytes from 139.178.84.217: icmp_seq=1 ttl=50 time=11.785 ms
IPv6:
<ns:~> trace6 dfw.source.kernel.org traceroute6 to dfw.source.kernel.org (2604:1380:4641:c500::1) from 2603:8080:REDACTED, 64 hops max, 20 byte packets 1 2603-8080-REDACTED.res6.spectrum.com 0.404 ms 0.340 ms 0.322 ms 2 2603-90c5-0003-000e-0000-0000-0000-0001.inf6.spectrum.com 10.308 ms 7.901 ms 9.902 ms 3 lag-63.artrtx2801h.netops.charter.com 17.008 ms 10.523 ms 11.077 ms 4 lag-22.artntxaf01r.netops.charter.com 14.638 ms * * 5 lag-23.mcr11crtntxjt.netops.charter.com 11.090 ms 11.612 ms 12.234 ms 6 * * * 7 lag-414.dllstx976iw-bcr00.netops.charter.com 12.572 ms * lag-24.dllstx976iw-bcr00.netops.charter.com 12.160 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 *^C
ping6 dfw.source.kernel.org PING6(56=40+8+8 bytes) 2603:8080:REDACTED --> 2604:1380:4641:c500::1 ^C --- dfw.source.kernel.org ping6 statistics --- 5 packets transmitted, 0 packets received, 100.0% packet loss
I have a Linode VM in Dallas that I also can't get to via IPv6. Traffic appears to take the same path for IPv4.
On Wed, Apr 26, 2023 at 10:50 AM Tom Rini <trini@konsulko.com> wrote:
Hey all,
I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done.
In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year.
-- Tom
I can confirm things have been resolved here, many thanks all! On Tue, May 02, 2023 at 02:43:29PM -0400, d@nielmarks.com wrote:
This has been “resolved", I finally got through to some awesome engineer at Spectrum who has rerouted traffic while they work with their hardware vendor (thanks Jake):
1 2605:6000:0:8::f:7acc (2605:6000:0:8::f:7acc) 0.372 ms 0.323 ms 0.282 ms 2 ae15.ARTNTXAF02H.chtrse.com (2605:6000:0:4::f:87e9) 4.002 ms 3.977 ms 10.884 ms 3 * * * 4 lag-23.mcr11dllbtxlb.netops.charter.com (2605:6000:0:4::2:1e) 1.464 ms 1.574 ms 1.560 ms 5 * * * 6 * * lag-26.hstqtx0209w-bcr00.netops.charter.com (2001:1998:0:4::528) 6.351 ms 7 * * * 8 lag-0.pr2.atl20.netops.charter.com (2001:1998:0:4::51f) 18.577 ms 18.572 ms 18.428 ms 9 * * * 10 e0-36.core2.bna1.he.net (2001:470:0:429::2) 32.553 ms 32.720 ms 32.563 ms 11 * * * 12 * * * 13 equinix-ix.dfw2.packet.net (2001:504:0:5:0:5:4825:1) 24.591 ms 24.553 ms 24.604 ms 14 * * * 15 * * * 16 dfw.source.kernel.org (2604:1380:4641:c500::1) 24.423 ms 26.020 ms 24.415 ms
On Apr 28, 2023, at 11:35 AM, Sam Thomas <sthomas@lart.net> wrote:
Actual data from a Spectrum residential customer in DFW.
First, IPv4:
<ns:~> trace dfw.source.kernel.org traceroute to dfw.source.kernel.org (139.178.84.217), 64 hops max, 40 byte packets 1 my.router 0.389 ms 0.350 ms 0.292 ms 2 142-254-130-077.inf.spectrum.com (142.254.130.77) 8.423 ms 8.408 ms 8.080 ms 3 lag-63.artrtx2801h.netops.charter.com (24.28.88.17) 27.167 ms 25.065 ms 21.977 ms 4 lag-22.artntxaf01r.netops.charter.com (24.175.49.233) 10.718 ms 10.083 ms 15.886 ms 5 lag-23.mcr11crtntxjt.netops.charter.com (24.175.36.224) 13.386 ms 11.560 ms 11.297 ms 6 lag-21.rcr01dllatx37.netops.charter.com (24.175.49.0) 11.339 ms lag-28.rcr01dllatx37.netops.charter.com (24.175.33.246) 11.904 ms 128.186 ms 7 lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.603 ms lag-14.dllstx976iw-bcr00.netops.charter.com (66.109.6.88) 12.172 ms lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52) 12.299 ms 8 lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 21.570 ms lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 11.763 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 12.182 ms 9 dls-b23-link.ip.twelve99.net (62.115.156.208) 11.515 ms * 11.706 ms 10 packethost-ic-369414.ip.twelve99-cust.net (213.248.72.3) 11.870 ms 30.246 ms 18.199 ms 11 * * * 12 * * * 13 dfw.source.kernel.org (139.178.84.217) 12.021 ms 12.076 ms 11.922 ms
ping dfw.source.kernel.org PING dfw.source.kernel.org (139.178.84.217): 56 data bytes 64 bytes from 139.178.84.217: icmp_seq=0 ttl=50 time=11.590 ms 64 bytes from 139.178.84.217: icmp_seq=1 ttl=50 time=11.785 ms
IPv6:
<ns:~> trace6 dfw.source.kernel.org traceroute6 to dfw.source.kernel.org (2604:1380:4641:c500::1) from 2603:8080:REDACTED, 64 hops max, 20 byte packets 1 2603-8080-REDACTED.res6.spectrum.com 0.404 ms 0.340 ms 0.322 ms 2 2603-90c5-0003-000e-0000-0000-0000-0001.inf6.spectrum.com 10.308 ms 7.901 ms 9.902 ms 3 lag-63.artrtx2801h.netops.charter.com 17.008 ms 10.523 ms 11.077 ms 4 lag-22.artntxaf01r.netops.charter.com 14.638 ms * * 5 lag-23.mcr11crtntxjt.netops.charter.com 11.090 ms 11.612 ms 12.234 ms 6 * * * 7 lag-414.dllstx976iw-bcr00.netops.charter.com 12.572 ms * lag-24.dllstx976iw-bcr00.netops.charter.com 12.160 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 *^C
ping6 dfw.source.kernel.org PING6(56=40+8+8 bytes) 2603:8080:REDACTED --> 2604:1380:4641:c500::1 ^C --- dfw.source.kernel.org ping6 statistics --- 5 packets transmitted, 0 packets received, 100.0% packet loss
I have a Linode VM in Dallas that I also can't get to via IPv6. Traffic appears to take the same path for IPv4.
On Wed, Apr 26, 2023 at 10:50 AM Tom Rini <trini@konsulko.com> wrote:
Hey all,
I'm posting this here in hopes of getting the attention of someone that can get this issue resolved, or at least an internal ticket filed. I've tried the customer-facing tech support and not been able to get such a thing done.
In short, from within Spectrum's US IPv6 network (verified in both North Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is unreachable and connections time out. This site is otherwise fine and globally accessible via IPv6, tested on both Qwest and T-Mobile hosted systems. This is a regression from some time in early April this year.
-- Tom
-- Tom
On May 2, 2023, at 2:43 PM, Daniel Marks via NANOG <nanog@nanog.org> wrote:
This has been “resolved", I finally got through to some awesome engineer at Spectrum who has rerouted traffic while they work with their hardware vendor (thanks Jake):
One of the tools that I’ve used in the past is the RIPE Atlas service to measure these things. It’s helped me isolate IP space reachability issues for new announcements, because you can get enough of a random sample of hosts to isolate things, and enough data about that endpoint to launch follow-up measurements. - Jared
My issue was just trying to convince Spectrum to look into the problem in the first place, I brought the Atlas probe receipts because it’s such a helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, even the escalation list) until I started lighting fires filing FCC complaints and using social media (which thankfully worked). Not sure how accurate it is (I hope it isn’t), but some of the techs I spoke to said a lot of the internal tooling for troubleshooting is incapable of dealing with IPv6, so they weren’t able to do things like run traceroutes to confirm what I was seeing. My guess is that this issue was caught in a catch-22 where they needed impossible to obtain proof on their end to escalate to a team who can actually deal with the issue. Sucks for us folk who went all in on v6 only to find out not even the ISP can help us. -Daniel Marks
On May 2, 2023, at 15:36, Jared Mauch <jared@puck.nether.net> wrote:
On May 2, 2023, at 2:43 PM, Daniel Marks via NANOG <nanog@nanog.org> wrote:
This has been “resolved", I finally got through to some awesome engineer at Spectrum who has rerouted traffic while they work with their hardware vendor (thanks Jake):
One of the tools that I’ve used in the past is the RIPE Atlas service to measure these things. It’s helped me isolate IP space reachability issues for new announcements, because you can get enough of a random sample of hosts to isolate things, and enough data about that endpoint to launch follow-up measurements.
- Jared
We know the feeling well. Try porting from them…..
On May 2, 2023, at 4:41 PM, Daniel Marks via NANOG <nanog@nanog.org> wrote:
My issue was just trying to convince Spectrum to look into the problem in the first place, I brought the Atlas probe receipts because it’s such a helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, even the escalation list) until I started lighting fires filing FCC complaints and using social media (which thankfully worked).
Not sure how accurate it is (I hope it isn’t), but some of the techs I spoke to said a lot of the internal tooling for troubleshooting is incapable of dealing with IPv6, so they weren’t able to do things like run traceroutes to confirm what I was seeing. My guess is that this issue was caught in a catch-22 where they needed impossible to obtain proof on their end to escalate to a team who can actually deal with the issue.
Sucks for us folk who went all in on v6 only to find out not even the ISP can help us.
-Daniel Marks
On May 2, 2023, at 15:36, Jared Mauch <jared@puck.nether.net> wrote:
On May 2, 2023, at 2:43 PM, Daniel Marks via NANOG <nanog@nanog.org> wrote:
This has been “resolved", I finally got through to some awesome engineer at Spectrum who has rerouted traffic while they work with their hardware vendor (thanks Jake):
One of the tools that I’ve used in the past is the RIPE Atlas service to measure these things. It’s helped me isolate IP space reachability issues for new announcements, because you can get enough of a random sample of hosts to isolate things, and enough data about that endpoint to launch follow-up measurements.
- Jared
My issue was just trying to convince Spectrum to look into the problem in the first place, I brought the Atlas probe receipts because it’s such a helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, even the escalation list) until I started lighting fires filing FCC complaints and using social media (which thankfully worked).
Sucks for us folk who went all in on v6 only to find out not even the ISP can help us.
While this situation presented itself on IPv6, it's certainly not a problem uniquely experienced by or related to IPv6. There are all manners of issues, regardless of address family, that can take too much time to resolve because of issues with knowledge / tooling / contacts / etc . There is a reason "Does anyone know someone over at FOO?" has been a common phrase for decades. On Tue, May 2, 2023 at 4:41 PM Daniel Marks via NANOG <nanog@nanog.org> wrote:
My issue was just trying to convince Spectrum to look into the problem in the first place, I brought the Atlas probe receipts because it’s such a helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, even the escalation list) until I started lighting fires filing FCC complaints and using social media (which thankfully worked).
Not sure how accurate it is (I hope it isn’t), but some of the techs I spoke to said a lot of the internal tooling for troubleshooting is incapable of dealing with IPv6, so they weren’t able to do things like run traceroutes to confirm what I was seeing. My guess is that this issue was caught in a catch-22 where they needed impossible to obtain proof on their end to escalate to a team who can actually deal with the issue.
Sucks for us folk who went all in on v6 only to find out not even the ISP can help us.
-Daniel Marks
On May 2, 2023, at 15:36, Jared Mauch <jared@puck.nether.net> wrote:
On May 2, 2023, at 2:43 PM, Daniel Marks via NANOG <nanog@nanog.org> wrote:
This has been “resolved", I finally got through to some awesome engineer at Spectrum who has rerouted traffic while they work with their hardware vendor (thanks Jake):
One of the tools that I’ve used in the past is the RIPE Atlas service to measure these things. It’s helped me isolate IP space reachability issues for new announcements, because you can get enough of a random sample of hosts to isolate things, and enough data about that endpoint to launch follow-up measurements.
- Jared
participants (12)
-
boothf＠boothlabs.me -
Brandon Jackson -
d＠nielmarks.com -
Daniel Marks
-
J. Hellenthal -
Jared Mauch -
Jeff P -
Sam Thomas -
Saunders, Brandon -
Shawn L -
Tom Beecher -
Tom Rini