CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List, I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story. If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run). On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS). The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members. On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion: https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio... The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box. Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle? I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there. I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink. Best, Neel Chauhan
Neel, Sounds like buffer bloat. Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic. Ryan -----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA) Hi NANOG Mailing List, I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story. If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run). On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS). The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members. On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion: https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio n The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box. Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle? I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there. I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink. Best, Neel Chauhan
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic. The issue occurs even if I'm using 50 Mbps out of my 940. It may be bufferbloat on CL's side but they keep denying the issue. I guess I'll have to break the bank and get Comcast Gigabit Pro. CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice. -Neel On 2021-11-01 20:52, Ryan Hamel wrote:
Neel,
Sounds like buffer bloat.
Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic.
Ryan
-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List,
I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story.
If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run).
On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png
If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png
Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS).
The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members.
On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion: https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio n
The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box.
Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle?
I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there.
I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink.
Best,
Neel Chauhan
Hi, I have taught of an (hackish) workaround for now. Enable my Tor relays, but at the same time switch my non-Tor traffic to Verizon "LTE Home". Then hope my neighbors have service calls with CenturyLink which forces them to fix the issue (a tech told me about "capacity issues"). Monitor the CL connection every day, and if or when CenturyLink fixes the issue, cancel Verizon and enjoy. I could get Xfinity Prepaid for much cheaper, but since the Coax drop on our house is cut and we have no Coax outlets, the install would be hairy and long. CenturyLink had an advantage here since while the home was being flipped CL upgraded the street to fiber. The copper drop is still there and attached (Bell System 305A2 anyone?), but will probably never be used again. -Neel On 2021-11-02 07:28, Neel Chauhan wrote:
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic.
The issue occurs even if I'm using 50 Mbps out of my 940.
It may be bufferbloat on CL's side but they keep denying the issue.
I guess I'll have to break the bank and get Comcast Gigabit Pro.
CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice.
-Neel
On 2021-11-01 20:52, Ryan Hamel wrote:
Neel,
Sounds like buffer bloat.
Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic.
Ryan
-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List,
I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story.
If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run).
On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png
If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png
Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS).
The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members.
On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion: https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio n
The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box.
Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle?
I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there.
I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink.
Best,
Neel Chauhan
If this is connection count related only, It is most likely an issue with the CPE (router), NAT table, or similar. On Tue, Nov 2, 2021 at 8:21 AM Neel Chauhan <neel@neelc.org> wrote:
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic.
The issue occurs even if I'm using 50 Mbps out of my 940.
It may be bufferbloat on CL's side but they keep denying the issue.
I guess I'll have to break the bank and get Comcast Gigabit Pro.
CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice.
-Neel
On 2021-11-01 20:52, Ryan Hamel wrote:
Neel,
Sounds like buffer bloat.
Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic.
Ryan
-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List,
I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story.
If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run).
On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png
If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png
Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS).
The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members.
On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion:
https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio
n
The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box.
Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle?
I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there.
I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink.
Best,
Neel Chauhan
I second this, most best effort Broadband cpe equipment will choke with lots of concurrent connections On Tue, Nov 2, 2021, 8:25 PM P C <pc50000@gmail.com> wrote:
If this is connection count related only, It is most likely an issue with the CPE (router), NAT table, or similar.
On Tue, Nov 2, 2021 at 8:21 AM Neel Chauhan <neel@neelc.org> wrote:
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic.
The issue occurs even if I'm using 50 Mbps out of my 940.
It may be bufferbloat on CL's side but they keep denying the issue.
I guess I'll have to break the bank and get Comcast Gigabit Pro.
CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice.
-Neel
On 2021-11-01 20:52, Ryan Hamel wrote:
Neel,
Sounds like buffer bloat.
Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic.
Ryan
-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List,
I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story.
If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run).
On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png
If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png
Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS).
The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members.
On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion:
https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio
n
The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box.
Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle?
I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there.
I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink.
Best,
Neel Chauhan
My OPNsense box has a really large NAT table, more than there are IPv4 ports, presumably due to the way FreeBSD's pf works. CenturyLink's routers are worse in this regard since NAT tables are small. Even if I were to run Tor on my CenturyLink connection, my neighbors are affected as well. When I was running Tor at "full speed" with these spikes, neighbors had truck rolls. I have ordered Verizon "LTE Home" as a temporary "workaround", namely to move my Wi-Fi traffic to it while I restart Tor. The reason for this is to force neighbors to put in repair tickets to force a GPON capacity increase. It's not nice to my neighbors, but I don't **realy** have another option (even running Tor I don't want to make it too unbearablel for neighbors). Maybe the other options are to (a) pay $329/mo for Comcast Gigabit Pro and get stuck in a 2-year contract and a steep install fee or (b) litigate that I could lose since I'm not a lawyer and don't have a J.D. from Harvard Law School specializing in telecom law, both which are impractical and expensive. I was about to sue CenturyLink in small claims, but instead decided to get Verizon LTE Home for 2 months while CenturyLink "fixes" their fiber network while restarting Tor there. Have the neighbors put in service requests to force CL to fix their network, but don't put in a request myself so the "solution" won't be to blame me. While I may have to worry about an ETF with LTE Home, it's cheaper than Gigabit Pro, and of $300 for two months including ETF is still cheaper than $329/mo for two years. It's a gamble but may pay off. -Neel On 2021-11-03 07:00, TJ Trout wrote:
I second this, most best effort Broadband cpe equipment will choke with lots of concurrent connections
On Tue, Nov 2, 2021, 8:25 PM P C <pc50000@gmail.com> wrote:
If this is connection count related only, It is most likely an issue with the CPE (router), NAT table, or similar.
On Tue, Nov 2, 2021 at 8:21 AM Neel Chauhan <neel@neelc.org> wrote:
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic.
The issue occurs even if I'm using 50 Mbps out of my 940.
It may be bufferbloat on CL's side but they keep denying the issue.
I guess I'll have to break the bank and get Comcast Gigabit Pro.
CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice.
-Neel
On 2021-11-01 20:52, Ryan Hamel wrote:
Neel,
Sounds like buffer bloat.
Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic.
Ryan
-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List,
I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story.
If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run).
On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png
If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png
Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS).
The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members.
On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion:
https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio
n
The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't
compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box.
Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle?
I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there.
I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink.
Best,
Neel Chauhan
Respectfully, if you start broaching topics like suing your residential ISP because a "best effort" no SLA last mile GPON service doesn't meet the same performance as a 1GbE active-E symmetric 1310/LR hand off for a server in a datacenter, the most likely thing you're going to get is dropped by the ISP entirely. I know that's what I would recommend if I were your ISP and you threatened lawsuits at my customer service representatives. "Sorry, but as of $DATE_30_DAYS_IN_FUTURE we will no longer be able to meet your needs and will discontinue service and billing for $YOUR_ACCOUNT_NUMBER". On Thu, Nov 4, 2021 at 2:59 PM Neel Chauhan <neel@neelc.org> wrote:
My OPNsense box has a really large NAT table, more than there are IPv4 ports, presumably due to the way FreeBSD's pf works. CenturyLink's routers are worse in this regard since NAT tables are small.
Even if I were to run Tor on my CenturyLink connection, my neighbors are affected as well. When I was running Tor at "full speed" with these spikes, neighbors had truck rolls.
I have ordered Verizon "LTE Home" as a temporary "workaround", namely to move my Wi-Fi traffic to it while I restart Tor. The reason for this is to force neighbors to put in repair tickets to force a GPON capacity increase.
It's not nice to my neighbors, but I don't **realy** have another option (even running Tor I don't want to make it too unbearablel for neighbors). Maybe the other options are to (a) pay $329/mo for Comcast Gigabit Pro and get stuck in a 2-year contract and a steep install fee or (b) litigate that I could lose since I'm not a lawyer and don't have a J.D. from Harvard Law School specializing in telecom law, both which are impractical and expensive.
I was about to sue CenturyLink in small claims, but instead decided to get Verizon LTE Home for 2 months while CenturyLink "fixes" their fiber network while restarting Tor there. Have the neighbors put in service requests to force CL to fix their network, but don't put in a request myself so the "solution" won't be to blame me.
While I may have to worry about an ETF with LTE Home, it's cheaper than Gigabit Pro, and of $300 for two months including ETF is still cheaper than $329/mo for two years.
It's a gamble but may pay off.
-Neel
On 2021-11-03 07:00, TJ Trout wrote:
I second this, most best effort Broadband cpe equipment will choke with lots of concurrent connections
On Tue, Nov 2, 2021, 8:25 PM P C <pc50000@gmail.com> wrote:
If this is connection count related only, It is most likely an issue with the CPE (router), NAT table, or similar.
On Tue, Nov 2, 2021 at 8:21 AM Neel Chauhan <neel@neelc.org> wrote:
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic.
The issue occurs even if I'm using 50 Mbps out of my 940.
It may be bufferbloat on CL's side but they keep denying the issue.
I guess I'll have to break the bank and get Comcast Gigabit Pro.
CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice.
-Neel
On 2021-11-01 20:52, Ryan Hamel wrote:
Neel,
Sounds like buffer bloat.
Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic.
Ryan
-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List,
I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story.
If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run).
On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png
If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png
Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS).
The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members.
On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion:
https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio
n
The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't
compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box.
Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle?
I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there.
I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink.
Best,
Neel Chauhan
Agreed with this... if anything, all of the repeated abuse causing poor performance for neighbors could eventually result in termination of service. I'm sure CenturyLink would love to sell OP a DIA that they can use to run Tor and P2P to their heart's content. ________________________________ From: NANOG <nanog-bounces+tim=mid.net@nanog.org> on behalf of Eric Kuhnke <eric.kuhnke@gmail.com> Sent: Saturday, November 6, 2021 1:19:17 PM To: Neel Chauhan; nanog@nanog.org list Subject: Re: CenturyLink Fiber Latency Issues (Seattle, WA) Respectfully, if you start broaching topics like suing your residential ISP because a "best effort" no SLA last mile GPON service doesn't meet the same performance as a 1GbE active-E symmetric 1310/LR hand off for a server in a datacenter, the most likely thing you're going to get is dropped by the ISP entirely. I know that's what I would recommend if I were your ISP and you threatened lawsuits at my customer service representatives. "Sorry, but as of $DATE_30_DAYS_IN_FUTURE we will no longer be able to meet your needs and will discontinue service and billing for $YOUR_ACCOUNT_NUMBER". On Thu, Nov 4, 2021 at 2:59 PM Neel Chauhan <neel@neelc.org<mailto:neel@neelc.org>> wrote: My OPNsense box has a really large NAT table, more than there are IPv4 ports, presumably due to the way FreeBSD's pf works. CenturyLink's routers are worse in this regard since NAT tables are small. Even if I were to run Tor on my CenturyLink connection, my neighbors are affected as well. When I was running Tor at "full speed" with these spikes, neighbors had truck rolls. I have ordered Verizon "LTE Home" as a temporary "workaround", namely to move my Wi-Fi traffic to it while I restart Tor. The reason for this is to force neighbors to put in repair tickets to force a GPON capacity increase. It's not nice to my neighbors, but I don't **realy** have another option (even running Tor I don't want to make it too unbearablel for neighbors). Maybe the other options are to (a) pay $329/mo for Comcast Gigabit Pro and get stuck in a 2-year contract and a steep install fee or (b) litigate that I could lose since I'm not a lawyer and don't have a J.D. from Harvard Law School specializing in telecom law, both which are impractical and expensive. I was about to sue CenturyLink in small claims, but instead decided to get Verizon LTE Home for 2 months while CenturyLink "fixes" their fiber network while restarting Tor there. Have the neighbors put in service requests to force CL to fix their network, but don't put in a request myself so the "solution" won't be to blame me. While I may have to worry about an ETF with LTE Home, it's cheaper than Gigabit Pro, and of $300 for two months including ETF is still cheaper than $329/mo for two years. It's a gamble but may pay off. -Neel On 2021-11-03 07:00, TJ Trout wrote:
I second this, most best effort Broadband cpe equipment will choke with lots of concurrent connections
On Tue, Nov 2, 2021, 8:25 PM P C <pc50000@gmail.com<mailto:pc50000@gmail.com>> wrote:
If this is connection count related only, It is most likely an issue with the CPE (router), NAT table, or similar.
On Tue, Nov 2, 2021 at 8:21 AM Neel Chauhan <neel@neelc.org<mailto:neel@neelc.org>> wrote:
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic.
The issue occurs even if I'm using 50 Mbps out of my 940.
It may be bufferbloat on CL's side but they keep denying the issue.
I guess I'll have to break the bank and get Comcast Gigabit Pro.
CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice.
-Neel
On 2021-11-01 20:52, Ryan Hamel wrote:
Neel,
Sounds like buffer bloat.
Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic.
Ryan
-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org<mailto:rkhtech.org@nanog.org>> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List,
I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story.
If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run).
On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png
If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png
Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS).
The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members.
On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion:
https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio
n
The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't
compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box.
Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle?
I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there.
I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink.
Best,
Neel Chauhan
If I wanted a DIA, Comcast's "Gigabit Pro" is much cheaper than a CenturyLink DIA. But this happens even if I don't use most of my connection, even if I only use a fraction of it. In reality, I don't usually run BitTorrent, I just used it to stimulate the packet loss to show a technician the issue. I normally download ISOs via HTTP. I do run Tor relays normally, but an big issue being on the west coast is getting Tor to see all my bandwidth. Tor has things called "bandwidth scanners", but because of the way they're designed, measuring a Tor relay bandwidth in Seattle or LA would result in lower values than in NYC or Boston as you'd be burdened by higher latency. I'm a "Tor Core" contributor (not as active as I used to be) and even I can say the bandwidth scanners SUCK. -Neel On 2021-11-08 08:35, Tim Burke wrote:
Agreed with this... if anything, all of the repeated abuse causing poor performance for neighbors could eventually result in termination of service. I'm sure CenturyLink would love to sell OP a DIA that they can use to run Tor and P2P to their heart's content.
-------------------------
From: NANOG <nanog-bounces+tim=mid.net@nanog.org> on behalf of Eric Kuhnke <eric.kuhnke@gmail.com> Sent: Saturday, November 6, 2021 1:19:17 PM To: Neel Chauhan; nanog@nanog.org list Subject: Re: CenturyLink Fiber Latency Issues (Seattle, WA)
Respectfully, if you start broaching topics like suing your residential ISP because a "best effort" no SLA last mile GPON service doesn't meet the same performance as a 1GbE active-E symmetric 1310/LR hand off for a server in a datacenter, the most likely thing you're going to get is dropped by the ISP entirely.
I know that's what I would recommend if I were your ISP and you threatened lawsuits at my customer service representatives. "Sorry, but as of $DATE_30_DAYS_IN_FUTURE we will no longer be able to meet your needs and will discontinue service and billing for $YOUR_ACCOUNT_NUMBER".
On Thu, Nov 4, 2021 at 2:59 PM Neel Chauhan <neel@neelc.org> wrote:
My OPNsense box has a really large NAT table, more than there are IPv4 ports, presumably due to the way FreeBSD's pf works. CenturyLink's routers are worse in this regard since NAT tables are small.
Even if I were to run Tor on my CenturyLink connection, my neighbors are affected as well. When I was running Tor at "full speed" with these spikes, neighbors had truck rolls.
I have ordered Verizon "LTE Home" as a temporary "workaround", namely to move my Wi-Fi traffic to it while I restart Tor. The reason for this is to force neighbors to put in repair tickets to force a GPON capacity
increase.
It's not nice to my neighbors, but I don't **realy** have another option (even running Tor I don't want to make it too unbearablel for neighbors). Maybe the other options are to (a) pay $329/mo for Comcast Gigabit Pro and get stuck in a 2-year contract and a steep install fee or (b) litigate that I could lose since I'm not a lawyer and don't have a J.D. from Harvard Law School specializing in telecom law, both which are impractical and expensive.
I was about to sue CenturyLink in small claims, but instead decided to get Verizon LTE Home for 2 months while CenturyLink "fixes" their fiber network while restarting Tor there. Have the neighbors put in service requests to force CL to fix their network, but don't put in a request myself so the "solution" won't be to blame me.
While I may have to worry about an ETF with LTE Home, it's cheaper than Gigabit Pro, and of $300 for two months including ETF is still cheaper than $329/mo for two years.
It's a gamble but may pay off.
-Neel
On 2021-11-03 07:00, TJ Trout wrote:
I second this, most best effort Broadband cpe equipment will choke with lots of concurrent connections
On Tue, Nov 2, 2021, 8:25 PM P C <pc50000@gmail.com> wrote:
If this is connection count related only, It is most likely an issue with the CPE (router), NAT table, or similar.
On Tue, Nov 2, 2021 at 8:21 AM Neel Chauhan <neel@neelc.org> wrote:
I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic.
The issue occurs even if I'm using 50 Mbps out of my 940.
It may be bufferbloat on CL's side but they keep denying the issue.
I guess I'll have to break the bank and get Comcast Gigabit Pro.
CenturyLink should just get bought out by another telco, like how Cablevision got bought by Altice.
-Neel
On 2021-11-01 20:52, Ryan Hamel wrote:
Neel,
Sounds like buffer bloat.
Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of CenturyLink's device with the bursts of traffic.
Ryan
-----Original Message----- From: NANOG <nanog-bounces+ryan=rkhtech.org@nanog.org> On Behalf Of Neel Chauhan Sent: Monday, November 1, 2021 6:44 PM To: nanog@nanog.org Subject: CenturyLink Fiber Latency Issues (Seattle, WA)
Hi NANOG Mailing List,
I don't know if any of you work at CenturyLink/Lumen, very less on their Fiber network in Seattle, WA. However, here's my story.
If I attempt to run certain applications that use 1000, or 10000 TCP connections, I get latency spikes. It is based on how many connections, but also how much bandwidth is used. This means certain things like Tor relays are off limits to me (which I wish to run).
On an idle connection, the PingPlotter outputs look like this: https://centurylinklatencyissues.com/image-000.png [1]
If I attempt to run BitTorrent with 1000 connections in Deluge, PingPlotter looks like this: https://centurylinklatencyissues.com/image-002.png [2]
Getting support, or even executive contacts to admit the issue hasn't worked. They all love to blame my equipment or applications, when CL routers also show the issue when I run the same things whereas my same exact OPNsense box on Google Fiber Webpass running Tor at another address had no issues whatsoever, and I can ping other Tor relays on CenturyLink AS209 just fine (from a VPS).
The most competent person I dealt with was actually one tech. He told me there was "capacity issues" in our neighborhood, and that's the reason for the issues. However, nothing was done about it afterwards, I'm guessing since I turned off my Tor relay after the visit to avoid complaints from family members.
On an AT&T forum, people have said GPON gives latency spikes/packet loss on congestion:
https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio
[3]
n
The capacity managers in Seattle are literally dragging their feet: it's 100x worse than AT&T's 802.1X. I know AT&T and CenturyLink don't
compete, but if I had to choose between AT&T Fiber and CenturyLink, I'll take AT&T in a heartbeat, no ifs, no buts, even if I have to use AT&T's crappy router instead of my OPNsense box.
Going back, do any of you who work at CenturyLink/Lumen can get me to the right people, hopefully the capacity managers in Seattle?
I could go with Comcast, but it's either (a) 35 Mbps uploads or (b) $329/mo for "Gigabit Pro" with a 2-year contract and a steep install fee. I am seriously considering Gigabit Pro even if it breaks the bank, but hope I won't have to go there.
I don't need 2 Gbps and would rather pay $65 than $329. 300-500 Mbps uploads when I need it is the sweet spot for me (even without Tor) which CL GPON should easily handle without a sweat. I also don't exactly **trust** Comcast, they're a horrible company in many metrics, but in some ways Comcast is more competent than CenturyLink.
Best,
Neel Chauhan
Links: ------ [1] https://centurylinklatencyissues.com/image-000.png [2] https://centurylinklatencyissues.com/image-002.png [3] https://www.dslreports.com/forum/r33242889-How-rare-is-GPON-XGSPON-saturatio
participants (6)
-
Eric Kuhnke -
Neel Chauhan -
P C -
Ryan Hamel -
Tim Burke -
TJ Trout