Comcast Enterprise Fiber Slow Connection Problem from TW Telecom
We have been having a problem receiving software releases from our developer. The releases are typically around 1G in size. The developer’s connection is a 100m metro fiber with TW Telecom, our connection is a 25m Comcast Enterprise Fiber. Our traffic graphs show very little utilization of our connection. Typically on average we are at about 7 meg utilization of our 25. Every other partner that shares in our software development that receives the software releases can receive the updates 3-4 times faster than we can. Typically we receive the releases at about 3mbps. I have tried contacting Comcast Enterprise Tech support, however I’ve been told that if I run a speed test from my connection and the test runs at the speed we are paying for, there is very little they are willing to look into. Can anyone check on the Comcast Routers on the Tracert below, or is there anything that can be throttling this connection between the two connections? Also, our firewall and connection is able to run at the full 25. We have no throttling or QOS set to prevent a good connection to our developer. For example, we can run a multi-threaded upload, in the middle of the night, to Amazon Glacier storage and completely saturate our connection when doing so. The firewall and connection is able to handle our full bandwidth capacity during that backup. If there is any other information I can provide to help track this problem down, please let me know. Thanks in advance, everyone! Trace Route below: 1 (172.16.150.1) 1.143 ms 1.132 ms 1.122 ms 2 (173.227.204.1) 1.585 ms 1.583 ms 1.574 ms 3 chi2-pr1-xe-0-3-0-0.us.twtelecom.net (66.192.245.166) 10.477 ms 10.485 ms 10.478 ms 4 x-eth-0-0-4-pe05.350ecermak.il.ibone.comcast.net (75.149.230.141) 10.470 ms 10.465 ms 10.457 ms 5 he-2-1-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.86.37) 10.733 ms 10.731 ms he-2-0-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.86.33) 12.146 ms 6 be-10206-cr01.newyork.ny.ibone.comcast.net (68.86.86.225) 33.202 ms 32.144 ms 32.127 ms 7 68.86.91.30 (68.86.91.30) 41.508 ms 41.322 ms 41.599 ms 8 te-0-0-0-1-sur01.greensburg.pa.pitt.comcast.net (69.139.168.26) 38.196 ms te-0-0-0-3-sur01.greensburg.pa.pitt.comcast.net (162.151.21.82) 44.644 ms te-0-0-0-0-sur01.greensburg.pa.pitt.comcast.net (69.139.195.18) 38.266 ms 9 (107.1.72.98) 39.781 ms 39.785 ms 39.912 ms
On 31 October 2014 18:32, Zachary Frederick <zcfrederick@gmail.com> wrote:
We have been having a problem receiving software releases from our developer. The releases are typically around 1G in size. The developer’s connection is a 100m metro fiber with TW Telecom, our connection is a 25m Comcast Enterprise Fiber.
Our traffic graphs show very little utilization of our connection. Typically on average we are at about 7 meg utilization of our 25.
Every other partner that shares in our software development that receives the software releases can receive the updates 3-4 times faster than we can.
Typically we receive the releases at about 3mbps.
Are you using an application that uses TCP transport for the transfer? https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=38&loss=1e-06&Calculate=Calculate&bw=100&rtt2=80&win=64 3Mbps looks about right. Time for a tune up
I have tried contacting Comcast Enterprise Tech support, however I’ve been told that if I run a speed test from my connection and the test runs at the speed we are paying for, there is very little they are willing to look into.
Can anyone check on the Comcast Routers on the Tracert below, or is there anything that can be throttling this connection between the two connections?
Also, our firewall and connection is able to run at the full 25. We have no throttling or QOS set to prevent a good connection to our developer. For example, we can run a multi-threaded upload, in the middle of the night, to Amazon Glacier storage and completely saturate our connection when doing so. The firewall and connection is able to handle our full bandwidth capacity during that backup.
If there is any other information I can provide to help track this problem down, please let me know.
Thanks in advance, everyone!
Trace Route below:
1 (172.16.150.1) 1.143 ms 1.132 ms 1.122 ms
2 (173.227.204.1) 1.585 ms 1.583 ms 1.574 ms
3 chi2-pr1-xe-0-3-0-0.us.twtelecom.net (66.192.245.166) 10.477 ms 10.485 ms 10.478 ms
4 x-eth-0-0-4-pe05.350ecermak.il.ibone.comcast.net (75.149.230.141) 10.470 ms 10.465 ms 10.457 ms
5 he-2-1-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.86.37) 10.733 ms 10.731 ms he-2-0-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.86.33) 12.146 ms
6 be-10206-cr01.newyork.ny.ibone.comcast.net (68.86.86.225) 33.202 ms 32.144 ms 32.127 ms
7 68.86.91.30 (68.86.91.30) 41.508 ms 41.322 ms 41.599 ms
8 te-0-0-0-1-sur01.greensburg.pa.pitt.comcast.net (69.139.168.26) 38.196 ms te-0-0-0-3-sur01.greensburg.pa.pitt.comcast.net (162.151.21.82) 44.644 ms te-0-0-0-0-sur01.greensburg.pa.pitt.comcast.net (69.139.195.18) 38.266 ms
9 (107.1.72.98) 39.781 ms 39.785 ms 39.912 ms
With a max bandwidth of 25 Mbps and a 40ms RTT, the max is more like 14MB/s or 1.75 Mbps. https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=80&loss=1e-06&bw=25&rtt2=35&win=64&Calculate=Calculate But that's only if either endpoint is stuck at a 64 KB receive window. A quick packet capture would be able to see what was happening. Check the TCP setup and make sure that both ends are doing TCP window scaling properly. John On Fri, Oct 31, 2014 at 1:02 PM, Pedro Cavaca <pmsac.nanog@gmail.com> wrote:
On 31 October 2014 18:32, Zachary Frederick <zcfrederick@gmail.com> wrote:
We have been having a problem receiving software releases from our developer. The releases are typically around 1G in size. The developer’s connection is a 100m metro fiber with TW Telecom, our connection is a 25m Comcast Enterprise Fiber.
Our traffic graphs show very little utilization of our connection. Typically on average we are at about 7 meg utilization of our 25.
Every other partner that shares in our software development that receives the software releases can receive the updates 3-4 times faster than we can.
Typically we receive the releases at about 3mbps.
Are you using an application that uses TCP transport for the transfer?
3Mbps looks about right. Time for a tune up
I have tried contacting Comcast Enterprise Tech support, however I’ve been told that if I run a speed test from my connection and the test runs at the speed we are paying for, there is very little they are willing to look into.
Can anyone check on the Comcast Routers on the Tracert below, or is there anything that can be throttling this connection between the two connections?
Also, our firewall and connection is able to run at the full 25. We have no throttling or QOS set to prevent a good connection to our developer. For example, we can run a multi-threaded upload, in the middle of the night, to Amazon Glacier storage and completely saturate our connection when doing so. The firewall and connection is able to handle our full bandwidth capacity during that backup.
If there is any other information I can provide to help track this problem down, please let me know.
Thanks in advance, everyone!
Trace Route below:
1 (172.16.150.1) 1.143 ms 1.132 ms 1.122 ms
2 (173.227.204.1) 1.585 ms 1.583 ms 1.574 ms
3 chi2-pr1-xe-0-3-0-0.us.twtelecom.net (66.192.245.166) 10.477 ms 10.485 ms 10.478 ms
4 x-eth-0-0-4-pe05.350ecermak.il.ibone.comcast.net (75.149.230.141) 10.470 ms 10.465 ms 10.457 ms
5 he-2-1-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.86.37) 10.733 ms 10.731 ms he-2-0-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.86.33) 12.146 ms
6 be-10206-cr01.newyork.ny.ibone.comcast.net (68.86.86.225) 33.202 ms 32.144 ms 32.127 ms
7 68.86.91.30 (68.86.91.30) 41.508 ms 41.322 ms 41.599 ms
8 te-0-0-0-1-sur01.greensburg.pa.pitt.comcast.net (69.139.168.26) 38.196 ms te-0-0-0-3-sur01.greensburg.pa.pitt.comcast.net (162.151.21.82) 44.644 ms te-0-0-0-0-sur01.greensburg.pa.pitt.comcast.net (69.139.195.18) 38.266 ms
9 (107.1.72.98) 39.781 ms 39.785 ms 39.912 ms
I apologize I should have said it starts out about 3 meg max and slows to about 400kpbs for most of the transfer.
On Oct 31, 2014, at 3:27 PM, John Neiberger <jneiberger@gmail.com> wrote:
With a max bandwidth of 25 Mbps and a 40ms RTT, the max is more like 14MB/s or 1.75 Mbps.
https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=80&loss=1e-06&bw=25&rtt2=35&win=64&Calculate=Calculate <https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=80&loss=1e-06&bw=25&rtt2=35&win=64&Calculate=Calculate>
But that's only if either endpoint is stuck at a 64 KB receive window. A quick packet capture would be able to see what was happening. Check the TCP setup and make sure that both ends are doing TCP window scaling properly.
John
On Fri, Oct 31, 2014 at 1:02 PM, Pedro Cavaca <pmsac.nanog@gmail.com <mailto:pmsac.nanog@gmail.com>> wrote: On 31 October 2014 18:32, Zachary Frederick <zcfrederick@gmail.com <mailto:zcfrederick@gmail.com>> wrote:
We have been having a problem receiving software releases from our developer. The releases are typically around 1G in size. The developer’s connection is a 100m metro fiber with TW Telecom, our connection is a 25m Comcast Enterprise Fiber.
Our traffic graphs show very little utilization of our connection. Typically on average we are at about 7 meg utilization of our 25.
Every other partner that shares in our software development that receives the software releases can receive the updates 3-4 times faster than we can.
Typically we receive the releases at about 3mbps.
Are you using an application that uses TCP transport for the transfer?
https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=38&loss=1e-06&Calculate=Calculate&bw=100&rtt2=80&win=64 <https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=38&loss=1e-06&Calculate=Calculate&bw=100&rtt2=80&win=64>
3Mbps looks about right. Time for a tune up
I have tried contacting Comcast Enterprise Tech support, however I’ve been told that if I run a speed test from my connection and the test runs at the speed we are paying for, there is very little they are willing to look into.
Can anyone check on the Comcast Routers on the Tracert below, or is there anything that can be throttling this connection between the two connections?
Also, our firewall and connection is able to run at the full 25. We have no throttling or QOS set to prevent a good connection to our developer. For example, we can run a multi-threaded upload, in the middle of the night, to Amazon Glacier storage and completely saturate our connection when doing so. The firewall and connection is able to handle our full bandwidth capacity during that backup.
If there is any other information I can provide to help track this problem down, please let me know.
Thanks in advance, everyone!
Trace Route below:
1 (172.16.150.1) 1.143 ms 1.132 ms 1.122 ms
2 (173.227.204.1) 1.585 ms 1.583 ms 1.574 ms
3 chi2-pr1-xe-0-3-0-0.us.twtelecom.net <http://chi2-pr1-xe-0-3-0-0.us.twtelecom.net/> (66.192.245.166) 10.477 ms 10.485 ms 10.478 ms
4 x-eth-0-0-4-pe05.350ecermak.il.ibone.comcast.net <http://x-eth-0-0-4-pe05.350ecermak.il.ibone.comcast.net/> (75.149.230.141) 10.470 ms 10.465 ms 10.457 ms
5 he-2-1-0-0-cr01.350ecermak.il.ibone.comcast.net <http://he-2-1-0-0-cr01.350ecermak.il.ibone.comcast.net/> (68.86.86.37) 10.733 ms 10.731 ms he-2-0-0-0-cr01.350ecermak.il.ibone.comcast.net <http://he-2-0-0-0-cr01.350ecermak.il.ibone.comcast.net/> (68.86.86.33) 12.146 ms
6 be-10206-cr01.newyork.ny.ibone.comcast.net <http://be-10206-cr01.newyork.ny.ibone.comcast.net/> (68.86.86.225) 33.202 ms 32.144 ms 32.127 ms
7 68.86.91.30 (68.86.91.30) 41.508 ms 41.322 ms 41.599 ms
8 te-0-0-0-1-sur01.greensburg.pa.pitt.comcast.net <http://te-0-0-0-1-sur01.greensburg.pa.pitt.comcast.net/> (69.139.168.26) 38.196 ms te-0-0-0-3-sur01.greensburg.pa.pitt.comcast.net <http://te-0-0-0-3-sur01.greensburg.pa.pitt.comcast.net/> (162.151.21.82) 44.644 ms te-0-0-0-0-sur01.greensburg.pa.pitt.comcast.net <http://te-0-0-0-0-sur01.greensburg.pa.pitt.comcast.net/> (69.139.195.18) 38.266 ms
9 (107.1.72.98) 39.781 ms 39.785 ms 39.912 ms
Sounds like a combination of packet loss and small TCP receive windows. If you can, grab a packet capture and make sure to get the TCP setup. That should show you what's happening under the hood. Also, I should mention that I totally hosed the units in my first reply. :) That's what I get for hurrying. John On Fri, Oct 31, 2014 at 1:29 PM, Zachary Frederick <zcfrederick@gmail.com> wrote:
I apologize I should have said it starts out about 3 meg max and slows to about 400kpbs for most of the transfer.
On Oct 31, 2014, at 3:27 PM, John Neiberger <jneiberger@gmail.com> wrote:
With a max bandwidth of 25 Mbps and a 40ms RTT, the max is more like 14MB/s or 1.75 Mbps.
But that's only if either endpoint is stuck at a 64 KB receive window. A quick packet capture would be able to see what was happening. Check the TCP setup and make sure that both ends are doing TCP window scaling properly.
John
On Fri, Oct 31, 2014 at 1:02 PM, Pedro Cavaca <pmsac.nanog@gmail.com> wrote:
On 31 October 2014 18:32, Zachary Frederick <zcfrederick@gmail.com> wrote:
We have been having a problem receiving software releases from our developer. The releases are typically around 1G in size. The developer’s connection is a 100m metro fiber with TW Telecom, our connection is a 25m Comcast Enterprise Fiber.
Our traffic graphs show very little utilization of our connection. Typically on average we are at about 7 meg utilization of our 25.
Every other partner that shares in our software development that receives the software releases can receive the updates 3-4 times faster than we can.
Typically we receive the releases at about 3mbps.
Are you using an application that uses TCP transport for the transfer?
3Mbps looks about right. Time for a tune up
I have tried contacting Comcast Enterprise Tech support, however I’ve been told that if I run a speed test from my connection and the test runs at the speed we are paying for, there is very little they are willing to look into.
Can anyone check on the Comcast Routers on the Tracert below, or is there anything that can be throttling this connection between the two connections?
Also, our firewall and connection is able to run at the full 25. We have no throttling or QOS set to prevent a good connection to our developer. For example, we can run a multi-threaded upload, in the middle of the night, to Amazon Glacier storage and completely saturate our connection when doing so. The firewall and connection is able to handle our full bandwidth capacity during that backup.
If there is any other information I can provide to help track this problem down, please let me know.
Thanks in advance, everyone!
Trace Route below:
1 (172.16.150.1) 1.143 ms 1.132 ms 1.122 ms
2 (173.227.204.1) 1.585 ms 1.583 ms 1.574 ms
3 chi2-pr1-xe-0-3-0-0.us.twtelecom.net (66.192.245.166) 10.477 ms 10.485 ms 10.478 ms
4 x-eth-0-0-4-pe05.350ecermak.il.ibone.comcast.net (75.149.230.141) 10.470 ms 10.465 ms 10.457 ms
5 he-2-1-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.86.37) 10.733 ms 10.731 ms he-2-0-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.86.33) 12.146 ms
6 be-10206-cr01.newyork.ny.ibone.comcast.net (68.86.86.225) 33.202 ms 32.144 ms 32.127 ms
7 68.86.91.30 (68.86.91.30) 41.508 ms 41.322 ms 41.599 ms
8 te-0-0-0-1-sur01.greensburg.pa.pitt.comcast.net (69.139.168.26) 38.196 ms te-0-0-0-3-sur01.greensburg.pa.pitt.comcast.net (162.151.21.82) 44.644 ms te-0-0-0-0-sur01.greensburg.pa.pitt.comcast.net (69.139.195.18) 38.266 ms
9 (107.1.72.98) 39.781 ms 39.785 ms 39.912 ms
Recommendations: 1) Use iperf in TCP mode to test the performance 2) use iperf in UDP mode to test the performance This is the best way to quickly triage the issue and determine if it's actual bandwidth issue or something else. It's quite common for various NAT/VPN devices to eat or meddle with packets such that performance is limited, sometimes severely. You need to check the CPU/performance of these devices. Do you have graphs showing the links as idle for each end while doing the test? Many people don't have this data or tools. Consider setting up a tool like observium to monitor the health and performance of the devices. Check other devices, such as switches, duplex settings, etc. If the device is 'unmanaged' and you can't tell if it negotiated 100-full or 100-half, consider it may have done half-duplex. I recall one customer case where after much troubleshooting for performance they had a duplex issue causing trouble. Hope this helps, - Jared On Fri, Oct 31, 2014 at 03:29:56PM -0400, Zachary Frederick wrote:
I apologize I should have said it starts out about 3 meg max and slows to about 400kpbs for most of the transfer.
On Oct 31, 2014, at 3:27 PM, John Neiberger <jneiberger@gmail.com> wrote:
With a max bandwidth of 25 Mbps and a 40ms RTT, the max is more like 14MB/s or 1.75 Mbps.
https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=80&loss=1e-06&bw=25&rtt2=35&win=64&Calculate=Calculate <https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=80&loss=1e-06&bw=25&rtt2=35&win=64&Calculate=Calculate>
But that's only if either endpoint is stuck at a 64 KB receive window. A quick packet capture would be able to see what was happening. Check the TCP setup and make sure that both ends are doing TCP window scaling properly.
John
On Fri, Oct 31, 2014 at 1:02 PM, Pedro Cavaca <pmsac.nanog@gmail.com <mailto:pmsac.nanog@gmail.com>> wrote: On 31 October 2014 18:32, Zachary Frederick <zcfrederick@gmail.com <mailto:zcfrederick@gmail.com>> wrote:
We have been having a problem receiving software releases from our developer. The releases are typically around 1G in size. The developer’s connection is a 100m metro fiber with TW Telecom, our connection is a 25m Comcast Enterprise Fiber.
Our traffic graphs show very little utilization of our connection. Typically on average we are at about 7 meg utilization of our 25.
Every other partner that shares in our software development that receives the software releases can receive the updates 3-4 times faster than we can.
Typically we receive the releases at about 3mbps.
Are you using an application that uses TCP transport for the transfer?
https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=38&loss=1e-06&Calculate=Calculate&bw=100&rtt2=80&win=64 <https://www.switch.ch/network/tools/tcp_throughput/index.html?mss=1460&rtt=38&loss=1e-06&Calculate=Calculate&bw=100&rtt2=80&win=64>
3Mbps looks about right. Time for a tune up
I have tried contacting Comcast Enterprise Tech support, however I’ve been told that if I run a speed test from my connection and the test runs at the speed we are paying for, there is very little they are willing to look into.
Can anyone check on the Comcast Routers on the Tracert below, or is there anything that can be throttling this connection between the two connections?
Also, our firewall and connection is able to run at the full 25. We have no throttling or QOS set to prevent a good connection to our developer. For example, we can run a multi-threaded upload, in the middle of the night, to Amazon Glacier storage and completely saturate our connection when doing so. The firewall and connection is able to handle our full bandwidth capacity during that backup.
If there is any other information I can provide to help track this problem down, please let me know.
Thanks in advance, everyone!
Trace Route below:
1 (172.16.150.1) 1.143 ms 1.132 ms 1.122 ms
2 (173.227.204.1) 1.585 ms 1.583 ms 1.574 ms
3 chi2-pr1-xe-0-3-0-0.us.twtelecom.net <http://chi2-pr1-xe-0-3-0-0.us.twtelecom.net/> (66.192.245.166) 10.477 ms 10.485 ms 10.478 ms
4 x-eth-0-0-4-pe05.350ecermak.il.ibone.comcast.net <http://x-eth-0-0-4-pe05.350ecermak.il.ibone.comcast.net/> (75.149.230.141) 10.470 ms 10.465 ms 10.457 ms
5 he-2-1-0-0-cr01.350ecermak.il.ibone.comcast.net <http://he-2-1-0-0-cr01.350ecermak.il.ibone.comcast.net/> (68.86.86.37) 10.733 ms 10.731 ms he-2-0-0-0-cr01.350ecermak.il.ibone.comcast.net <http://he-2-0-0-0-cr01.350ecermak.il.ibone.comcast.net/> (68.86.86.33) 12.146 ms
6 be-10206-cr01.newyork.ny.ibone.comcast.net <http://be-10206-cr01.newyork.ny.ibone.comcast.net/> (68.86.86.225) 33.202 ms 32.144 ms 32.127 ms
7 68.86.91.30 (68.86.91.30) 41.508 ms 41.322 ms 41.599 ms
8 te-0-0-0-1-sur01.greensburg.pa.pitt.comcast.net <http://te-0-0-0-1-sur01.greensburg.pa.pitt.comcast.net/> (69.139.168.26) 38.196 ms te-0-0-0-3-sur01.greensburg.pa.pitt.comcast.net <http://te-0-0-0-3-sur01.greensburg.pa.pitt.comcast.net/> (162.151.21.82) 44.644 ms te-0-0-0-0-sur01.greensburg.pa.pitt.comcast.net <http://te-0-0-0-0-sur01.greensburg.pa.pitt.comcast.net/> (69.139.195.18) 38.266 ms
9 (107.1.72.98) 39.781 ms 39.785 ms 39.912 ms
-- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
participants (4)
-
Jared Mauch -
John Neiberger -
Pedro Cavaca -
Zachary Frederick