is something weird going on with cox, level3, and/or cogent
Hi Folks, I'm visiting AZ, and seeing some really really poor performance accessing some of our servers via Cox broadband. The folks at Cox technical support are useless - all they say is "well you're on a DOCSIS 2 modem." Meanwhile, everything I'm seeing is several hops upstream of the local segment - and all that Cox level2 tech support will say is "if there was a backbone problem, our backbone people would have dealt with it." I'm having problems reaching both our own server, and sites like google, facebook, windows update. Traceroutes to and from our server are illustrative - note that for most of the past week, the average ping time was 85msec. Now we're seeing this: From 98.177.135.186 - the public IP address on Cox's local broadband service. To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint data center, near Boston) traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 10.128.128.1 (10.128.128.1) 199.893 ms 75.319 ms 27.295 ms 2 100.127.69.178 (100.127.69.178) 38.710 ms 40.075 ms 43.598 ms 3 72.215.229.22 (72.215.229.22) 39.674 ms 201.368 ms * 4 lag-157.bear2.phoenix1.level3.net (4.28.82.53) 686.499 ms 1837.141 ms 16.273 ms 5 ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 35.498 ms 964.377 ms * 6 ae-3-80.edge1.losangeles6.level3.net (4.69.144.144) 551.760 ms 525.014 ms ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 2061.191 ms 7 be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129) 847.778 ms 87.601 ms 71.504 ms 8 be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1) 79.060 ms 225.647 ms be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77) 60.306 ms 9 * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms 185.180 ms 10 be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66) 61.208 ms be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78) 386.149 ms 1278.868 ms 11 be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161) 384.136 ms be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms 615.415 ms 12 be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 233.061 ms be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69) 87.902 ms be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 861.159 ms 13 be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221) 998.858 ms be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157) 249.930 ms * 14 be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 768.461 ms be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105) 136.772 ms be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 288.225 ms 15 be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 271.736 ms 166.224 ms be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42) 565.015 ms 16 te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms 17 te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82) 107.207 ms te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78) 295.881 ms 185.453 ms 18 38.122.127.18 (38.122.127.18) 115.652 ms 461.168 ms 615.526 ms 19 207.154.0.57 (207.154.0.57) 1871.023 ms 1987.832 ms 2165.248 ms 20 server1.ntcorp.com (207.154.13.58) 587.560 ms 263.328 ms 333.542 ms Traceroute in the reverse direction: 1 207.154.13.47 (207.154.13.47) 0.000 ms 0.000 ms 0.000 ms 2 * * * 3 h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms 0.000 ms 0.000 ms 4 xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms 12.000 ms 12.000 ms 5 et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms 20.000 ms 20.000 ms 6 10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37) 72.001 ms 20.000 ms 16.000 ms 7 chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193) 20.000 ms 20.000 ms 20.000 ms 8 chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211) 72.001 ms 72.001 ms 72.001 ms 9 * * * 10 * * * 11 * * * <snip> 30 * * * <looks like a lot of the later hops don't respond to pings> Two things jump out at me: 1. The rather large number of hops from cox to ntcorp - with high delays from several nodes in both the level3 and cogent networks. 2. That there's a rather more direct path from the datacenter to cox, that shows up in the reverse direction. Some kind of routing or peering issue, perhaps? (And I also note the earlier string of messages regarding youtube streaming problems - that also seemed to involve cox and level3. Thanks for any insight (and better, for any fixes!). Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
It looks like one or more circuits are down, so you're seeing asymmetrical routing over congested paths in one direction. -mel via cell
On Feb 12, 2017, at 7:14 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Hi Folks,
I'm visiting AZ, and seeing some really really poor performance accessing some of our servers via Cox broadband. The folks at Cox technical support are useless - all they say is "well you're on a DOCSIS 2 modem." Meanwhile, everything I'm seeing is several hops upstream of the local segment - and all that Cox level2 tech support will say is "if there was a backbone problem, our backbone people would have dealt with it."
I'm having problems reaching both our own server, and sites like google, facebook, windows update.
Traceroutes to and from our server are illustrative - note that for most of the past week, the average ping time was 85msec. Now we're seeing this:
From 98.177.135.186 - the public IP address on Cox's local broadband service. To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint data center, near Boston) traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 10.128.128.1 (10.128.128.1) 199.893 ms 75.319 ms 27.295 ms 2 100.127.69.178 (100.127.69.178) 38.710 ms 40.075 ms 43.598 ms 3 72.215.229.22 (72.215.229.22) 39.674 ms 201.368 ms * 4 lag-157.bear2.phoenix1.level3.net (4.28.82.53) 686.499 ms 1837.141 ms 16.273 ms 5 ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 35.498 ms 964.377 ms * 6 ae-3-80.edge1.losangeles6.level3.net (4.69.144.144) 551.760 ms 525.014 ms ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 2061.191 ms 7 be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129) 847.778 ms 87.601 ms 71.504 ms 8 be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1) 79.060 ms 225.647 ms be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77) 60.306 ms 9 * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms 185.180 ms 10 be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66) 61.208 ms be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78) 386.149 ms 1278.868 ms 11 be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161) 384.136 ms be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms 615.415 ms 12 be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 233.061 ms be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69) 87.902 ms be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 861.159 ms 13 be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221) 998.858 ms be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157) 249.930 ms * 14 be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 768.461 ms be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105) 136.772 ms be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 288.225 ms 15 be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 271.736 ms 166.224 ms be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42) 565.015 ms 16 te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms 17 te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82) 107.207 ms te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78) 295.881 ms 185.453 ms 18 38.122.127.18 (38.122.127.18) 115.652 ms 461.168 ms 615.526 ms 19 207.154.0.57 (207.154.0.57) 1871.023 ms 1987.832 ms 2165.248 ms 20 server1.ntcorp.com (207.154.13.58) 587.560 ms 263.328 ms 333.542 ms
Traceroute in the reverse direction: 1 207.154.13.47 (207.154.13.47) 0.000 ms 0.000 ms 0.000 ms 2 * * * 3 h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms 0.000 ms 0.000 ms 4 xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms 12.000 ms 12.000 ms 5 et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms 20.000 ms 20.000 ms 6 10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37) 72.001 ms 20.000 ms 16.000 ms 7 chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193) 20.000 ms 20.000 ms 20.000 ms 8 chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211) 72.001 ms 72.001 ms 72.001 ms 9 * * * 10 * * * 11 * * * <snip> 30 * * * <looks like a lot of the later hops don't respond to pings>
Two things jump out at me: 1. The rather large number of hops from cox to ntcorp - with high delays from several nodes in both the level3 and cogent networks. 2. That there's a rather more direct path from the datacenter to cox, that shows up in the reverse direction.
Some kind of routing or peering issue, perhaps? (And I also note the earlier string of messages regarding youtube streaming problems - that also seemed to involve cox and level3.
Thanks for any insight (and better, for any fixes!).
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
Miles, Have you tried trace routing through cellular data connections? The results you're seeing could be explained by congestion at the point of your modem, which I think is with the cox techs are implying. -mel via cell
On Feb 12, 2017, at 7:46 PM, Mel Beckman <mel@beckman.org> wrote:
It looks like one or more circuits are down, so you're seeing asymmetrical routing over congested paths in one direction.
-mel via cell
On Feb 12, 2017, at 7:14 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Hi Folks,
I'm visiting AZ, and seeing some really really poor performance accessing some of our servers via Cox broadband. The folks at Cox technical support are useless - all they say is "well you're on a DOCSIS 2 modem." Meanwhile, everything I'm seeing is several hops upstream of the local segment - and all that Cox level2 tech support will say is "if there was a backbone problem, our backbone people would have dealt with it."
I'm having problems reaching both our own server, and sites like google, facebook, windows update.
Traceroutes to and from our server are illustrative - note that for most of the past week, the average ping time was 85msec. Now we're seeing this:
From 98.177.135.186 - the public IP address on Cox's local broadband service. To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint data center, near Boston) traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 10.128.128.1 (10.128.128.1) 199.893 ms 75.319 ms 27.295 ms 2 100.127.69.178 (100.127.69.178) 38.710 ms 40.075 ms 43.598 ms 3 72.215.229.22 (72.215.229.22) 39.674 ms 201.368 ms * 4 lag-157.bear2.phoenix1.level3.net (4.28.82.53) 686.499 ms 1837.141 ms 16.273 ms 5 ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 35.498 ms 964.377 ms * 6 ae-3-80.edge1.losangeles6.level3.net (4.69.144.144) 551.760 ms 525.014 ms ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 2061.191 ms 7 be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129) 847.778 ms 87.601 ms 71.504 ms 8 be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1) 79.060 ms 225.647 ms be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77) 60.306 ms 9 * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms 185.180 ms 10 be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66) 61.208 ms be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78) 386.149 ms 1278.868 ms 11 be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161) 384.136 ms be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms 615.415 ms 12 be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 233.061 ms be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69) 87.902 ms be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 861.159 ms 13 be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221) 998.858 ms be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157) 249.930 ms * 14 be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 768.461 ms be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105) 136.772 ms be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 288.225 ms 15 be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 271.736 ms 166.224 ms be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42) 565.015 ms 16 te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms 17 te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82) 107.207 ms te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78) 295.881 ms 185.453 ms 18 38.122.127.18 (38.122.127.18) 115.652 ms 461.168 ms 615.526 ms 19 207.154.0.57 (207.154.0.57) 1871.023 ms 1987.832 ms 2165.248 ms 20 server1.ntcorp.com (207.154.13.58) 587.560 ms 263.328 ms 333.542 ms
Traceroute in the reverse direction: 1 207.154.13.47 (207.154.13.47) 0.000 ms 0.000 ms 0.000 ms 2 * * * 3 h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms 0.000 ms 0.000 ms 4 xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms 12.000 ms 12.000 ms 5 et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms 20.000 ms 20.000 ms 6 10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37) 72.001 ms 20.000 ms 16.000 ms 7 chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193) 20.000 ms 20.000 ms 20.000 ms 8 chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211) 72.001 ms 72.001 ms 72.001 ms 9 * * * 10 * * * 11 * * * <snip> 30 * * * <looks like a lot of the later hops don't respond to pings>
Two things jump out at me: 1. The rather large number of hops from cox to ntcorp - with high delays from several nodes in both the level3 and cogent networks. 2. That there's a rather more direct path from the datacenter to cox, that shows up in the reverse direction.
Some kind of routing or peering issue, perhaps? (And I also note the earlier string of messages regarding youtube streaming problems - that also seemed to involve cox and level3.
Thanks for any insight (and better, for any fixes!).
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
Now isn't that interesting. 1. Verizon mobile seems to also route through level3 and cogent. 2. But.. the performance seems to be a lot better. 3. That's really odd. 1 192.168.43.1 (192.168.43.1) 2.063 ms 2.382 ms 1.913 ms 2 7.sub-66-174-33.myvzw.com (66.174.33.7) 50.676 ms 36.720 ms 41.142 ms 3 164.sub-69-83-172.myvzw.com (69.83.172.164) 39.831 ms 40.240 ms 40.190 ms 4 178.sub-69-83-173.myvzw.com (69.83.173.178) 39.851 ms 48.650 ms 30.921 ms 5 194.sub-69-83-173.myvzw.com (69.83.173.194) 49.134 ms 31.131 ms 39.946 ms 6 8.sub-69-83-162.myvzw.com (69.83.162.8) 42.841 ms 39.532 ms 40.024 ms 7 73.sub-66-174-29.myvzw.com (66.174.29.73) 44.074 ms 31.248 ms 42.199 ms 8 5-1-1.bear1.phoenix1.level3.net (4.16.142.89) 38.474 ms 39.555 ms 57.860 ms 9 ae-2-70.edge1.losangeles6.level3.net (4.69.144.80) 86.101 ms 78.856 ms ae-3-80.edge1.losangeles6.level3.net (4.69.144.144) 70.001 ms 10 ae-4-90.edge1.losangeles6.level3.net (4.69.144.208) 48.840 ms ae-2-70.edge1.losangeles6.level3.net (4.69.144.80) 71.847 ms ae-3-80.edge1.losangeles6.level3.net (4.69.144.144) 81.841 ms 11 be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129) 70.538 ms 39.175 ms 45.213 ms 12 be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77) 89.463 ms 78.656 ms 72.936 ms 13 be2932.ccr22.phx02.atlas.cogentco.com (154.54.45.161) 106.896 ms 82.389 ms 87.310 ms 14 be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66) 72.782 ms 76.034 ms 114.967 ms 15 be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161) 87.883 ms 100.641 ms be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 117.904 ms 16 be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69) 144.881 ms be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 135.465 ms 110.129 ms 17 be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157) 93.713 ms be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221) 128.171 ms be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157) 109.207 ms 18 be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105) 136.620 ms be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 113.346 ms be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105) 118.301 ms 19 be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42) 148.778 ms 166.763 ms be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 153.433 ms 20 te0-4-1-7.agr21.bos01.atlas.cogentco.com (154.54.47.254) 143.449 ms te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 242.964 ms te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 198.678 ms 21 te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78) 139.454 ms te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82) 132.726 ms 107.254 ms 22 38.122.127.18 (38.122.127.18) 157.142 ms 158.882 ms 152.458 ms 23 207.154.0.57 (207.154.0.57) 144.913 ms 112.629 ms 112.519 ms 24 server1.ntcorp.com (207.154.13.58) 116.986 ms 120.276 ms 128.333 ms Miles On 2/12/17 8:57 PM, Mel Beckman wrote:
Miles,
Have you tried trace routing through cellular data connections? The results you're seeing could be explained by congestion at the point of your modem, which I think is with the cox techs are implying.
-mel via cell
On Feb 12, 2017, at 7:46 PM, Mel Beckman <mel@beckman.org> wrote:
It looks like one or more circuits are down, so you're seeing asymmetrical routing over congested paths in one direction.
-mel via cell
On Feb 12, 2017, at 7:14 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Hi Folks,
I'm visiting AZ, and seeing some really really poor performance accessing some of our servers via Cox broadband. The folks at Cox technical support are useless - all they say is "well you're on a DOCSIS 2 modem." Meanwhile, everything I'm seeing is several hops upstream of the local segment - and all that Cox level2 tech support will say is "if there was a backbone problem, our backbone people would have dealt with it."
I'm having problems reaching both our own server, and sites like google, facebook, windows update.
Traceroutes to and from our server are illustrative - note that for most of the past week, the average ping time was 85msec. Now we're seeing this:
From 98.177.135.186 - the public IP address on Cox's local broadband service. To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint data center, near Boston) traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 10.128.128.1 (10.128.128.1) 199.893 ms 75.319 ms 27.295 ms 2 100.127.69.178 (100.127.69.178) 38.710 ms 40.075 ms 43.598 ms 3 72.215.229.22 (72.215.229.22) 39.674 ms 201.368 ms * 4 lag-157.bear2.phoenix1.level3.net (4.28.82.53) 686.499 ms 1837.141 ms 16.273 ms 5 ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 35.498 ms 964.377 ms * 6 ae-3-80.edge1.losangeles6.level3.net (4.69.144.144) 551.760 ms 525.014 ms ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 2061.191 ms 7 be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129) 847.778 ms 87.601 ms 71.504 ms 8 be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1) 79.060 ms 225.647 ms be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77) 60.306 ms 9 * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms 185.180 ms 10 be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66) 61.208 ms be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78) 386.149 ms 1278.868 ms 11 be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161) 384.136 ms be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms 615.415 ms 12 be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 233.061 ms be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69) 87.902 ms be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 861.159 ms 13 be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221) 998.858 ms be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157) 249.930 ms * 14 be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 768.461 ms be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105) 136.772 ms be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 288.225 ms 15 be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 271.736 ms 166.224 ms be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42) 565.015 ms 16 te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms 17 te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82) 107.207 ms te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78) 295.881 ms 185.453 ms 18 38.122.127.18 (38.122.127.18) 115.652 ms 461.168 ms 615.526 ms 19 207.154.0.57 (207.154.0.57) 1871.023 ms 1987.832 ms 2165.248 ms 20 server1.ntcorp.com (207.154.13.58) 587.560 ms 263.328 ms 333.542 ms
Traceroute in the reverse direction: 1 207.154.13.47 (207.154.13.47) 0.000 ms 0.000 ms 0.000 ms 2 * * * 3 h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms 0.000 ms 0.000 ms 4 xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms 12.000 ms 12.000 ms 5 et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms 20.000 ms 20.000 ms 6 10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37) 72.001 ms 20.000 ms 16.000 ms 7 chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193) 20.000 ms 20.000 ms 20.000 ms 8 chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211) 72.001 ms 72.001 ms 72.001 ms 9 * * * 10 * * * 11 * * * <snip> 30 * * * <looks like a lot of the later hops don't respond to pings>
Two things jump out at me: 1. The rather large number of hops from cox to ntcorp - with high delays from several nodes in both the level3 and cogent networks. 2. That there's a rather more direct path from the datacenter to cox, that shows up in the reverse direction.
Some kind of routing or peering issue, perhaps? (And I also note the earlier string of messages regarding youtube streaming problems - that also seemed to involve cox and level3.
Thanks for any insight (and better, for any fixes!).
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
From 98.177.135.186 - the public IP address on Cox's local broadband service. To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint data center, near Boston)
It helps to follow some general guidelines to traceroute interpretation: 1. Intermediate hop time is not necessarily a measure of performance, as traceroute TTL expiration processing has low priority. 2. Time measurements areround-trip, not latency, and the bulk of the time may be incurred on the return trip. The only way to know for sure is bidirectional traceroutes. 3. Seeing reported latency in the first few hops indicates a probable issue on the local network level. In your original traceroute, latency jumps from 40ms to 200ms at 72.215.229.22, which is a Cox address. Since this is so close to you (third hop), I would apply guideline #3. All the hops after that have high latencies too, but if most of that is being injected in hop 3 then that means the problem started there. Verify that you don’t have high traffic volume on your individual Cox circuit (e.g. a backup running or something). If that’s not the case, then you’re likely seeing a Cox problem with their own internal traffic engineering. If Cox is doing weekend maintenance, some circuits may be out of service for a while. Cox is not good about passing that info down to residential customers, so you may never know what happened. A DIA business customer can usually ask about global tickets, but Cox is loathe to give that information out even then. -mel On Feb 12, 2017, at 8:27 PM, Miles Fidelman <mfidelman@meetinghouse.net<mailto:mfidelman@meetinghouse.net>> wrote: Now isn't that interesting. 1. Verizon mobile seems to also route through level3 and cogent. 2. But.. the performance seems to be a lot better. 3. That's really odd. 1 192.168.43.1 (192.168.43.1) 2.063 ms 2.382 ms 1.913 ms 2 7.sub-66-174-33.myvzw.com<http://7.sub-66-174-33.myvzw.com> (66.174.33.7) 50.676 ms 36.720 ms 41.142 ms 3 164.sub-69-83-172.myvzw.com<http://164.sub-69-83-172.myvzw.com> (69.83.172.164) 39.831 ms 40.240 ms 40.190 ms 4 178.sub-69-83-173.myvzw.com<http://178.sub-69-83-173.myvzw.com> (69.83.173.178) 39.851 ms 48.650 ms 30.921 ms 5 194.sub-69-83-173.myvzw.com<http://194.sub-69-83-173.myvzw.com> (69.83.173.194) 49.134 ms 31.131 ms 39.946 ms 6 8.sub-69-83-162.myvzw.com<http://8.sub-69-83-162.myvzw.com> (69.83.162.8) 42.841 ms 39.532 ms 40.024 ms 7 73.sub-66-174-29.myvzw.com<http://73.sub-66-174-29.myvzw.com> (66.174.29.73) 44.074 ms 31.248 ms 42.199 ms 8 5-1-1.bear1.phoenix1.level3.net<http://5-1-1.bear1.phoenix1.level3.net> (4.16.142.89) 38.474 ms 39.555 ms 57.860 ms 9 ae-2-70.edge1.losangeles6.level3.net<http://ae-2-70.edge1.losangeles6.level3.net> (4.69.144.80) 86.101 ms 78.856 ms ae-3-80.edge1.losangeles6.level3.net<http://ae-3-80.edge1.losangeles6.level3.net> (4.69.144.144) 70.001 ms 10 ae-4-90.edge1.losangeles6.level3.net<http://ae-4-90.edge1.losangeles6.level3.net> (4.69.144.208) 48.840 ms ae-2-70.edge1.losangeles6.level3.net<http://ae-2-70.edge1.losangeles6.level3.net> (4.69.144.80) 71.847 ms ae-3-80.edge1.losangeles6.level3.net<http://ae-3-80.edge1.losangeles6.level3.net> (4.69.144.144) 81.841 ms 11 be3036.ccr41.lax04.atlas.cogentco.com<http://ccr41.lax04.atlas.cogentco.com> (154.54.14.129) 70.538 ms 39.175 ms 45.213 ms 12 be2964.ccr21.lax01.atlas.cogentco.com<http://ccr21.lax01.atlas.cogentco.com> (154.54.44.77) 89.463 ms 78.656 ms 72.936 ms 13 be2932.ccr22.phx02.atlas.cogentco.com<http://ccr22.phx02.atlas.cogentco.com> (154.54.45.161) 106.896 ms 82.389 ms 87.310 ms 14 be2929.ccr21.elp01.atlas.cogentco.com<http://ccr21.elp01.atlas.cogentco.com> (154.54.42.66) 72.782 ms 76.034 ms 114.967 ms 15 be2928.ccr22.iah01.atlas.cogentco.com<http://ccr22.iah01.atlas.cogentco.com> (154.54.30.161) 87.883 ms 100.641 ms be2927.ccr41.iah01.atlas.cogentco.com<http://ccr41.iah01.atlas.cogentco.com> (154.54.29.221) 117.904 ms 16 be2687.ccr41.atl01.atlas.cogentco.com<http://ccr41.atl01.atlas.cogentco.com> (154.54.28.69) 144.881 ms be2690.ccr42.atl01.atlas.cogentco.com<http://ccr42.atl01.atlas.cogentco.com> (154.54.28.129) 135.465 ms 110.129 ms 17 be2112.ccr41.dca01.atlas.cogentco.com<http://ccr41.dca01.atlas.cogentco.com> (154.54.7.157) 93.713 ms be2113.ccr42.dca01.atlas.cogentco.com<http://ccr42.dca01.atlas.cogentco.com> (154.54.24.221) 128.171 ms be2112.ccr41.dca01.atlas.cogentco.com<http://ccr41.dca01.atlas.cogentco.com> (154.54.7.157) 109.207 ms 18 be2806.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco.com> (154.54.40.105) 136.620 ms be2807.ccr42.jfk02.atlas.cogentco.com<http://ccr42.jfk02.atlas.cogentco.com> (154.54.40.109) 113.346 ms be2806.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco.com> (154.54.40.105) 118.301 ms 19 be2096.ccr22.bos01.atlas.cogentco.com<http://ccr22.bos01.atlas.cogentco.com> (154.54.30.42) 148.778 ms 166.763 ms be2094.ccr21.bos01.atlas.cogentco.com<http://ccr21.bos01.atlas.cogentco.com> (154.54.30.14) 153.433 ms 20 te0-4-1-7.agr21.bos01.atlas.cogentco.com<http://agr21.bos01.atlas.cogentco.com> (154.54.47.254) 143.449 ms te0-4-1-6.agr21.bos01.atlas.cogentco.com<http://agr21.bos01.atlas.cogentco.com> (154.54.47.230) 242.964 ms te0-4-1-6.agr22.bos01.atlas.cogentco.com<http://agr22.bos01.atlas.cogentco.com> (154.54.80.10) 198.678 ms 21 te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com<http://b000254-0.bos01.atlas.cogentco.com> (154.24.9.78) 139.454 ms te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com<http://b000254-0.bos01.atlas.cogentco.com> (154.24.9.82) 132.726 ms 107.254 ms 22 38.122.127.18 (38.122.127.18) 157.142 ms 158.882 ms 152.458 ms 23 207.154.0.57 (207.154.0.57) 144.913 ms 112.629 ms 112.519 ms 24 server1.ntcorp.com<http://server1.ntcorp.com> (207.154.13.58) 116.986 ms 120.276 ms 128.333 ms Miles On 2/12/17 8:57 PM, Mel Beckman wrote: Miles, Have you tried trace routing through cellular data connections? The results you're seeing could be explained by congestion at the point of your modem, which I think is with the cox techs are implying. -mel via cell On Feb 12, 2017, at 7:46 PM, Mel Beckman <mel@beckman.org><mailto:mel@beckman.org> wrote: It looks like one or more circuits are down, so you're seeing asymmetrical routing over congested paths in one direction. -mel via cell On Feb 12, 2017, at 7:14 PM, Miles Fidelman <mfidelman@meetinghouse.net><mailto:mfidelman@meetinghouse.net> wrote: Hi Folks, I'm visiting AZ, and seeing some really really poor performance accessing some of our servers via Cox broadband. The folks at Cox technical support are useless - all they say is "well you're on a DOCSIS 2 modem." Meanwhile, everything I'm seeing is several hops upstream of the local segment - and all that Cox level2 tech support will say is "if there was a backbone problem, our backbone people would have dealt with it." I'm having problems reaching both our own server, and sites like google, facebook, windows update. Traceroutes to and from our server are illustrative - note that for most of the past week, the average ping time was 85msec. Now we're seeing this: traceroute to ntcorp.com<http://ntcorp.com> (207.154.13.58), 64 hops max, 52 byte packets 1 10.128.128.1 (10.128.128.1) 199.893 ms 75.319 ms 27.295 ms 2 100.127.69.178 (100.127.69.178) 38.710 ms 40.075 ms 43.598 ms 3 72.215.229.22 (72.215.229.22) 39.674 ms 201.368 ms * 4 lag-157.bear2.phoenix1.level3.net<http://lag-157.bear2.phoenix1.level3.net> (4.28.82.53) 686.499 ms 1837.141 ms 16.273 ms 5 ae-1-60.edge1.losangeles6.level3.net<http://ae-1-60.edge1.losangeles6.level3.net> (4.69.144.16) 35.498 ms 964.377 ms * 6 ae-3-80.edge1.losangeles6.level3.net<http://ae-3-80.edge1.losangeles6.level3.net> (4.69.144.144) 551.760 ms 525.014 ms ae-1-60.edge1.losangeles6.level3.net<http://ae-1-60.edge1.losangeles6.level3.net> (4.69.144.16) 2061.191 ms 7 be3036.ccr41.lax04.atlas.cogentco.com<http://ccr41.lax04.atlas.cogentco.com> (154.54.14.129) 847.778 ms 87.601 ms 71.504 ms 8 be2965.ccr22.lax01.atlas.cogentco.com<http://ccr22.lax01.atlas.cogentco.com> (154.54.45.1) 79.060 ms 225.647 ms be2964.ccr21.lax01.atlas.cogentco.com<http://ccr21.lax01.atlas.cogentco.com> (154.54.44.77) 60.306 ms 9 * be2931.ccr21.phx02.atlas.cogentco.com<http://ccr21.phx02.atlas.cogentco.com> (154.54.44.85) 2264.071 ms 185.180 ms 10 be2929.ccr21.elp01.atlas.cogentco.com<http://ccr21.elp01.atlas.cogentco.com> (154.54.42.66) 61.208 ms be2930.ccr21.elp01.atlas.cogentco.com<http://ccr21.elp01.atlas.cogentco.com> (154.54.42.78) 386.149 ms 1278.868 ms 11 be2928.ccr22.iah01.atlas.cogentco.com<http://ccr22.iah01.atlas.cogentco.com> (154.54.30.161) 384.136 ms be2927.ccr41.iah01.atlas.cogentco.com<http://ccr41.iah01.atlas.cogentco.com> (154.54.29.221) 2339.833 ms 615.415 ms 12 be2690.ccr42.atl01.atlas.cogentco.com<http://ccr42.atl01.atlas.cogentco.com> (154.54.28.129) 233.061 ms be2687.ccr41.atl01.atlas.cogentco.com<http://ccr41.atl01.atlas.cogentco.com> (154.54.28.69) 87.902 ms be2690.ccr42.atl01.atlas.cogentco.com<http://ccr42.atl01.atlas.cogentco.com> (154.54.28.129) 861.159 ms 13 be2113.ccr42.dca01.atlas.cogentco.com<http://ccr42.dca01.atlas.cogentco.com> (154.54.24.221) 998.858 ms be2112.ccr41.dca01.atlas.cogentco.com<http://ccr41.dca01.atlas.cogentco.com> (154.54.7.157) 249.930 ms * 14 be2807.ccr42.jfk02.atlas.cogentco.com<http://ccr42.jfk02.atlas.cogentco.com> (154.54.40.109) 768.461 ms be2806.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco.com> (154.54.40.105) 136.772 ms be2807.ccr42.jfk02.atlas.cogentco.com<http://ccr42.jfk02.atlas.cogentco.com> (154.54.40.109) 288.225 ms 15 be2094.ccr21.bos01.atlas.cogentco.com<http://ccr21.bos01.atlas.cogentco.com> (154.54.30.14) 271.736 ms 166.224 ms be2096.ccr22.bos01.atlas.cogentco.com<http://ccr22.bos01.atlas.cogentco.com> (154.54.30.42) 565.015 ms 16 te0-4-1-7.agr22.bos01.atlas.cogentco.com<http://agr22.bos01.atlas.cogentco.com> (154.54.80.34) 1944.479 ms te0-4-1-6.agr22.bos01.atlas.cogentco.com<http://agr22.bos01.atlas.cogentco.com> (154.54.80.10) 149.803 ms te0-4-1-6.agr21.bos01.atlas.cogentco.com<http://agr21.bos01.atlas.cogentco.com> (154.54.47.230) 897.115 ms 17 te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com<http://b000254-0.bos01.atlas.cogentco.com> (154.24.9.82) 107.207 ms te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com<http://b000254-0.bos01.atlas.cogentco.com> (154.24.9.78) 295.881 ms 185.453 ms 18 38.122.127.18 (38.122.127.18) 115.652 ms 461.168 ms 615.526 ms 19 207.154.0.57 (207.154.0.57) 1871.023 ms 1987.832 ms 2165.248 ms 20 server1.ntcorp.com<http://server1.ntcorp.com> (207.154.13.58) 587.560 ms 263.328 ms 333.542 ms Traceroute in the reverse direction: 1 207.154.13.47 (207.154.13.47) 0.000 ms 0.000 ms 0.000 ms 2 * * * 3 h130.207.190.173.static.ip.windstream.net<http://static.ip.windstream.net> (173.190.207.130) 0.000 ms 0.000 ms 0.000 ms 4 xe1-2-0-0.cr01.cley01-oh.us.windstream.net<http://cr01.cley01-oh.us.windstream.net> (40.128.250.166) 12.000 ms 12.000 ms 12.000 ms 5 et11-0-0-0.cr01.chcg01-il.us.windstream.net<http://cr01.chcg01-il.us.windstream.net> (40.128.248.71) 20.000 ms 20.000 ms 20.000 ms 6 10gigabitethernet4-1.core1.chi1.he.NET<http://10gigabitethernet4-1.core1.chi1.he.net> (206.223.119.37) 72.001 ms 20.000 ms 16.000 ms 7 chgobbrj01pos010100.r2.ch.cox.net<http://chgobbrj01pos010100.r2.ch.cox.net> (68.105.30.193) 20.000 ms 20.000 ms 20.000 ms 8 chnddsrj01-ae1.0.rd.ph.cox.net<http://rd.ph.cox.net> (68.1.5.211) 72.001 ms 72.001 ms 72.001 ms 9 * * * 10 * * * 11 * * * <snip> 30 * * * <looks like a lot of the later hops don't respond to pings> Two things jump out at me: 1. The rather large number of hops from cox to ntcorp - with high delays from several nodes in both the level3 and cogent networks. 2. That there's a rather more direct path from the datacenter to cox, that shows up in the reverse direction. Some kind of routing or peering issue, perhaps? (And I also note the earlier string of messages regarding youtube streaming problems - that also seemed to involve cox and level3. Thanks for any insight (and better, for any fixes!). Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
participants (2)
-
Mel Beckman -
Miles Fidelman