Help interpret a strange traceroute?
Happy monday all! Any idea how a traceroute (into my network) could end up this fubar'd? Discovered this wierd routing while investigating horrendously slow speeds (albeit no packet loss) to a particular ISP abroad. It's like - coming into us - the packets are taking every available path, simultaniously. Tracing back to him looks perfectly normal. Thank you in advance for any comments on or off list. Him to us: traceroute to mirror.ash.fastserv.com (208.85.240.29), 64 hops max, 52 byte packets 1 192.168.1.1 (192.168.1.1) 1.552 ms 1.522 ms 1.261 ms 2 br1.l.hiweb.ir (46.224.0.12) 17.750 ms 18.932 ms 18.811 ms 3 172.16.5.65 (172.16.5.65) 18.505 ms 20.044 ms 18.414 ms 4 int.gr1.s.hiweb.ir (46.224.0.125) 19.096 ms 18.447 ms 18.752 ms 5 int.gr1.l.hiweb.ir (46.224.0.117) 18.454 ms 18.768 ms 103.441 ms 6 10.21.252.166 (10.21.252.166) 66.308 ms 18.728 ms 18.867 ms 7 * * * 8 185.100.209.139 (185.100.209.139) 162.126 ms ae5.istanbul1.ist.seabone.net (93.186.132.212) 210.124 ms 126.208 ms 9 et7-1-0.franco71.fra.seabone.net (195.22.214.131) 102.694 ms if-ae-9-3.tcore1.pvu-paris.as6453.net (195.219.87.14) 189.955 ms 185.100.209.197 (185.100.209.197) 218.539 ms 10 te0-3-0-4.rcr21.b023101-0.lon01.atlas.cogentco.com (149.14.144.89) 201.223 ms 215.996 ms 185.100.209.1 (185.100.209.1) 164.381 ms 11 if-ae-2-2.tcore2.av2-amsterdam.as6453.net (195.219.194.6) 199.161 ms be2950.ccr22.lon01.atlas.cogentco.com (130.117.2.109) 216.648 ms if-ae-3-6.tcore1.l78-london.as6453.net (80.231.130.85) 271.750 ms 12 if-ae-4-2.thar1.njy-newark.as6453.net (80.231.130.34) 183.984 ms * be2814.ccr42.ams03.atlas.cogentco.com (130.117.0.141) 109.045 ms 13 if-ae-1-3.thar2.njy-newark.as6453.net (216.6.57.2) 207.371 ms be2870.ccr41.lon13.atlas.cogentco.com (154.54.58.173) 186.804 ms be12488.ccr42.lon13.atlas.cogentco.com (130.117.51.41) 190.236 ms 14 if-ae-14-14.tcore2.nto-new-york.as6453.net (66.198.111.126) 203.156 ms if-ae-9-2.tcore1.n75-new-york.as6453.net (63.243.128.122) 273.177 ms be2807.ccr42.dca01.atlas.cogentco.com (154.54.40.110) 235.632 ms 15 66.110.96.134 (66.110.96.134) 182.695 ms be2806.ccr41.dca01.atlas.cogentco.com (154.54.40.106) 278.593 ms 240.075 ms 16 66.110.96.142 (66.110.96.142) 245.454 ms be3084.ccr41.iad02.atlas.cogentco.com (154.54.30.66) 191.973 ms be3083.ccr41.iad02.atlas.cogentco.com (154.54.30.54) 240.740 ms 17 te0-0-0-0.nr13.b023801-0.iad01.atlas.cogentco.com (154.24.36.18) 257.027 ms be3083.ccr41.iad02.atlas.cogentco.com (154.54.30.54) 236.617 ms hu-1-3-0-2-cr02.newyork.ny.ibone.comcast.net (68.86.83.97) 221.186 ms 18 be-10102-cr02.ashburn.va.ibone.comcast.net (68.86.85.161) 194.766 ms 250.493 ms be-10203-cr01.newark.nj.ibone.comcast.net (68.86.85.185) 191.052 ms 19 be-10102-cr02.ashburn.va.ibone.comcast.net (68.86.85.161) 197.694 ms te0-0-2-0.nr11.b023801-0.iad01.atlas.cogentco.com (154.24.36.2) 266.914 ms 235.444 ms 20 te0-0-2-3.nr11.b023801-0.iad01.atlas.cogentco.com (154.24.36.6) 255.756 ms 38.88.249.210 (38.88.249.210) 216.677 ms te0-0-2-0.nr11.b023801-0.iad01.atlas.cogentco.com (154.24.36.2) 211.364 ms 21 208.85.240.29 (208.85.240.29) 274.692 ms 212.896 ms ash01-mls-dc-core-a.latisys.net (67.217.175.37) 208.976 ms And back to him: traceroute to 46.224.145.65 (46.224.145.65), 30 hops max, 60 byte packets 1 104.153.64.146 (104.153.64.146) 22.067 ms 22.049 ms * 2 xe-3-1-1.er1.iad10.us.above.net (208.185.24.1) 0.398 ms 0.392 ms 0.420 ms 3 zayo-tata.iad10.us.zip.zayo.com (64.125.13.170) 0.708 ms 0.580 ms 0.623 ms 4 if-ae-11-2.thar2.NJY-Newark.as6453.net (216.6.87.138) 92.900 ms if-ae-11-3.thar2.NJY-Newark.as6453.net (216.6.87.242) 92.802 ms if-ae-11-4.thar2.NJY-Newark.as6453.net (216.6.87.169) 97.177 ms 5 if-ae-1-3.thar1.NJY-Newark.as6453.net (216.6.57.1) 92.940 ms 93.008 ms 89.431 ms 6 if-ae-8-2.tcore1.LDN-London.as6453.net (66.198.70.174) 97.315 ms 96.049 ms 95.855 ms 7 if-ae-17-2.tcore1.L78-London.as6453.net (80.231.130.129) 98.074 ms if-ae-3-6.tcore1.PYE-Paris.as6453.net (80.231.130.86) 92.733 ms 92.898 ms 8 if-ae-2-2.tcore1.PVU-Paris.as6453.net (80.231.154.17) 93.636 ms if-ae-3-6.tcore1.PYE-Paris.as6453.net (80.231.130.86) 97.917 ms 98.363 ms 9 if-ae-2-2.tcore1.PVU-Paris.as6453.net (80.231.154.17) 100.909 ms if-ae-9-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.13) 92.055 ms 93.173 ms 10 195.219.87.46 (195.219.87.46) 208.894 ms 209.042 ms if-ae-9-2.tcore2.FNM-Frankfurt.as6453.net (195.219.87.9) 94.718 ms 11 * 195.219.87.46 (195.219.87.46) 209.640 ms int.gr1.s.hiweb.ir (46.224.0.118) 202.201 ms 12 int.cr1.s.hiweb.ir (46.224.0.126) 205.150 ms * * 13 * * * 14 int.gr1.s.hiweb.ir (46.224.0.118) 201.803 ms 201.799 ms int.cr1.s.hiweb.ir (46.224.0.126) 201.819 ms 15 int.cr1.s.hiweb.ir (46.224.0.126) 204.952 ms 198.969 ms * 16 46.224.145.65 (46.224.145.65) 210.507 ms 211.144 ms 206.998 ms
On Mon, Oct 31, 2016 at 3:33 PM, Randy <amps@djlab.com> wrote:
Any idea how a traceroute (into my network) could end up this fubar'd? Discovered this wierd routing while investigating horrendously slow speeds (albeit no packet loss) to a particular ISP abroad.
Hi Randy, This is per-packet load balancing. In the forward path the alternates are different lengths but the traceroute stops as soon as at least one of the paths reaches the destination. The return path is also engaged in per-packet load balancing but the paths are all the same length. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
Hi Randy, ECMP loadbalancing is most frequently done on layer3+layer4 headers, and unixlike traceroute use UDP with increasing destination port number for each packet (usually starting at 33434), which allows to see the different available paths, as wrote William. Would you want/need to stick to only one traceroute path, you may use ICMP traceroute instead of UDP traceroute (no port in ICMP, so only layer 3 available to loadbalance, so all packets will go through the same interface). Usually it is achieved by using traceroute -I yourdest Windows tracert is ICMP only traceroute by the way. MTR tool is also ICMP based by default. Keep in mind that it looses some useful information, though (since you see only one path and don't decide which). So, you can also use UDP traceroute with fixed port (by example 33434 with no port increase), and try again the same traceroute with another destport (with fixed port too, by example 33435), which would display two different paths in a more readable way. RTFM is required since the options depend on your traceroute particular specie :) Olivier
On 31 oct. 2016 à 20:42, William Herrin <bill@herrin.us> wrote :
On Mon, Oct 31, 2016 at 3:33 PM, Randy <amps@djlab.com> wrote:
Any idea how a traceroute (into my network) could end up this fubar'd? Discovered this wierd routing while investigating horrendously slow speeds (albeit no packet loss) to a particular ISP abroad.
Hi Randy,
This is per-packet load balancing. In the forward path the alternates are different lengths but the traceroute stops as soon as at least one of the paths reaches the destination.
The return path is also engaged in per-packet load balancing but the paths are all the same length.
On 10/31/16 4:20 PM, Olivier Benghozi wrote:
Hi Randy,
ECMP loadbalancing is most frequently done on layer3+layer4 headers, and unixlike traceroute use UDP with increasing destination port number for each packet (usually starting at 33434), which allows to see the different available paths, as wrote William.
Would you want/need to stick to only one traceroute path, you may use ICMP traceroute instead of UDP traceroute (no port in ICMP, so only layer 3 available to loadbalance, so all packets will go through the same interface).
Usually it is achieved by using traceroute -I yourdest Windows tracert is ICMP only traceroute by the way. MTR tool is also ICMP based by default.
Keep in mind that it looses some useful information, though (since you see only one path and don't decide which). So, you can also use UDP traceroute with fixed port (by example 33434 with no port increase), and try again the same traceroute with another destport (with fixed port too, by example 33435), which would display two different paths in a more readable way. RTFM is required since the options depend on your traceroute particular specie :)
Olivier
On 31 oct. 2016 à 20:42, William Herrin <bill@herrin.us> wrote :
On Mon, Oct 31, 2016 at 3:33 PM, Randy <amps@djlab.com> wrote:
Any idea how a traceroute (into my network) could end up this fubar'd? Discovered this wierd routing while investigating horrendously slow speeds (albeit no packet loss) to a particular ISP abroad.
Hi Randy,
This is per-packet load balancing. In the forward path the alternates are different lengths but the traceroute stops as soon as at least one of the paths reaches the destination.
The return path is also engaged in per-packet load balancing but the paths are all the same length.
This is also a handy tool that addresses the same issues: https://paris-traceroute.net/
Does anyone have an IP that involves a load balancing router to test with? On Mon, Oct 31, 2016 at 5:54 PM, Bryan Holloway <bryan@shout.net> wrote:
On 10/31/16 4:20 PM, Olivier Benghozi wrote:
Hi Randy,
ECMP loadbalancing is most frequently done on layer3+layer4 headers, and unixlike traceroute use UDP with increasing destination port number for each packet (usually starting at 33434), which allows to see the different available paths, as wrote William.
Would you want/need to stick to only one traceroute path, you may use ICMP traceroute instead of UDP traceroute (no port in ICMP, so only layer 3 available to loadbalance, so all packets will go through the same interface).
Usually it is achieved by using traceroute -I yourdest Windows tracert is ICMP only traceroute by the way. MTR tool is also ICMP based by default.
Keep in mind that it looses some useful information, though (since you see only one path and don't decide which). So, you can also use UDP traceroute with fixed port (by example 33434 with no port increase), and try again the same traceroute with another destport (with fixed port too, by example 33435), which would display two different paths in a more readable way. RTFM is required since the options depend on your traceroute particular specie :)
Olivier
On 31 oct. 2016 à 20:42, William Herrin <bill@herrin.us> wrote :
On Mon, Oct 31, 2016 at 3:33 PM, Randy <amps@djlab.com> wrote:
Any idea how a traceroute (into my network) could end up this fubar'd? Discovered this wierd routing while investigating horrendously slow speeds (albeit no packet loss) to a particular ISP abroad.
Hi Randy,
This is per-packet load balancing. In the forward path the alternates are different lengths but the traceroute stops as soon as at least one of the paths reaches the destination.
The return path is also engaged in per-packet load balancing but the paths are all the same length.
This is also a handy tool that addresses the same issues:
Hi, I am trying to determine the physical diversity of the Zayo and Level3 networks vis-a-vis each other on the European racetrack - London/Amsterdam/Frankfurt/Paris/London. It is for a client of mine. Regards, Roderick.
On 1 Nov 2016, at 00:15, Rod Beck <rod.beck@unitedcablecompany.com> wrote:
I am trying to determine the physical diversity of the Zayo and Level3 networks vis-a-vis each other on the European racetrack - London/Amsterdam/Frankfurt/Paris/London. It is for a client of mine.
try Telegeography.com best regards Wolfgang -- Wolfgang Tremmel Phone +49 69 1730902 26 | Fax +49 69 4056 2716 | Mobile +49 171 8600 816 | wolfgang.tremmel@de-cix.net Geschaeftsfuehrer Harald A. Summa | Registergericht AG Köln HRB 51135 DE-CIX Management GmbH | Lindleystrasse 12 | 60314 Frankfurt am Main | Germany | www.de-cix.net
participants (7)
-
Bryan Holloway -
Dovid Bender -
Olivier Benghozi -
Randy -
Rod Beck -
William Herrin -
Wolfgang Tremmel