Help me make sense of these traceroutes please
Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing. First some background, I have Level3 circuits in the US and some services in Europe. From Comcast to the US level3 IPs the performance is excellent. The same traceroute to Europe is terrible. The strange part is the problem appears to begin stateside on the same infrastructure that carriers the us traffic. Here is a trace to one of my IPs in the US from Comcast Tracing route to 4.30.x.x over a maximum of 30 hops 1 3 ms 1 ms 1 ms 10.1.1.1 2 30 ms 29 ms 29 ms 71.62.150.1 3 9 ms 9 ms 9 ms xe-0-1-0-32767-sur01.winchester.va.richmond.comc ast.net [68.85.71.165] 4 9 ms 14 ms 10 ms xe-9-0-3-0-ar02.staplesmllrd.va.richmond.comcast .net [68.86.125.149] 5 32 ms 30 ms 34 ms 68.86.91.153 6 36 ms 38 ms 53 ms 23.30.207.98 7 34 ms 28 ms 33 ms vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62] 8 29 ms 28 ms 20 ms ae-63-63.ebr3.Atlanta2.Level3.net [4.69.148.241] 9 27 ms 29 ms 30 ms ae-2-2.ebr1.Washington1.Level3.net [4.69.132.86] 10 24 ms 30 ms 24 ms ae-71-71.csw2.Washington1.Level3.net [4.69.134.1 34] 11 29 ms 31 ms 39 ms ae-41-90.car1.Washington1.Level3.net [4.69.149.1 95] 12 30 ms 30 ms 29 ms ae-2-23.edge7.Washington1.Level3.net [4.68.106.2 38] 13 38 ms 44 ms 43 ms 4.79.x.x 14 * * * Request timed out. (My firewall) 15 39 ms 39 ms 39 ms 4.30.x.x Trace complete. Now here is the same computer tracing to a level3 circuit in Ireland. Tracing route to xxx.yyy.ie [193.1.x.x] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 10.1.1.1 2 38 ms 33 ms 25 ms 71.62.150.1 3 10 ms 9 ms 9 ms xe-0-1-0-32767-sur01.winchester.va.richmond.comc ast.net [68.85.71.165] 4 14 ms 15 ms 15 ms xe-9-0-3-0-ar02.staplesmllrd.va.richmond.comcast .net [68.86.125.149] 5 28 ms 30 ms 30 ms 68.86.95.65 6 37 ms 37 ms 37 ms 23.30.207.98 7 118 ms * 218 ms vlan51.ebr1.Atlanta2.Level3.net [4.69.150.62] 8 119 ms 218 ms 119 ms ae-63-63.ebr3.Atlanta2.Level3.net [4.69.148.241] 9 221 ms 119 ms 119 ms ae-2-2.ebr1.Washington1.Level3.net [4.69.132.86] 10 118 ms 119 ms 118 ms ae-91-91.csw4.Washington1.Level3.net [4.69.134.1 42] 11 119 ms 119 ms 119 ms ae-92-92.ebr2.Washington1.Level3.net [4.69.134.1 57] 12 117 ms 126 ms 120 ms ae-43-43.ebr2.Paris1.Level3.net [4.69.137.57] 13 128 ms 118 ms 120 ms ae-6-6.car1.Dublin3.Level3.net [4.69.148.53] 14 122 ms 225 ms 124 ms 4.69.148.58 15 124 ms 118 ms 120 ms ae-11-11.car1.Dublin1.Level3.net [4.69.136.93] Notice that the hop from 23.30.207.98 to 4.69.150.62 seems very respectable at around 30ms for US bound traffic. However when I'm tracing from the same Comcast network to an IP that is in Europe the very same hope produces 100ms of latency and about 12% packet loss. Why does this hop treat traffic differently based on it's destination? Is this some weird result of complex asymmetrical routing or something else? I can route around this problem, but it does seem strange and I want to understand it Thanks, Sam Moats
On 2013-12-25 00:16, Sam Moats wrote:
Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing.
You are likely seeing the effects of asymmetric routing. [..]
Tracing route to xxx.yyy.ie [193.1.x.x]
www.heanet.ie by chance? :) Though you could use for instance: http://planchet.heanet.ie/toolkit/gui/reverse_traceroute.cgi to do a reverse traceroute, do make sure you force your connectivity to IPv4 as that host will do IPv6 too. (locally nullrouting the destination /128 is the trick I use for 'disabling' IPv6 temporarily). Otherwise the HEANET folks are extremely helpful and clued in, you can always ask them for help with issues. It is the end-of-year though and those Irish folks have lots of really good whiskey, Guinness thus you might have to be patient till the new year. Alternatively, you could use a tool like 'tracepath' or 'mtr' as those reports multiple answers to a response and also check for the TTL on the return packets. Greets, Jeroen
On 2013-12-24 18:55, Jeroen Massar wrote:
On 2013-12-25 00:16, Sam Moats wrote:
Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing.
You are likely seeing the effects of asymmetric routing.
That's what I was thinking to.
[..]
Tracing route to xxx.yyy.ie [193.1.x.x]
www.heanet.ie by chance? :)
Yes they were the owners of the IP I used for the example case and the heanet folks are actually totally awesome :-)
Though you could use for instance: http://planchet.heanet.ie/toolkit/gui/reverse_traceroute.cgi
to do a reverse traceroute, do make sure you force your connectivity to IPv4 as that host will do IPv6 too. (locally nullrouting the destination /128 is the trick I use for 'disabling' IPv6 temporarily).
Otherwise the HEANET folks are extremely helpful and clued in, you can always ask them for help with issues. It is the end-of-year though and those Irish folks have lots of really good whiskey, Guinness thus you might have to be patient till the new year.
Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness
Alternatively, you could use a tool like 'tracepath' or 'mtr' as those reports multiple answers to a response and also check for the TTL on the return packets.
Greets, Jeroen
Thanks, this isn't affecting my service now I've worked around it so it's more a curiosity than anything. It seems really odd to me that the same L3 edge router would route the ICMP unreachable back to me via different paths based on the final destination IP of the of the ICMP echo packet. Well its Christmas eve here and the customers are happy so Guinness seems like the best approach now :-) Thanks and have a good Holiday, Sam Moats
On 25 December 2013 00:03, Sam Moats <sam@circlenet.us> wrote:
On 2013-12-24 18:55, Jeroen Massar wrote:
On 2013-12-25 00:16, Sam Moats wrote:
Hello Nanog community, I would like to enlist your help with understanding this latency I'm seeing.
You are likely seeing the effects of asymmetric routing.
That's what I was thinking to.
[..]
Tracing route to xxx.yyy.ie [193.1.x.x]
www.heanet.ie by chance? :)
Yes they were the owners of the IP I used for the example case and the heanet folks are actually totally awesome :-)
Though you could use for instance: http://planchet.heanet.ie/toolkit/gui/reverse_traceroute.cgi
to do a reverse traceroute, do make sure you force your connectivity to IPv4 as that host will do IPv6 too. (locally nullrouting the destination /128 is the trick I use for 'disabling' IPv6 temporarily).
Otherwise the HEANET folks are extremely helpful and clued in, you can always ask them for help with issues. It is the end-of-year though and those Irish folks have lots of really good whiskey, Guinness thus you might have to be patient till the new year.
Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness
Alternatively, you could use a tool like 'tracepath' or 'mtr' as those reports multiple answers to a response and also check for the TTL on the return packets.
Greets, Jeroen
Thanks, this isn't affecting my service now I've worked around it so it's more a curiosity than anything. It seems really odd to me that the same L3 edge router would route the ICMP unreachable back to me via different paths based on the final destination IP of the of the ICMP echo packet.
Based on the data you provided, my guess is some kind of MPLS transport (please refer to https://www.nanog.org/meetings/nanog45/presentations/Sunday/RAS_traceroute_N..., pages 46-48). HTH.
Well its Christmas eve here and the customers are happy so Guinness seems like the best approach now :-)
Thanks and have a good Holiday, Sam Moats
On Tue, 24 Dec 2013 19:03:02 -0500, Sam Moats said:
Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness
I once heard the claim that if you couldn't explain your network design and have the listener understand it after you had split a pitcher of Guiness, it was probably too complicated.
Pitcher of Guinness!?! What blasphemy is this, the only way to drink it is via individually poured pint glasses. Back to the issues I'd say MPLS or GHCQ before NSA. On 25 Dec 2013 15:52, <Valdis.Kletnieks@vt.edu> wrote:
On Tue, 24 Dec 2013 19:03:02 -0500, Sam Moats said:
Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness
I once heard the claim that if you couldn't explain your network design and have the listener understand it after you had split a pitcher of Guiness, it was probably too complicated.
Thats why you're a bacon zombie. If you were a living person you'd know free beer tastes the same irrespective of the containment vessel. ;) I hope Santa brought all of you what you wanted. If not, blame UPS. Sent from my Mobile Device. -------- Original message -------- From: Bacon Zombie <baconzombie@gmail.com> Date: 12/25/2013 11:24 AM (GMT-09:00) To: Valdis.Kletnieks@vt.edu Cc: sam@circlenet.us,nanog@nanog.org Subject: Re: Help me make sense of these traceroutes please Pitcher of Guinness!?! What blasphemy is this, the only way to drink it is via individually poured pint glasses. Back to the issues I'd say MPLS or GHCQ before NSA. On 25 Dec 2013 15:52, <Valdis.Kletnieks@vt.edu> wrote:
On Tue, 24 Dec 2013 19:03:02 -0500, Sam Moats said:
Also you'd be amazed how many network issues can be solved with a bunch of IT folks and an ample supply of Guinness
I once heard the claim that if you couldn't explain your network design and have the listener understand it after you had split a pitcher of Guiness, it was probably too complicated.
participants (6)
-
Bacon Zombie -
Jeroen Massar -
Pedro Cavaca -
Sam Moats -
Valdis.Kletnieks＠vt.edu -
Warren Bailey