RTT of ICMP "TTL exceeded" messages in Level3 network remains the same throughout the network
Hi, if I make a traceroute to a host in San Jose in Level3 network from DigitalOcean server in Amsterdam, then in Level3 network(hop 6 in example below) the RTT remains the same: # traceroute -q 1 -I ZYNGA-INC.edge1.SanJose3.Level3.net traceroute to ZYNGA-INC.edge1.SanJose3.Level3.net (4.53.208.114), 30 hops max, 60 byte packets 1 5.101.103.253 (5.101.103.253) 0.265 ms 2 95.85.0.229 (95.85.0.229) 0.236 ms 3 ix-4-2-0-0.tcore1.AV2-Amsterdam.as6453.net (195.219.194.25) 0.275 ms 4 if-7-2.tcore1.AD1-Amsterdam.as6453.net (195.219.194.46) 0.630 ms 5 4.68.63.41 (4.68.63.41) 0.635 ms 6 vl-3603-ve-227.csw2.Amsterdam1.Level3.net (4.69.162.153) 155.309 ms 7 ae-56-221.ebr2.Amsterdam1.Level3.net (4.69.153.201) 155.627 ms 8 ae-46-46.ebr2.London1.Level3.net (4.69.143.74) 153.470 ms 9 * 10 ae-61-61.csw1.NewYork1.Level3.net (4.69.134.66) 148.972 ms 11 * 12 ae-2-2.ebr1.SanJose1.Level3.net (4.69.135.185) 147.881 ms 13 ae-91-91.csw4.SanJose1.Level3.net (4.69.153.14) 149.632 ms 14 ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208) 151.107 ms 15 ZYNGA-INC.edge1.SanJose3.Level3.net (4.53.208.114) 154.431 ms # In other words, one sees the RTT of the end-host as a RTT for all the hops in Level3 netwotk. If I make the traceroute to penultimate hop ae-4-90.edge1.SanJose3.Level3.net, then RTT is as expected: root@vserver:~# traceroute -q 1 -I ae-4-90.edge1.SanJose3.Level3.net traceroute to ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208), 30 hops max, 60 byte packets 1 5.101.103.254 (5.101.103.254) 0.228 ms 2 95.85.0.237 (95.85.0.237) 0.217 ms 3 ix-4-2-0-0.tcore1.AV2-Amsterdam.as6453.net (195.219.194.25) 0.276 ms 4 if-7-2.tcore1.AD1-Amsterdam.as6453.net (195.219.194.46) 0.656 ms 5 4.68.63.41 (4.68.63.41) 0.607 ms 6 vl-3604-ve-228.csw2.Amsterdam1.Level3.net (4.69.162.157) 0.696 ms 7 ae-56-221.ebr2.Amsterdam1.Level3.net (4.69.153.201) 0.677 ms 8 ae-45-45.ebr2.London1.Level3.net (4.69.143.70) 7.059 ms 9 ae-44-44.ebr1.NewYork1.Level3.net (4.69.137.78) 76.311 ms 10 ae-81-81.csw3.NewYork1.Level3.net (4.69.134.74) 76.265 ms 11 ae-82-82.ebr2.NewYork1.Level3.net (4.69.148.41) 76.820 ms 12 ae-2-2.ebr1.SanJose1.Level3.net (4.69.135.185) 149.101 ms 13 ae-91-91.csw4.SanJose1.Level3.net (4.69.153.14) 150.557 ms 14 ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208) 162.022 ms root@vserver:~# All the ICMP "TTL exceeded" messages except the first and the penultimate one in Level3 network have MPLS extensions header(s24.postimg.org/4z9at9z45/ICMP_echo_reply_MPLS_extensions.png) which is always the same except the tag value changes. How does this technically work? What are the advantages of such setup? thanks, Martin
How does this technically work? What are the advantages of such setup?
http://forums.juniper.net/t5/Routing/what-does-quot-icmp-tunneling-quot-mean... http://www.juniper.net/techpubs/en_US/junos12.1/topics/usage-guidelines/mpls... ...and from https://www.nanog.org/meetings/nanog49/presentations/Sunday/mpls-nanog49.pdf:
Some networks also run MPLS-only cores, which carry no IP routes. • This presents a problem, since if they did want to show the hops in traceroute, the router can’t do IP routing to return the ICMP TTL Exceed. • To solve this problem, an “icmp tunneling” feature was implemented. • If an ICMP message is generated inside an LSP, the ICMP message is carried all the way to the end of the LSP before being routed back. • This can make traceroute look really weird, since you see all the hops along the LSP, but they all appear to have the same latency as the final hop. This causes much end-user confusion.
-- Hugo On Wed 2014-Aug-13 18:25:11 +0300, Martin T <m4rtntns@gmail.com> wrote:
Hi,
if I make a traceroute to a host in San Jose in Level3 network from DigitalOcean server in Amsterdam, then in Level3 network(hop 6 in example below) the RTT remains the same:
# traceroute -q 1 -I ZYNGA-INC.edge1.SanJose3.Level3.net traceroute to ZYNGA-INC.edge1.SanJose3.Level3.net (4.53.208.114), 30 hops max, 60 byte packets 1 5.101.103.253 (5.101.103.253) 0.265 ms 2 95.85.0.229 (95.85.0.229) 0.236 ms 3 ix-4-2-0-0.tcore1.AV2-Amsterdam.as6453.net (195.219.194.25) 0.275 ms 4 if-7-2.tcore1.AD1-Amsterdam.as6453.net (195.219.194.46) 0.630 ms 5 4.68.63.41 (4.68.63.41) 0.635 ms 6 vl-3603-ve-227.csw2.Amsterdam1.Level3.net (4.69.162.153) 155.309 ms 7 ae-56-221.ebr2.Amsterdam1.Level3.net (4.69.153.201) 155.627 ms 8 ae-46-46.ebr2.London1.Level3.net (4.69.143.74) 153.470 ms 9 * 10 ae-61-61.csw1.NewYork1.Level3.net (4.69.134.66) 148.972 ms 11 * 12 ae-2-2.ebr1.SanJose1.Level3.net (4.69.135.185) 147.881 ms 13 ae-91-91.csw4.SanJose1.Level3.net (4.69.153.14) 149.632 ms 14 ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208) 151.107 ms 15 ZYNGA-INC.edge1.SanJose3.Level3.net (4.53.208.114) 154.431 ms #
In other words, one sees the RTT of the end-host as a RTT for all the hops in Level3 netwotk. If I make the traceroute to penultimate hop ae-4-90.edge1.SanJose3.Level3.net, then RTT is as expected:
root@vserver:~# traceroute -q 1 -I ae-4-90.edge1.SanJose3.Level3.net traceroute to ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208), 30 hops max, 60 byte packets 1 5.101.103.254 (5.101.103.254) 0.228 ms 2 95.85.0.237 (95.85.0.237) 0.217 ms 3 ix-4-2-0-0.tcore1.AV2-Amsterdam.as6453.net (195.219.194.25) 0.276 ms 4 if-7-2.tcore1.AD1-Amsterdam.as6453.net (195.219.194.46) 0.656 ms 5 4.68.63.41 (4.68.63.41) 0.607 ms 6 vl-3604-ve-228.csw2.Amsterdam1.Level3.net (4.69.162.157) 0.696 ms 7 ae-56-221.ebr2.Amsterdam1.Level3.net (4.69.153.201) 0.677 ms 8 ae-45-45.ebr2.London1.Level3.net (4.69.143.70) 7.059 ms 9 ae-44-44.ebr1.NewYork1.Level3.net (4.69.137.78) 76.311 ms 10 ae-81-81.csw3.NewYork1.Level3.net (4.69.134.74) 76.265 ms 11 ae-82-82.ebr2.NewYork1.Level3.net (4.69.148.41) 76.820 ms 12 ae-2-2.ebr1.SanJose1.Level3.net (4.69.135.185) 149.101 ms 13 ae-91-91.csw4.SanJose1.Level3.net (4.69.153.14) 150.557 ms 14 ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208) 162.022 ms root@vserver:~#
All the ICMP "TTL exceeded" messages except the first and the penultimate one in Level3 network have MPLS extensions header(s24.postimg.org/4z9at9z45/ICMP_echo_reply_MPLS_extensions.png) which is always the same except the tag value changes.
How does this technically work? What are the advantages of such setup?
thanks, Martin
Depends on the setup. With MPLS and traffic engineering tunnels you can make a 30 hop path look like one hop easily. On 8/13/14, 9:25 AM, "Martin T" <m4rtntns@gmail.com> wrote:
Hi,
if I make a traceroute to a host in San Jose in Level3 network from DigitalOcean server in Amsterdam, then in Level3 network(hop 6 in example below) the RTT remains the same:
# traceroute -q 1 -I ZYNGA-INC.edge1.SanJose3.Level3.net traceroute to ZYNGA-INC.edge1.SanJose3.Level3.net (4.53.208.114), 30 hops max, 60 byte packets 1 5.101.103.253 (5.101.103.253) 0.265 ms 2 95.85.0.229 (95.85.0.229) 0.236 ms 3 ix-4-2-0-0.tcore1.AV2-Amsterdam.as6453.net (195.219.194.25) 0.275 ms 4 if-7-2.tcore1.AD1-Amsterdam.as6453.net (195.219.194.46) 0.630 ms 5 4.68.63.41 (4.68.63.41) 0.635 ms 6 vl-3603-ve-227.csw2.Amsterdam1.Level3.net (4.69.162.153) 155.309 ms 7 ae-56-221.ebr2.Amsterdam1.Level3.net (4.69.153.201) 155.627 ms 8 ae-46-46.ebr2.London1.Level3.net (4.69.143.74) 153.470 ms 9 * 10 ae-61-61.csw1.NewYork1.Level3.net (4.69.134.66) 148.972 ms 11 * 12 ae-2-2.ebr1.SanJose1.Level3.net (4.69.135.185) 147.881 ms 13 ae-91-91.csw4.SanJose1.Level3.net (4.69.153.14) 149.632 ms 14 ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208) 151.107 ms 15 ZYNGA-INC.edge1.SanJose3.Level3.net (4.53.208.114) 154.431 ms #
In other words, one sees the RTT of the end-host as a RTT for all the hops in Level3 netwotk. If I make the traceroute to penultimate hop ae-4-90.edge1.SanJose3.Level3.net, then RTT is as expected:
root@vserver:~# traceroute -q 1 -I ae-4-90.edge1.SanJose3.Level3.net traceroute to ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208), 30 hops max, 60 byte packets 1 5.101.103.254 (5.101.103.254) 0.228 ms 2 95.85.0.237 (95.85.0.237) 0.217 ms 3 ix-4-2-0-0.tcore1.AV2-Amsterdam.as6453.net (195.219.194.25) 0.276 ms 4 if-7-2.tcore1.AD1-Amsterdam.as6453.net (195.219.194.46) 0.656 ms 5 4.68.63.41 (4.68.63.41) 0.607 ms 6 vl-3604-ve-228.csw2.Amsterdam1.Level3.net (4.69.162.157) 0.696 ms 7 ae-56-221.ebr2.Amsterdam1.Level3.net (4.69.153.201) 0.677 ms 8 ae-45-45.ebr2.London1.Level3.net (4.69.143.70) 7.059 ms 9 ae-44-44.ebr1.NewYork1.Level3.net (4.69.137.78) 76.311 ms 10 ae-81-81.csw3.NewYork1.Level3.net (4.69.134.74) 76.265 ms 11 ae-82-82.ebr2.NewYork1.Level3.net (4.69.148.41) 76.820 ms 12 ae-2-2.ebr1.SanJose1.Level3.net (4.69.135.185) 149.101 ms 13 ae-91-91.csw4.SanJose1.Level3.net (4.69.153.14) 150.557 ms 14 ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208) 162.022 ms root@vserver:~#
All the ICMP "TTL exceeded" messages except the first and the penultimate one in Level3 network have MPLS extensions header(s24.postimg.org/4z9at9z45/ICMP_echo_reply_MPLS_extensions.png) which is always the same except the tag value changes.
How does this technically work? What are the advantages of such setup?
thanks, Martin
participants (3)
-
Corey Touchet -
Hugo Slabbert -
Martin T