Possible explanations for a large hop in latency
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending that traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting. Is this a real performance issue, or is there some logical explanation? Frank
Deep Packet Inspection engine delay. <G> On Jun 26, 2008, at 6:51 PM, Frank Bulk wrote:
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending that traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting.
Is this a real performance issue, or is there some logical explanation?
Frank
James R. Cutler james.cutler@consultant.com
When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup. --John Frank Bulk wrote:
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending that traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting.
Is this a real performance issue, or is there some logical explanation?
Frank
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending
Did that satisfy you? I guess with MPLS they could tag the traffic and send it around the country twice and I wouldn't see it at L3. Frank -----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup. --John Frank Bulk wrote: that
traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting.
Is this a real performance issue, or is there some logical explanation?
Frank
The explanation I got, was that the latency seen at the first hop was actually a reply from the last hop in the path across their MPLS network. Hence, all the following hops had very similar latency. Personally, I thought it was rather strange for them to do that. And, I've never seen that occur on any other network. Perhaps someone from ATT would like to chime in. --John Frank Bulk - iNAME wrote:
Did that satisfy you? I guess with MPLS they could tag the traffic and send it around the country twice and I wouldn't see it at L3.
Frank
-----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency
When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup.
--John
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending
Frank Bulk wrote: that
traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting.
Is this a real performance issue, or is there some logical explanation?
Frank
On Thu, 26 Jun 2008, John T. Yocum wrote:
The explanation I got, was that the latency seen at the first hop was actually a reply from the last hop in the path across their MPLS network. Hence, all the following hops had very similar latency.
Personally, I thought it was rather strange for them to do that. And, I've never seen that occur on any other network.
This is standard for MPLS, the ICMP TTL expire message is sent along the LSP and returned via the router at the end of the LSP. -- Mikael Abrahamsson email: swmike@swm.pp.se
They probably don't propagate TTL w/in their MPLS core. Depending on how they have MPLS implemented, you may only see 2 hops on the network; the ingress and egress routers. If the ingress router was in NYC and the egress in Seattle, you could understandably expect a large jump in RTT. Not an ATT customer but do know other providers run their MPLS core's this way... -Robert On Thu, Jun 26, 2008 at 6:09 PM, John T. Yocum <john@fluidhosting.com> wrote:
The explanation I got, was that the latency seen at the first hop was actually a reply from the last hop in the path across their MPLS network. Hence, all the following hops had very similar latency.
Personally, I thought it was rather strange for them to do that. And, I've never seen that occur on any other network.
Perhaps someone from ATT would like to chime in.
--John
Frank Bulk - iNAME wrote:
Did that satisfy you? I guess with MPLS they could tag the traffic and send it around the country twice and I wouldn't see it at L3.
Frank
-----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency
When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup.
--John
Frank Bulk wrote:
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending
that
traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting.
Is this a real performance issue, or is there some logical explanation?
Frank
Interestingly enough, when I trace from my Cisco router it seems to show some MPLS labels after the hop of interest (12.88.71.13 to 12.122.112.78, only 24 msec here!). I'm not sure how our Cisco box derives these from a foreign network. Router#traceroute 69.28.226.193 Type escape sequence to abort. Tracing the route to 69.28.226.193 1 sxct.sxcy.mtcnet.net (167.142.156.197) 0 msec 0 msec 0 msec 2 siouxcenter.sxcy.137.netins.net (167.142.180.137) 4 msec 4 msec 4 msec 3 ins-b12-et-4-0-112.desm.netins.net (167.142.57.106) 8 msec 8 msec 8 msec 4 ins-h2-et-1-10-127.desm.netins.net (167.142.57.129) 8 msec 8 msec 8 msec 5 ins-c2-et-pc2-0.desm.netins.net (167.142.57.142) 8 msec 8 msec 8 msec 6 12.88.71.13 28 msec 24 msec 28 msec 7 tbr2.sl9mo.ip.att.net (12.122.112.78) [MPLS: Label 30663 Exp 0] 52 msec 48 msec 52 msec 8 cr2.sl9mo.ip.att.net (12.122.18.69) [MPLS: Label 17306 Exp 0] 52 msec 52 msec 52 msec 9 cr2.cgcil.ip.att.net (12.122.2.21) [MPLS: Label 16558 Exp 0] 52 msec 52 msec 52 msec 10 cr1.cgcil.ip.att.net (12.122.2.53) [MPLS: Label 17002 Exp 0] 48 msec 52 msec 52 msec 11 cr1.n54ny.ip.att.net (12.122.1.189) [MPLS: Label 17033 Exp 0] 52 msec 52 msec 48 msec 12 tbr1.n54ny.ip.att.net (12.122.16.138) [MPLS: Label 32364 Exp 0] 52 msec 52 msec 52 msec 13 12.122.86.165 48 msec 48 msec 52 msec 14 12.118.100.58 60 msec 60 msec 64 msec 15 oc48-po2-0.tor-151f7-cor-2.peer1.net (216.187.115.125) 52 msec 52 msec 68 msec 16 oc48-po7-0.tor-151f-dis-1.peer1.net (216.187.114.149) 52 msec 52 msec 48 msec 17 tor-fe3-5a.ne.peer1.net (216.187.68.6) 52 msec 52 msec * Router# Wondering why the RTT dropped to 24 msec for that hop, I entered both 69.28.226.192 and the IP address that my customer has been complaining about (12.129.255.4) into PingPlotter and I see that those behave very differently. I'm now guessing that AT&T is routing back traffic sent to 12.129.255.4 in a different way (perhaps asymmetrically) than traffic sent to 69.28.226.192, but it doesn't show up until it hits 12.122.112.22. Perhaps it's all those 1's and 2'. ;) I notice that in the low RTT trace router 12.88.71.13 goes to tbr2.sl9mo.ip.att.net (12.122.112.78), but in the high RTT trace, roouter 12.88.71.13 goes to tbr1.sl9mo.ip.att.net (12.122.112.22). Must be something about the way AT&T gets to tbr1.sl9mo.ip.att.net (12.122.112.22). I can't traceroute to either of those networks directly. In fact, I don't appear to be able to traceroute to any of the 12.122.x.x or 12.129.x.x I see in my traceroutes, perhaps because AT&T uses some of that space internally and doesn't advertise it. Frank From: Robert Richardson [mailto:bobrmr@gmail.com] Sent: Thursday, June 26, 2008 8:20 PM To: John T. Yocum Cc: frnkblk@iname.com; nanog list Subject: Re: Possible explanations for a large hop in latency They probably don't propagate TTL w/in their MPLS core. Depending on how they have MPLS implemented, you may only see 2 hops on the network; the ingress and egress routers. If the ingress router was in NYC and the egress in Seattle, you could understandably expect a large jump in RTT. Not an ATT customer but do know other providers run their MPLS core's this way... -Robert On Thu, Jun 26, 2008 at 6:09 PM, John T. Yocum <john@fluidhosting.com> wrote: The explanation I got, was that the latency seen at the first hop was actually a reply from the last hop in the path across their MPLS network. Hence, all the following hops had very similar latency. Personally, I thought it was rather strange for them to do that. And, I've never seen that occur on any other network. Perhaps someone from ATT would like to chime in. --John Frank Bulk - iNAME wrote: Did that satisfy you? I guess with MPLS they could tag the traffic and send it around the country twice and I wouldn't see it at L3. Frank -----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup. --John Frank Bulk wrote: Our upstream provider has a connection to AT&T (12.88.71.13 <http://12.88.71.13/> ) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22 <http://12.122.112.22/> ) comes in with a RTT of 85 msec. Unless AT&T is sending that traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 <http://12.122.112.22/> has some kind of ICMP rate-limiting. Is this a real performance issue, or is there some logical explanation? Frank
Just google "tbr1.sl9mo.ip.att.net" and it's clear that high latency through that point has occurred before. And guess what kind of customer complained to me about the latency? A gamer. Frank -----Original Message----- From: Frank Bulk - iNAME [mailto:frnkblk@iname.com] Sent: Thursday, June 26, 2008 9:27 PM To: 'Robert Richardson'; John T. Yocum Cc: nanog list Subject: RE: Possible explanations for a large hop in latency <snip> Wondering why the RTT dropped to 24 msec for that hop, I entered both 69.28.226.192 and the IP address that my customer has been complaining about (12.129.255.4) into PingPlotter and I see that those behave very differently. I'm now guessing that AT&T is routing back traffic sent to 12.129.255.4 in a different way (perhaps asymmetrically) than traffic sent to 69.28.226.192, but it doesn't show up until it hits 12.122.112.22. Perhaps it's all those 1's and 2'. ;) I notice that in the low RTT trace router 12.88.71.13 goes to tbr2.sl9mo.ip.att.net (12.122.112.78), but in the high RTT trace, roouter 12.88.71.13 goes to tbr1.sl9mo.ip.att.net (12.122.112.22). Must be something about the way AT&T gets to tbr1.sl9mo.ip.att.net (12.122.112.22). I can't traceroute to either of those networks directly. In fact, I don't appear to be able to traceroute to any of the 12.122.x.x or 12.129.x.x I see in my traceroutes, perhaps because AT&T uses some of that space internally and doesn't advertise it. Frank From: Robert Richardson [mailto:bobrmr@gmail.com] Sent: Thursday, June 26, 2008 8:20 PM To: John T. Yocum Cc: frnkblk@iname.com; nanog list Subject: Re: Possible explanations for a large hop in latency They probably don't propagate TTL w/in their MPLS core. Depending on how they have MPLS implemented, you may only see 2 hops on the network; the ingress and egress routers. If the ingress router was in NYC and the egress in Seattle, you could understandably expect a large jump in RTT. Not an ATT customer but do know other providers run their MPLS core's this way... -Robert On Thu, Jun 26, 2008 at 6:09 PM, John T. Yocum <john@fluidhosting.com> wrote: The explanation I got, was that the latency seen at the first hop was actually a reply from the last hop in the path across their MPLS network. Hence, all the following hops had very similar latency. Personally, I thought it was rather strange for them to do that. And, I've never seen that occur on any other network. Perhaps someone from ATT would like to chime in. --John Frank Bulk - iNAME wrote: Did that satisfy you? I guess with MPLS they could tag the traffic and send it around the country twice and I wouldn't see it at L3. Frank -----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup. --John Frank Bulk wrote: Our upstream provider has a connection to AT&T (12.88.71.13 <http://12.88.71.13/> ) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22 <http://12.122.112.22/> ) comes in with a RTT of 85 msec. Unless AT&T is sending that traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 <http://12.122.112.22/> has some kind of ICMP rate-limiting. Is this a real performance issue, or is there some logical explanation? Frank
Frank Bulk - iNAME wrote:
Just google "tbr1.sl9mo.ip.att.net" and it's clear that high latency through that point has occurred before. And guess what kind of customer complained to me about the latency? A gamer.
you can pay a lot of money for the net propagation anomaly detection services that gamers give you for free. randy
On Jun 26, 2008, at 11:36 PM, Randy Bush wrote:
Frank Bulk - iNAME wrote:
Just google "tbr1.sl9mo.ip.att.net" and it's clear that high latency through that point has occurred before. And guess what kind of customer complained to me about the latency? A gamer.
you can pay a lot of money for the net propagation anomaly detection services that gamers give you for free.
---- Many years ago I worked for a small Mom-and-Pop type ISP in New York state (I was the only network / technical person there) -- it was a very free wheeling place and I built the network by doing whatever made sense at the time. One of my "favorite" customers (Joe somebody) was somehow related to the owner of the ISP and was a gamer. This was back in the day when the gaming magazines would give you useful tips like "Type 'tracert $gameserver' and make sure that there are less than N hops". Joe would call up tech support, me, the owner, etc and complain that there was N+3 hops and most of them were in our network. I spent much time explaining things about packet-loss, latency, etc but couldn't shake his belief that hop count was the only metric that mattered. Finally, one night he called me at home well after midnight (no, I didn't give him my home phone number, he looked me up in the phonebook!) to complain that his gaming was suffering because it was "too many hops to get out of your network". I finally snapped and built a static GRE tunnel from the RAS box that he connected to all over the network -- it was a thing of beauty, it went through almost every device that we owned and took the most convoluted path I could come up with. "Yay!", I figured, "now I can demonstrate that latency is more important than hop count" and I went to bed. The next morning I get a call from him. He is ecstatic and wildly impressed by how well the network is working for him now and how great his gaming performance is. "Oh well", I think, "at least he is happy and will leave me alone now". I don't document the purpose of this GRE anywhere and after some time forget about it. A few months later I am doing some routine cleanup work and stumble across a weird looking tunnel -- its bizarre, it goes all over the place and is all kinds of crufty -- there are static routes and policy routing and bizarre things being done on the RADIUS server to make sure some user always gets a certain IP... I look in my pile of notes and old configs and then decide to just yank it out. That night I get an enraged call (at home again) from Joe *screaming* that the network is all broken again because it is now way too many hops to get out of the network and that people keep shooting him... What I learnt from this: 1: Make sure you document everything (and no, the network isn't documentation) 2: Gamers are weird. 3: Making changes to your network in anger provides short term pleasure but long term pain. ----- W
randy
-- Do not meddle in the affairs of dragons, for you are crunchy and taste good with ketchup.
On Thu, 26 Jun 2008, Frank Bulk - iNAME wrote:
Interestingly enough, when I trace from my Cisco router it seems to show some MPLS labels after the hop of interest (12.88.71.13 to 12.122.112.78, only 24 msec here!). I'm not sure how our Cisco box derives these from a foreign network.
The ICMP packet (TTL exceeded in transit) contains a copy of the packet which TTL expired, including the labels, so label information is available to traceroute. -- Mikael Abrahamsson email: swmike@swm.pp.se
Even if they are decrementing TTL inside of their MPLS core, the TTL expired message still has to traverse the entire MPLS LSP (tunnel), so the latency reported for each "hop" is in fact the latency of the last hop in the MPLS network. Always. Sam Robert Richardson wrote:
They probably don't propagate TTL w/in their MPLS core. Depending on how they have MPLS implemented, you may only see 2 hops on the network; the ingress and egress routers. If the ingress router was in NYC and the egress in Seattle, you could understandably expect a large jump in RTT.
Not an ATT customer but do know other providers run their MPLS core's this way...
-Robert
On Thu, Jun 26, 2008 at 6:09 PM, John T. Yocum <john@fluidhosting.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Sam Stickland wrote: | Even if they are decrementing TTL inside of their MPLS core, the TTL | expired message still has to traverse the entire MPLS LSP (tunnel), so | the latency reported for each "hop" is in fact the latency of the last | hop in the MPLS network. Always. | And who said tunneling protocols aren't fun :-) - -- ========= bep -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIavwUE1XcgMgrtyYRArGuAJwJa3g/BiIDqNL1L1lItDu+BL3b/ACeMrPT DtiH+THvgfPz31MAK2QmsZ4= =m5il -----END PGP SIGNATURE-----
Thanks for the added information. Even if their MPLS path went from the midwest (where I'm located) to San Francisco and then back to St. Louis (where 12.122.112.22 appears to be), I don't think that accounts for a 70 msec jump in traffic. And I don't think they would (intentionally) create such an inefficient MPLS path. Someone off-list told me they tried to trace to 12.88.71.13, but once they hit an AT&T router their ICMP traffic appeared to be blocked. Frank -----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 8:09 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency The explanation I got, was that the latency seen at the first hop was actually a reply from the last hop in the path across their MPLS network. Hence, all the following hops had very similar latency. Personally, I thought it was rather strange for them to do that. And, I've never seen that occur on any other network. Perhaps someone from ATT would like to chime in. --John Frank Bulk - iNAME wrote:
Did that satisfy you? I guess with MPLS they could tag the traffic and send it around the country twice and I wouldn't see it at L3.
Frank
-----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency
When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup.
--John
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending
Frank Bulk wrote: that
traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting.
Is this a real performance issue, or is there some logical explanation?
Frank
We had a similar situation going from Minneapolis to Kansas City via Chicago. Normal latency from Minneapolis to Chicago via Level3 MPLS network is about 14msec RTT. When the the circuit from Minneapolis to Chicago went out for one reason or another, our MPLS link went from Minneapolis to Tulsa, to Dallas, and then to Chicago.. That added a little latency in the path from Minneapolis to Chicago.. We didn't need to exceed the SLA in order to cry foul. They didn't intentionally create an inefficient path.. The problem was recognized and fixed the same day. Latency on an MPLS circuit is the cumulative latency on the Label Switch Path, and a number of the hops are invisible. The latency per hop is still the same... you just can't see that your traffic is travelling to say Denver or Dallas. Tim Peiffer Network Support Engineer Networking and Telecommunications Services University of Minnesota/NorthernLights GigaPOP Frank Bulk - iNAME wrote:
Thanks for the added information.
Even if their MPLS path went from the midwest (where I'm located) to San Francisco and then back to St. Louis (where 12.122.112.22 appears to be), I don't think that accounts for a 70 msec jump in traffic. And I don't think they would (intentionally) create such an inefficient MPLS path.
Someone off-list told me they tried to trace to 12.88.71.13, but once they hit an AT&T router their ICMP traffic appeared to be blocked.
Frank
-----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 8:09 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency
The explanation I got, was that the latency seen at the first hop was actually a reply from the last hop in the path across their MPLS network. Hence, all the following hops had very similar latency.
Personally, I thought it was rather strange for them to do that. And, I've never seen that occur on any other network.
Perhaps someone from ATT would like to chime in.
--John
Frank Bulk - iNAME wrote:
Did that satisfy you? I guess with MPLS they could tag the traffic and
send
it around the country twice and I wouldn't see it at L3.
Frank
-----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency
When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup.
--John
Frank Bulk wrote:
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending
that
traffic over a cable modem or to Europe and back, I can't see a reason
why
there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that
12.122.112.22
has some kind of ICMP rate-limiting.
Is this a real performance issue, or is there some logical explanation?
Frank
Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending
Depending on whether TTL is propagated into MPLS, this could be true. Though it should also be pointed out that ICMP responses aren't exactly a precise scientific tool... The responding router could just be busy, and the response time could be reflective of load more than link latency etc. Similarly, failure to get any response at all from a router isn't necessarily indicative of packet loss... Cheers, -Benson Sent via BlackBerry from T-Mobile -----Original Message----- From: "Frank Bulk - iNAME" <frnkblk@iname.com> Date: Thu, 26 Jun 2008 19:54:42 To:"'John T. Yocum'" <john@fluidhosting.com> Cc:nanog list <nanog@merit.edu> Subject: RE: Possible explanations for a large hop in latency Did that satisfy you? I guess with MPLS they could tag the traffic and send it around the country twice and I wouldn't see it at L3. Frank -----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup. --John Frank Bulk wrote: that
traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting.
Is this a real performance issue, or is there some logical explanation?
Frank
Just to close this issue on the list: a (top) engineer from AT&T contacted me offline and helped us out. Turns out that 12.88.71.13 is located in Kansas City and tbr1.sl9mo.ip.att.net (12.122.112.22) is in St. Louis. AT&T has two L1 connections to that site for redundancy, but traffic was flowing over the longer loop. The engineer tweaked route weights so that the traffic prefers to flow over the shorter link to tbr2.sl9mo.ip.att.net (12.122.112.78), shaving about 12 msec. He also explained that the jump of ~70 msec is due to how ICMP traffic within MPLS tunnels is handled. It wasn't until I ran a traceroute from a Cisco router that I even saw the MPLS labels (that included in the ICMP responses) for each of the hops within the tunnel. Apparently each ICMP packet within an MPLS tunnel (where TTL decrementing is allowed) is sent to the *end* of the tunnel and back again, so my next "hop" to tbr1.sl9mo.ip.att.net (12.122.112.22) was really showing the RTT to the end of the tunnel, Los Angeles. Frank -----Original Message----- From: Frank Bulk [mailto:frnkblk@iname.com] Sent: Thursday, June 26, 2008 5:52 PM To: nanog list Subject: Possible explanations for a large hop in latency Our upstream provider has a connection to AT&T (12.88.71.13) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22) comes in with a RTT of 85 msec. Unless AT&T is sending that traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 has some kind of ICMP rate-limiting. Is this a real performance issue, or is there some logical explanation? Frank
participants (12)
-
bensons＠riot.queuefull.net -
Bruce Pinsky -
Frank Bulk -
Frank Bulk - iNAME
-
James R. Cutler -
John T. Yocum -
Mikael Abrahamsson -
Randy Bush -
Robert Richardson -
Sam Stickland -
Tim Peiffer -
Warren Kumari