Interestingly enough, when I trace from my Cisco router it seems to show some MPLS labels after the hop of interest (12.88.71.13 to 12.122.112.78, only 24 msec here!). I'm not sure how our Cisco box derives these from a foreign network. Router#traceroute 69.28.226.193 Type escape sequence to abort. Tracing the route to 69.28.226.193 1 sxct.sxcy.mtcnet.net (167.142.156.197) 0 msec 0 msec 0 msec 2 siouxcenter.sxcy.137.netins.net (167.142.180.137) 4 msec 4 msec 4 msec 3 ins-b12-et-4-0-112.desm.netins.net (167.142.57.106) 8 msec 8 msec 8 msec 4 ins-h2-et-1-10-127.desm.netins.net (167.142.57.129) 8 msec 8 msec 8 msec 5 ins-c2-et-pc2-0.desm.netins.net (167.142.57.142) 8 msec 8 msec 8 msec 6 12.88.71.13 28 msec 24 msec 28 msec 7 tbr2.sl9mo.ip.att.net (12.122.112.78) [MPLS: Label 30663 Exp 0] 52 msec 48 msec 52 msec 8 cr2.sl9mo.ip.att.net (12.122.18.69) [MPLS: Label 17306 Exp 0] 52 msec 52 msec 52 msec 9 cr2.cgcil.ip.att.net (12.122.2.21) [MPLS: Label 16558 Exp 0] 52 msec 52 msec 52 msec 10 cr1.cgcil.ip.att.net (12.122.2.53) [MPLS: Label 17002 Exp 0] 48 msec 52 msec 52 msec 11 cr1.n54ny.ip.att.net (12.122.1.189) [MPLS: Label 17033 Exp 0] 52 msec 52 msec 48 msec 12 tbr1.n54ny.ip.att.net (12.122.16.138) [MPLS: Label 32364 Exp 0] 52 msec 52 msec 52 msec 13 12.122.86.165 48 msec 48 msec 52 msec 14 12.118.100.58 60 msec 60 msec 64 msec 15 oc48-po2-0.tor-151f7-cor-2.peer1.net (216.187.115.125) 52 msec 52 msec 68 msec 16 oc48-po7-0.tor-151f-dis-1.peer1.net (216.187.114.149) 52 msec 52 msec 48 msec 17 tor-fe3-5a.ne.peer1.net (216.187.68.6) 52 msec 52 msec * Router# Wondering why the RTT dropped to 24 msec for that hop, I entered both 69.28.226.192 and the IP address that my customer has been complaining about (12.129.255.4) into PingPlotter and I see that those behave very differently. I'm now guessing that AT&T is routing back traffic sent to 12.129.255.4 in a different way (perhaps asymmetrically) than traffic sent to 69.28.226.192, but it doesn't show up until it hits 12.122.112.22. Perhaps it's all those 1's and 2'. ;) I notice that in the low RTT trace router 12.88.71.13 goes to tbr2.sl9mo.ip.att.net (12.122.112.78), but in the high RTT trace, roouter 12.88.71.13 goes to tbr1.sl9mo.ip.att.net (12.122.112.22). Must be something about the way AT&T gets to tbr1.sl9mo.ip.att.net (12.122.112.22). I can't traceroute to either of those networks directly. In fact, I don't appear to be able to traceroute to any of the 12.122.x.x or 12.129.x.x I see in my traceroutes, perhaps because AT&T uses some of that space internally and doesn't advertise it. Frank From: Robert Richardson [mailto:bobrmr@gmail.com] Sent: Thursday, June 26, 2008 8:20 PM To: John T. Yocum Cc: frnkblk@iname.com; nanog list Subject: Re: Possible explanations for a large hop in latency They probably don't propagate TTL w/in their MPLS core. Depending on how they have MPLS implemented, you may only see 2 hops on the network; the ingress and egress routers. If the ingress router was in NYC and the egress in Seattle, you could understandably expect a large jump in RTT. Not an ATT customer but do know other providers run their MPLS core's this way... -Robert On Thu, Jun 26, 2008 at 6:09 PM, John T. Yocum <john@fluidhosting.com> wrote: The explanation I got, was that the latency seen at the first hop was actually a reply from the last hop in the path across their MPLS network. Hence, all the following hops had very similar latency. Personally, I thought it was rather strange for them to do that. And, I've never seen that occur on any other network. Perhaps someone from ATT would like to chime in. --John Frank Bulk - iNAME wrote: Did that satisfy you? I guess with MPLS they could tag the traffic and send it around the country twice and I wouldn't see it at L3. Frank -----Original Message----- From: John T. Yocum [mailto:john@fluidhosting.com] Sent: Thursday, June 26, 2008 7:04 PM To: frnkblk@iname.com Cc: nanog list Subject: Re: Possible explanations for a large hop in latency When I asked ATT about the sudden latency jump I see in traceroutes, they told me it was due to how their MPLS network is setup. --John Frank Bulk wrote: Our upstream provider has a connection to AT&T (12.88.71.13 <http://12.88.71.13/> ) where I relatively consistently measure with a RTT of 15 msec, but the next hop (12.122.112.22 <http://12.122.112.22/> ) comes in with a RTT of 85 msec. Unless AT&T is sending that traffic over a cable modem or to Europe and back, I can't see a reason why there is a consistent ~70 msec jump in RTT. Hops farther along the route are just a few msec more each hop, so it doesn't appear that 12.122.112.22 <http://12.122.112.22/> has some kind of ICMP rate-limiting. Is this a real performance issue, or is there some logical explanation? Frank