On Mon, Jan 31, 2005 at 04:20:31AM -0500, Charles Shen wrote:
We did a "traceroute" end-to-end routing measurement in 2004 and found about 5-10% of measuremnts exhibiting rapidly-variable routing on the time scale of a single traceroute (seconds to minutes). In other words, the packets belonging to a single traceroute took multiple paths. [...] Route change in such a short scale for packets in the same flow could be troublesome. But the occurrence of such behavior does not seem to have reduced over the past years at least from our measurements. Does anyone know how to explain this behavior? Thanks!
Yes, this is normal per-flow load balancing on parallel backbone lines. Usually, "flows" are defined via a hash on L3 and possibly L4 addressing information (IP source/dest, and TCP/UDP port source/dest, ICMP code, etc.). If the "flow hash" contains L4 information, every traceroute probe packet is considered a different "flow" and you see exactly that:
5 pos5-0-2488M.cr2.FRA2.gblx.net (67.17.65.53) 2.448 ms pos6-0-2488M.cr1.FRA2.gblx.net (67.17.65.77) 2.368 ms 2.281 ms 6 so3-0-0-2488M.ar2.FRA3.gblx.net (67.17.65.82) 2.676 ms 2.750 ms so2-0-0-2488M.ar2.FRA3.gblx.net (67.17.65.58) 2.569 ms
You would NOT see the same effect with packets of e.g. the same TCP session, so this (multipath forwarding) is usually no problem (as for TCP and UDP applications there is no reordering happening). So your analysis results (traceroute) are misleading for most real-life applications. I agree that it's irritating and I personally favor using aggregated SONET/Ethernet devices (IEEE 801.3ad) to bundle parallel lines if possible. Best regards, Daniel -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0