On Fri, Aug 21, 2009 at 10:07:44AM -0400, Dustin Schuemann wrote:

Is anyone else seeing packet loss on Level3.

6. ge-6-11-137.car2.Detroit1.Level3.net 2.9% 35 372.1 148.6 19.4 704.8 127.4 7. ae-11-11.car1.Detroit1.Level3.net 8.6% 35 268.1 161.5 21.3 691.8 156.0 8. ae-8-8.ebr2.Chicago1.Level3.net 0.0% 35 173.6 142.8 34.5 532.4 117.1 9. ae-21-52.car1.Chicago1.Level3.net 5.7% 35 78.1 210.3 35.7 631.2 157.9

If the loss doesn't continue for every hop along the path, it isn't real loss. For example, if there was actually a problem at hop 6, it would also affect the packets that are traveling to hop 8, so hop 8 wouldn't come up loss-free. Of course it's always possible that there is loss on a particular circuit that is part of a link-aggregate bundle, which causes only certain packets hashed certain ways to go down the "bad" channel, but typically you wouldn't see any loss at all since it's so hard to target that one bad channel channel. The far more likely scenario is that you're seeing routers with overloaded control-plane or exception processor rate-limits, due to excessive icmp generation demands. Running mtr is actually a leading cause of this type of overload, so you might actually be helping contribute to the problem. Unfortunately many routers have hard-coded rate-limits for icmp generation which can't be bumped, despite the fact that we routinely hit them due to "ordinary" things like excessive mtr use. I don't have a single router which hasn't bumped the rate limits by pretty significant amounts, and most are dropping through the day under normal traffic loads: ras@arandomrouter> show pfe statistics ip icmp | match throttled 5195868898 throttled icmps Take a look at a presentation I did at NANOG 45 for more details. http://www.nanog.org/meetings/nanog45/presentations/Sunday/RAS_traceroute_N4... -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)