comcast routing issue question
Question: What could cause the first trace below to succeed, but the second trace to fail? $ mtr 69.61.40.35 HOST: blue Loss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 1 4.3 4.3 4.3 4.3 2. 73.62.48.1 0.0% 1 10.6 10.6 10.6 10.6 3. 68.86.108.25 0.0% 1 11.4 11.4 11.4 11.4 4. 68.86.106.54 0.0% 1 9.8 9.8 9.8 9.8 5. 68.86.106.9 0.0% 1 20.5 20.5 20.5 20.5 6. 68.86.90.121 0.0% 1 11.3 11.3 11.3 11.3 7. 68.86.84.70 0.0% 1 27.7 27.7 27.7 27.7 8. 64.213.76.77 0.0% 1 24.5 24.5 24.5 24.5 9. 208.50.254.150 0.0% 1 39.4 39.4 39.4 39.4 10. 208.49.83.237 0.0% 1 46.6 46.6 46.6 46.6 11. 208.49.83.234 0.0% 1 40.7 40.7 40.7 40.7 12. 69.61.40.35 0.0% 1 43.9 43.9 43.9 43.9 $ mtr 69.61.40.34 HOST: blue Loss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 1 1.1 1.1 1.1 1.1 2. 73.62.48.1 0.0% 1 9.9 9.9 9.9 9.9 3. 68.86.108.25 0.0% 1 9.3 9.3 9.3 9.3 4. 68.86.106.54 0.0% 1 9.6 9.6 9.6 9.6 5. 68.86.106.9 0.0% 1 9.0 9.0 9.0 9.0 6. 68.86.90.121 0.0% 1 18.2 18.2 18.2 18.2 7. 68.86.84.70 0.0% 1 23.9 23.9 23.9 23.9 8. ??? 100.0 1 0.0 0.0 0.0 0.0 Taking the 69.61.40.33/28 subnet a bit further, .36 drops at 68.86.84.70 but .37 - .39 make it. .40 drops at 68.86.84.70, but .41 makes it. Crazy. -Jim P.
On Thu, 2006-11-30 at 00:06 -0500, Jim Popovitch wrote:
Question: What could cause the first trace below to succeed, but the second trace to fail?
$ mtr 69.61.40.35 HOST: blue Loss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 1 4.3 4.3 4.3 4.3 2. 73.62.48.1 0.0% 1 10.6 10.6 10.6 10.6 3. 68.86.108.25 0.0% 1 11.4 11.4 11.4 11.4 4. 68.86.106.54 0.0% 1 9.8 9.8 9.8 9.8 5. 68.86.106.9 0.0% 1 20.5 20.5 20.5 20.5 6. 68.86.90.121 0.0% 1 11.3 11.3 11.3 11.3 7. 68.86.84.70 0.0% 1 27.7 27.7 27.7 27.7 8. 64.213.76.77 0.0% 1 24.5 24.5 24.5 24.5 9. 208.50.254.150 0.0% 1 39.4 39.4 39.4 39.4 10. 208.49.83.237 0.0% 1 46.6 46.6 46.6 46.6 11. 208.49.83.234 0.0% 1 40.7 40.7 40.7 40.7 12. 69.61.40.35 0.0% 1 43.9 43.9 43.9 43.9
$ mtr 69.61.40.34 HOST: blue Loss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 1 1.1 1.1 1.1 1.1 2. 73.62.48.1 0.0% 1 9.9 9.9 9.9 9.9 3. 68.86.108.25 0.0% 1 9.3 9.3 9.3 9.3 4. 68.86.106.54 0.0% 1 9.6 9.6 9.6 9.6 5. 68.86.106.9 0.0% 1 9.0 9.0 9.0 9.0 6. 68.86.90.121 0.0% 1 18.2 18.2 18.2 18.2 7. 68.86.84.70 0.0% 1 23.9 23.9 23.9 23.9 8. ??? 100.0 1 0.0 0.0 0.0 0.0
Taking the 69.61.40.33/28 subnet a bit further, .36 drops at 68.86.84.70 but .37 - .39 make it. .40 drops at 68.86.84.70, but .41 makes it.
Crazy.
Btw, the problem has now been resolved, however I'm still curious as to what scenario could have caused that. -Jim P.
On Thu, Nov 30, 2006 at 12:28:30AM -0500, Jim Popovitch wrote:
On Thu, 2006-11-30 at 00:06 -0500, Jim Popovitch wrote:
Question: What could cause the first trace below to succeed, but the second trace to fail?
[snip]
Taking the 69.61.40.33/28 subnet a bit further, .36 drops at 68.86.84.70 but .37 - .39 make it. .40 drops at 68.86.84.70, but .41 makes it.
Crazy.
Btw, the problem has now been resolved, however I'm still curious as to what scenario could have caused that.
Perhaps CEF-style load balancing over multiple paths, with one of them down and not properly failing over? Per-flow balancing would decide which path to use based on source & target IP.
-Jim P.
-- Jordan.
On 11/29/06, Jim Popovitch <jimpop@yahoo.com> wrote:
On Thu, 2006-11-30 at 00:06 -0500, Jim Popovitch wrote:
Question: What could cause the first trace below to succeed, but the second trace to fail?
$ mtr 69.61.40.35 HOST: blue Loss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 1 4.3 4.3 4.3 4.3 2. 73.62.48.1 0.0% 1 10.6 10.6 10.6 10.6 3. 68.86.108.25 0.0% 1 11.4 11.4 11.4 11.4 4. 68.86.106.54 0.0% 1 9.8 9.8 9.8 9.8 5. 68.86.106.9 0.0% 1 20.5 20.5 20.5 20.5 6. 68.86.90.121 0.0% 1 11.3 11.3 11.3 11.3 7. 68.86.84.70 0.0% 1 27.7 27.7 27.7 27.7 8. 64.213.76.77 0.0% 1 24.5 24.5 24.5 24.5 9. 208.50.254.150 0.0% 1 39.4 39.4 39.4 39.4 10. 208.49.83.237 0.0% 1 46.6 46.6 46.6 46.6 11. 208.49.83.234 0.0% 1 40.7 40.7 40.7 40.7 12. 69.61.40.35 0.0% 1 43.9 43.9 43.9 43.9
$ mtr 69.61.40.34 HOST: blue Loss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 1 1.1 1.1 1.1 1.1 2. 73.62.48.1 0.0% 1 9.9 9.9 9.9 9.9 3. 68.86.108.25 0.0% 1 9.3 9.3 9.3 9.3 4. 68.86.106.54 0.0% 1 9.6 9.6 9.6 9.6 5. 68.86.106.9 0.0% 1 9.0 9.0 9.0 9.0 6. 68.86.90.121 0.0% 1 18.2 18.2 18.2 18.2 7. 68.86.84.70 0.0% 1 23.9 23.9 23.9 23.9 8. ??? 100.0 1 0.0 0.0 0.0 0.0
Taking the 69.61.40.33/28 subnet a bit further, .36 drops at 68.86.84.70 but .37 - .39 make it. .40 drops at 68.86.84.70, but .41 makes it.
Crazy.
Btw, the problem has now been resolved, however I'm still curious as to what scenario could have caused that.
-Jim P.
eBGP multihop peering across a pair of 10 gigE links with static routes pointing to the remote router loopback; one link goes south, but the interface still shows as up/up, and voila, depending upon the hash, your packets may go across the good link, or they may disappear into the black hole of oblivion. This is why multipath is a good thing, and eBGP multihop with static routes is a Bad Thing(tm). Matt
On Thu, Nov 30, 2006 at 12:06:29AM -0500, Jim Popovitch wrote:
$ mtr 69.61.40.34 HOST: blue Loss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 1 1.1 1.1 1.1 1.1 2. 73.62.48.1 0.0% 1 9.9 9.9 9.9 9.9 3. 68.86.108.25 0.0% 1 9.3 9.3 9.3 9.3 4. 68.86.106.54 0.0% 1 9.6 9.6 9.6 9.6 5. 68.86.106.9 0.0% 1 9.0 9.0 9.0 9.0 6. 68.86.90.121 0.0% 1 18.2 18.2 18.2 18.2 7. 68.86.84.70 0.0% 1 23.9 23.9 23.9 23.9 8. ??? 100.0 1 0.0 0.0 0.0 0.0
Taking the 69.61.40.33/28 subnet a bit further, .36 drops at 68.86.84.70 but .37 - .39 make it. .40 drops at 68.86.84.70, but .41 makes it.
You're not the only one who noticed this. http://www.dslreports.com/forum/remark,17368208 -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Comcast broke themselves doing a maintenance.. On Thu, November 30, 2006 01:59, Jeremy Chadwick wrote:
On Thu, Nov 30, 2006 at 12:06:29AM -0500, Jim Popovitch wrote:
$ mtr 69.61.40.34 HOST: blue Loss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 1 1.1 1.1 1.1 1.1 2. 73.62.48.1 0.0% 1 9.9 9.9 9.9 9.9 3. 68.86.108.25 0.0% 1 9.3 9.3 9.3 9.3 4. 68.86.106.54 0.0% 1 9.6 9.6 9.6 9.6 5. 68.86.106.9 0.0% 1 9.0 9.0 9.0 9.0 6. 68.86.90.121 0.0% 1 18.2 18.2 18.2 18.2 7. 68.86.84.70 0.0% 1 23.9 23.9 23.9 23.9 8. ??? 100.0 1 0.0 0.0 0.0 0.0
Taking the 69.61.40.33/28 subnet a bit further, .36 drops at 68.86.84.70 but .37 - .39 make it. .40 drops at 68.86.84.70, but .41 makes it.
You're not the only one who noticed this.
http://www.dslreports.com/forum/remark,17368208
-- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
-- Jamie Dahl "Thousands of tired, nerve-shaken, over-civilized people are beginning to find out that going to the mountains is going home; that wilderness is a necessity; and that mountain parks and reservations are useful not only as fountains of timber and irrigating rivers, but as fountains of life." --John Muir
On Wed, Nov 29, 2006, Jeremy Chadwick wrote:
Taking the 69.61.40.33/28 subnet a bit further, .36 drops at 68.86.84.70 but .37 - .39 make it. .40 drops at 68.86.84.70, but .41 makes it.
You're not the only one who noticed this.
Not that its probably it; but i've seen some crappy etherchannel links with IP path selection exhibit this whacky behaviour. Adrian
participants (6)
-
Adrian Chadd -
Jamie Dahl -
Jeremy Chadwick -
Jim Popovitch -
Jordan Hazen -
Matthew Petach