My setup is fairly simple, i dont really know where to look... M7i border routers, one with a Fast-E to Mizma and GLBX and one with Sprint Fast-E, they both run IBGP with BI4k (jetCore), out to L2s. I have seen this issue across both the Mzima link and the GLBX link, the only common thing with them is they are on the same router and 4 port PIC card, and they both travel via a L2 MPLS link to Ashburn, VA via the same transport provider. Brendan ----- Original Message ----- From: "Daniel Senie" <dts@senie.com> To: "Brendan Mannella" <bmannella@teraswitch.com> Sent: Saturday, July 19, 2008 10:53:12 PM GMT -05:00 US/Canada Eastern Subject: Re: Savvis Related Routing Issue The fact that changing the target IP address by one changes things makes me think you may have a multi-wire trunking setup between some Ethernet switches, and that trunking is spreading load based on hashing IP addresses. Look close to home, or in your link to your upstream as likely places for this. Might not be your issue, but I've seen this kind of thing before, and found the issue in switch-to-switch trunking or the equivalent in multi-circuit load balancing between routers (e.g. multiple T1's between two routers, where ML-PPP should have been used, and wasn't). At 10:36 PM 7/19/2008, you wrote:

Hello,

I am seeing a really weird routing issue. I have customers who are having trouble connecting to hosts on the internet. The common item seems to be Savvis.

orange:~>traceroute mail.tecumsehherald.com traceroute to mail.tecumsehherald.com (64.14.74.42) 1 204.16.245.97 (204.16.245.97) 0.580 ms 0.800 ms 1.020 ms 2 204.16.241.225 (204.16.241.225) 0.496 ms 0.478 ms 0.699 ms 3 64.209.102.233 (64.209.102.233) 7.177 ms 7.420 ms 7.417 ms 4 te1-3-10g.ar2.DCA3.gblx.net (67.17.108.145) 7.630 ms 7.870 ms 8.105 ms 5 * * * 6 * * *

Hop 5 is a Savvis router.

Now if i go to Global Crossings looking glass and run a trace from their DC router to the same host, it makes it.

1 64.214.14.161 (64.214.14.161) 0.457 ms 0.429 ms 2 te7-1-10G.ar2.DCA3.gblx.net (67.17.109.34) 1.401 ms 1.658 ms 3 savvis-1.ar2.DCA3.gblx.net (64.212.107.26) 1.567 ms 1.339 ms 4 ber1-tenge-2-1.virginiaequinix.savvis.net (204.70.193.6) 1.586 ms 1.511 ms 5 cr1-tengig0-7-2-0.washington.savvis.net (204.70.197.242) 2.367 ms 2.547 ms 6 cr1-pos-0-0-0-0.boston.savvis.net (204.70.193.177) 15.163 ms 15.836 ms 7 hr1-pos-1-0-0.Waltham2bo2.savvis.net (208.172.51.66) 11.702 ms 11.780 ms 8 csr1-ve242.Waltham1bo1.savvis.net (64.14.70.19) 12.202 ms 11.728 ms 9 64.14.67.130 (64.14.67.130) 11.570 ms 11.592 ms 10 ns2.s426.sureserver.com (64.14.74.42) 11.811 ms 11.927 ms

Now to top it all of, if i run a trace to the 64.14.74.41, instead of 64.14.74.42 it works...

agg1.pit#traceroute 64.14.74.41 Type Control-c to abort Tracing the route to IP node 64.14.74.41 from 1 to 30 hops

1 <1 ms <1 ms <1 ms 204.16.241.225 2 7 ms 7 ms 6 ms 64.209.102.233 3 7 ms 6 ms 6 ms te1-3-10g.ar2.DCA3.gblx.net [67.17.108.145] 4 6 ms 6 ms 6 ms savvis-1.ar2.DCA3.gblx.net [64.212.107.26] 5 45 ms 6 ms 7 ms ber1-tenge-2-1.virginiaequinix.savvis.net [204.70.193.6] 6 14 ms 14 ms 15 ms cr1-tengig0-7-2-0.washington.savvis.net [204.70.197.242] 7 21 ms 22 ms 23 ms cr1-pos-0-0-0-0.boston.savvis.net [204.70.193.177] 8 20 ms 19 ms 20 ms hr1-pos-1-0-0.Waltham2bo2.savvis.net [208.172.51.66] 9 19 ms 20 ms 19 ms csr1-ve242.Waltham1bo1.savvis.net [64.14.70.19] 10 20 ms 19 ms 19 ms 64.14.67.130 11 20 ms 20 ms 20 ms s426.sureserver.com [64.14.74.41]

Does anyone have and ideas?

Thanks,

Brendan