Are dups generated on traffic going over that DS3 from (rather than to) the Ocala side? Does the DS3 cross Sprint's network?
Then we noticed the really weird stuff. Pings to anything in Ocala responded with multiple dupes and ttl exceeded messages from a Level3 IP. Traceroutes to certain IPs in Ocala would get as far our Ocala router, then inexplicably hop onto Sprintlink's network, come back to us over our Level3 transit connection, get to Ocala, then hop over to Sprintlink again, repeating that loop as many times as max TTL would permit. Pings from router to router crossing just the DS3 would work, but we'd see 10 duplicate packets for every 1 expected packet. BTW, the cisco CLI hides dupes unless you turn on ip icmp debugging.
What would happen if you pinged the Ocala router such that the TTL was 1 when travelling over the DS3? From your traceroute it seems it travelled two IP hops that did not send ICMP error messages, but it might just be that the ICMP errors from the Ocala router are arriving first.
traffic was actually jumping off our network and coming back in via Level3, I could see/block at least some of that using an ACL on our interface to Level3. How do you explain it, when you ping the remote end of a DS3 interface with a single echo request packet and see 5 copies of that echo request arrive at one of your transit provider interfaces?
Just clarifying: 5 duplicates were being generated for every packet that crossed the DS3, not just 1 packet that looped causing 5 duplicates?
Here's a typical traceroute with the first few hops (from my home internet connection) removed. BTW, hop 9 is a customer router conveniently configured with no ip unreachables. 7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 47.951 ms 56.096 ms 56.154 ms 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 56.199 ms 56.320 ms 56.196 ms 9 * * * 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 80.774 ms 81.030 ms 81.821 ms
Was the first visibile IP hop of the dups always that Sprint router?
If someone from Level3 transport can wrap their head around this, I'd love to know what's really going on...but at least it's no longer an urgent problem for me.
Level3 is your circuit provider?