On Tue, 12 Aug 2008 list-nanog@pwns.ms wrote:
Are dups generated on traffic going over that DS3 from (rather than to) the Ocala side?
The dupes are only generated in the Orlando->Ocala direction.
Does the DS3 cross Sprint's network?
The DS3 enters an Embarq (the telco formerly known as Sprint) central office. AFAIK, the only portion of the circuit handled by Embarq is where it's handed to them in the CO where our gear is colo'd.
What would happen if you pinged the Ocala router such that the TTL was 1 when travelling over the DS3? From your traceroute it seems it travelled two IP hops that did not send ICMP error messages, but it might just be that the ICMP errors from the Ocala router are arriving first.
Based on where the dupes are coming from, I assume pinging across the DS3 with TTL tuned to expire at the Ocala side would result in TTL exceeded messages from both Ocala and the Sprint router where the packets are injected into Sprint's network. It doesn't look as if IOS gives the option to set TTL on ping...so I'd try this from a Linux machine in our data center.
traffic was actually jumping off our network and coming back in via Level3, I could see/block at least some of that using an ACL on our interface to Level3. How do you explain it, when you ping the remote end of a DS3 interface with a single echo request packet and see 5 copies of that echo request arrive at one of your transit provider interfaces?
Just clarifying: 5 duplicates were being generated for every packet that crossed the DS3, not just 1 packet that looped causing 5 duplicates?
Yes. With the ACL on our Level3 transit, I blocked 5 dupes for each echo request sent from the Orlando end of the DS3 to the Ocala end.
9 * * * 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 80.774 ms 81.030 ms 81.821 ms
Was the first visibile IP hop of the dups always that Sprint router?
No. That's one of the wild things about it. Depending on who's network you trace from (we did traces from a bunch of route servers and looking glasses. Some traces would show a pair of private IP hops before the Sprintlink IPs. Some would simply show a different Sprint router as the first off-net hop. If I break it again some night, I'll collect a few different examples.
Level3 is your circuit provider?
Yes. Originally it was a Progress Telecom circuit...but Level3 borged them. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________