After all the messages recently about how to fix DNS, I was seriously tempted to title this messsage "And now, for something completely different", but impossible circuit is more descriptive. Before you read further, I need everyone to put on their thinking WAY outside the box hats. I've heard from enough people already that I'm nuts and what I'm seeing can't happen, so it must not be happening...even though we see the results of it happening. I've got this private line DS3. It connects cisco 7206 routers in Orlando (at our data center) and in Ocala (a colo rack in the Embarq CO). According to the DLR, it's a real circuit, various portions of it ride varying sized OC circuits, and then it's handed off to us at each end the usual way (copper/coax) and plugged into PA-2T3 cards. Last Tuesday, at about 2:30PM, "something bad happened." We saw a serious jump in traffic to Ocala, and in particular we noticed one customer's connection (a group of load sharing T1s) was just totally full. We quickly assumed it was a DDoS aimed at that customer, but looking at the traffic, we couldn't pinpoint anything that wasn't expected flows. Then we noticed the really weird stuff. Pings to anything in Ocala responded with multiple dupes and ttl exceeded messages from a Level3 IP. Traceroutes to certain IPs in Ocala would get as far our Ocala router, then inexplicably hop onto Sprintlink's network, come back to us over our Level3 transit connection, get to Ocala, then hop over to Sprintlink again, repeating that loop as many times as max TTL would permit. Pings from router to router crossing just the DS3 would work, but we'd see 10 duplicate packets for every 1 expected packet. BTW, the cisco CLI hides dupes unless you turn on ip icmp debugging. I've seen some sort of similar things (though contained within an AS) with MPLS and routing misconfigurations, but traffic jumping off our network (to a network to which we're not directly connected) was seemingly impossible. We did all sorts of things to troubleshoot it (studied our router configs in rancid, temporarily shut every interface on the Ocala side other than the DS3, changed IOS versions, changed out the hardware, opened a ticket with cisco TAC) but then it occurred to me, that if traffic was actually jumping off our network and coming back in via Level3, I could see/block at least some of that using an ACL on our interface to Level3. How do you explain it, when you ping the remote end of a DS3 interface with a single echo request packet and see 5 copies of that echo request arrive at one of your transit provider interfaces? Here's a typical traceroute with the first few hops (from my home internet connection) removed. BTW, hop 9 is a customer router conveniently configured with no ip unreachables. 7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 47.951 ms 56.096 ms 56.154 ms 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 56.199 ms 56.320 ms 56.196 ms 9 * * * 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 80.774 ms 81.030 ms 81.821 ms 11 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 75.731 ms 75.902 ms 77.128 ms 12 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 46.548 ms 53.200 ms 45.736 ms 13 vlan69.csw1.Washington1.Level3.net (4.68.17.62) 42.918 ms vlan79.csw2.Washington1.Level3.net (4.68.17.126) 55.438 ms vlan69.csw1.Washington1.Level3.net (4.68.17.62) 42.693 ms 14 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 48.935 ms ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 49.317 ms ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) 48.865 ms 15 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 59.642 ms 56.278 ms 56.671 ms 16 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 47.401 ms 62.980 ms 62.640 ms 17 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 40.300 ms 40.101 ms 42.690 ms 18 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 40.959 ms 40.963 ms 41.016 ms 19 unknown.Level3.net (63.209.98.66) 246.744 ms 240.826 ms 239.758 ms 20 andc-br-3-f2-0.atlantic.net (209.208.9.138) 39.725 ms 37.751 ms 42.262 ms 21 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 43.524 ms 45.844 ms 43.392 ms 22 * * * 23 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 63.752 ms 61.648 ms 60.839 ms 24 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 66.923 ms 65.258 ms 70.609 ms 25 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 67.106 ms 93.415 ms 73.932 ms 26 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 88.919 ms 75.306 ms vlan79.csw2.Washington1.Level3.net (4.68.17.126) 75.048 ms 27 ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 69.508 ms 68.401 ms ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 79.128 ms 28 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 64.048 ms 67.764 ms 67.704 ms 29 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 68.372 ms 67.025 ms 68.162 ms 30 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 65.112 ms 65.584 ms 65.525 ms Our circuit provider's support people have basically just maintained that this behavior isn't possible and so there's nothing they can do about it. i.e. that the problem has to be something other than the circuit. I got tired of talking to their brick wall, so I contacted Sprint and was able to confirm with them that the traffic in question really was inexplicably appearing on their network...and not terribly close geographically to the Orlando/Ocala areas. So, I have a circuit that's bleeding duplicate packets onto an unrelated IP network, a circuit provider who's got their head in the sand and keeps telling me "this can't happen, we can't help you", and customers who were getting tired of receiving all their packets in triplicate (or more) saturating their connections and confusing their applications. After a while, I had to give up on finding the problem and focus on just making it stop. After trying a couple of things, the solution I found was to change the encapsulation we use at each end of the DS3. I haven't gotten confirmation of this from Sprint, but I assume they're now seeing massive input errors one the one or more circuits where our packets were/are appearing. The important thing (for me) is that this makes the packets invalid to Sprint's routers and so it keeps them from forwarding the packets to us. Cisco TAC finally got back to us the day after I "fixed" the circuit...but since it was obviously not a problem with our cisco gear, I haven't pursued it with them. The only things I can think of that might be the cause are misconfiguration in a DACS/mux somewhere along the circuit path or perhaps a mishandled lawful intercept. I don't have enough experience with either or enough access to the systems that provide the circuit to do any more than speculate. Has anyone else ever seen anything like this? If someone from Level3 transport can wrap their head around this, I'd love to know what's really going on...but at least it's no longer an urgent problem for me. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
A strange one indeed, especially if you have no connectivity to Sprint there. Since your fix was layer 2 you might be onto something. And you have the time it happened, and as we all know - somebody changed somethin' even if they won't fess up. I'm trying to think how you could cause something like that with a conventional DACS or one of the newer packet friendly types that might be more prone to a layer 2 bug since software is fairly new. Course it would make more sense if it was just crossed Ethernet rather than DS3 frames but who knows. There are plenty of carriers putting them in (including us). Shame it's not the kind of thing you can duplicate without being service affecting. George Jon Lewis wrote:
After all the messages recently about how to fix DNS, I was seriously tempted to title this messsage "And now, for something completely different", but impossible circuit is more descriptive.
Before you read further, I need everyone to put on their thinking WAY outside the box hats. I've heard from enough people already that I'm nuts and what I'm seeing can't happen, so it must not be happening...even though we see the results of it happening.
I've got this private line DS3. It connects cisco 7206 routers in Orlando (at our data center) and in Ocala (a colo rack in the Embarq CO).
According to the DLR, it's a real circuit, various portions of it ride varying sized OC circuits, and then it's handed off to us at each end the usual way (copper/coax) and plugged into PA-2T3 cards.
Last Tuesday, at about 2:30PM, "something bad happened." We saw a serious jump in traffic to Ocala, and in particular we noticed one customer's connection (a group of load sharing T1s) was just totally full. We quickly assumed it was a DDoS aimed at that customer, but looking at the traffic, we couldn't pinpoint anything that wasn't expected flows.
Then we noticed the really weird stuff. Pings to anything in Ocala responded with multiple dupes and ttl exceeded messages from a Level3 IP. Traceroutes to certain IPs in Ocala would get as far our Ocala router, then inexplicably hop onto Sprintlink's network, come back to us over our Level3 transit connection, get to Ocala, then hop over to Sprintlink again, repeating that loop as many times as max TTL would permit. Pings from router to router crossing just the DS3 would work, but we'd see 10 duplicate packets for every 1 expected packet. BTW, the cisco CLI hides dupes unless you turn on ip icmp debugging.
I've seen some sort of similar things (though contained within an AS) with MPLS and routing misconfigurations, but traffic jumping off our network (to a network to which we're not directly connected) was seemingly impossible. We did all sorts of things to troubleshoot it (studied our router configs in rancid, temporarily shut every interface on the Ocala side other than the DS3, changed IOS versions, changed out the hardware, opened a ticket with cisco TAC) but then it occurred to me, that if traffic was actually jumping off our network and coming back in via Level3, I could see/block at least some of that using an ACL on our interface to Level3. How do you explain it, when you ping the remote end of a DS3 interface with a single echo request packet and see 5 copies of that echo request arrive at one of your transit provider interfaces?
Here's a typical traceroute with the first few hops (from my home internet connection) removed. BTW, hop 9 is a customer router conveniently configured with no ip unreachables.
7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 47.951 ms 56.096 ms 56.154 ms 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 56.199 ms 56.320 ms 56.196 ms 9 * * * 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 80.774 ms 81.030 ms 81.821 ms 11 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 75.731 ms 75.902 ms 77.128 ms 12 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 46.548 ms 53.200 ms 45.736 ms 13 vlan69.csw1.Washington1.Level3.net (4.68.17.62) 42.918 ms vlan79.csw2.Washington1.Level3.net (4.68.17.126) 55.438 ms vlan69.csw1.Washington1.Level3.net (4.68.17.62) 42.693 ms 14 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 48.935 ms ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 49.317 ms ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) 48.865 ms 15 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 59.642 ms 56.278 ms 56.671 ms 16 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 47.401 ms 62.980 ms 62.640 ms 17 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 40.300 ms 40.101 ms 42.690 ms 18 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 40.959 ms 40.963 ms 41.016 ms 19 unknown.Level3.net (63.209.98.66) 246.744 ms 240.826 ms 239.758 ms 20 andc-br-3-f2-0.atlantic.net (209.208.9.138) 39.725 ms 37.751 ms 42.262 ms 21 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 43.524 ms 45.844 ms 43.392 ms 22 * * * 23 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 63.752 ms 61.648 ms 60.839 ms 24 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 66.923 ms 65.258 ms 70.609 ms 25 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 67.106 ms 93.415 ms 73.932 ms 26 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 88.919 ms 75.306 ms vlan79.csw2.Washington1.Level3.net (4.68.17.126) 75.048 ms 27 ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 69.508 ms 68.401 ms ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 79.128 ms 28 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 64.048 ms 67.764 ms 67.704 ms 29 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 68.372 ms 67.025 ms 68.162 ms 30 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 65.112 ms 65.584 ms 65.525 ms
Our circuit provider's support people have basically just maintained that this behavior isn't possible and so there's nothing they can do about it. i.e. that the problem has to be something other than the circuit.
I got tired of talking to their brick wall, so I contacted Sprint and was able to confirm with them that the traffic in question really was inexplicably appearing on their network...and not terribly close geographically to the Orlando/Ocala areas.
So, I have a circuit that's bleeding duplicate packets onto an unrelated IP network, a circuit provider who's got their head in the sand and keeps telling me "this can't happen, we can't help you", and customers who were getting tired of receiving all their packets in triplicate (or more) saturating their connections and confusing their applications. After a while, I had to give up on finding the problem and focus on just making it stop. After trying a couple of things, the solution I found was to change the encapsulation we use at each end of the DS3. I haven't gotten confirmation of this from Sprint, but I assume they're now seeing massive input errors one the one or more circuits where our packets were/are appearing. The important thing (for me) is that this makes the packets invalid to Sprint's routers and so it keeps them from forwarding the packets to us. Cisco TAC finally got back to us the day after I "fixed" the circuit...but since it was obviously not a problem with our cisco gear, I haven't pursued it with them.
The only things I can think of that might be the cause are misconfiguration in a DACS/mux somewhere along the circuit path or perhaps a mishandled lawful intercept. I don't have enough experience with either or enough access to the systems that provide the circuit to do any more than speculate. Has anyone else ever seen anything like this?
If someone from Level3 transport can wrap their head around this, I'd love to know what's really going on...but at least it's no longer an urgent problem for me.
---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
George Carey wrote:
Since your fix was layer 2 you might be onto something. And you have the time it happened, and as we all know - somebody changed somethin' even if they won't fess up.
I have not pencil-and-papered this to see if there is anything to it, but I was wondering what would happened if you put a layer-two bridge into a back-bone fabric and turned off "learning" so every packet is flooded to every port. -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs
Laurence F. Sheldon, Jr. wrote:
George Carey wrote:
I have not pencil-and-papered this to see if there is anything to it, but I was wondering what would happened if you put a layer-two bridge into a back-bone fabric and turned off "learning" so every packet is flooded to every port.
Though not the same circumstances on having the same symptoms as the OP's problem, I saw this happen once at a University I used to work for. A system's administrator insisted on having a hub between the SP's router and our core campus switch so he could sniff traffic. Since the hub was there and I couldn't eliminate it I went ahead and used it myself for my own traffic capture point in the network with an OS X box running EtherPeek. I did an OS update on the box one morning and went to a meeting. During the meeting it was reported that the network was down. I started looking into the problem at that point. All Internet traffic was dead except SSH connections. So I started sniffing on my NOC server for that server's traffic. All my outbound TCP connections from the NOC were getting a RST from one L2 host and a SYN-ACK from another. The MAC address sending the RST looked familiar but I couldn't identify it. After searching through the network for the MAC I found it on the interface facing our border router and that damn hub. The MAC was my OS X sniffing box. The other MAC was the backside of the provider's router. The OS X update I applied was the one that installed a host-based firewall. The update automatically turned on the FW and permitted all local servers that were configured to run, in my case SSH, with everything else being denied. The FW on the OS X box normally wouldn't see packets not destined for it until you put a nic in promisc mode such as what happens when you run EtherPeek. The OS X box's FW was getting hits from traffic denied by it's ACL and was sending TCP RSTs faster than hosts on the 'Net could respond. It did this for everything except SSH which it permitted (but higher up the IP stack it ignored because the IP packet was address to the local box). This isn't in any way related to the problem at hand but it does demonstrate that weird things happen when devices in unusual places flood out all ports. Justin
On Mon, Aug 11, 2008 at 03:17:18PM -0500, Justin Shore wrote:
The OS X update I applied was the one that installed a host-based firewall. The update automatically turned on the FW and permitted all local servers that were configured to run, in my case SSH, with everything else being denied. The FW on the OS X box normally wouldn't see packets not destined for it until you put a nic in promisc mode such as what happens when you run EtherPeek. The OS X box's FW was getting hits from traffic denied by it's ACL and was sending TCP RSTs faster than hosts on the 'Net could respond. It did this for everything except SSH which it permitted (but higher up the IP stack it ignored because the IP packet was address to the local box).
This isn't in any way related to the problem at hand but it does demonstrate that weird things happen when devices in unusual places flood out all ports.
And this explains why in Bellovin's Wily Hacker book, there's an anecdote about a sniffer machine on which they had to *physically cut the transmit wire* because they could *not* get the machine to not... do something. ARP queries? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Those who cast the vote decide nothing. Those who count the vote decide everything. -- (Josef Stalin)
Are dups generated on traffic going over that DS3 from (rather than to) the Ocala side? Does the DS3 cross Sprint's network?
Then we noticed the really weird stuff. Pings to anything in Ocala responded with multiple dupes and ttl exceeded messages from a Level3 IP. Traceroutes to certain IPs in Ocala would get as far our Ocala router, then inexplicably hop onto Sprintlink's network, come back to us over our Level3 transit connection, get to Ocala, then hop over to Sprintlink again, repeating that loop as many times as max TTL would permit. Pings from router to router crossing just the DS3 would work, but we'd see 10 duplicate packets for every 1 expected packet. BTW, the cisco CLI hides dupes unless you turn on ip icmp debugging.
What would happen if you pinged the Ocala router such that the TTL was 1 when travelling over the DS3? From your traceroute it seems it travelled two IP hops that did not send ICMP error messages, but it might just be that the ICMP errors from the Ocala router are arriving first.
traffic was actually jumping off our network and coming back in via Level3, I could see/block at least some of that using an ACL on our interface to Level3. How do you explain it, when you ping the remote end of a DS3 interface with a single echo request packet and see 5 copies of that echo request arrive at one of your transit provider interfaces?
Just clarifying: 5 duplicates were being generated for every packet that crossed the DS3, not just 1 packet that looped causing 5 duplicates?
Here's a typical traceroute with the first few hops (from my home internet connection) removed. BTW, hop 9 is a customer router conveniently configured with no ip unreachables. 7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 47.951 ms 56.096 ms 56.154 ms 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 56.199 ms 56.320 ms 56.196 ms 9 * * * 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 80.774 ms 81.030 ms 81.821 ms
Was the first visibile IP hop of the dups always that Sprint router?
If someone from Level3 transport can wrap their head around this, I'd love to know what's really going on...but at least it's no longer an urgent problem for me.
Level3 is your circuit provider?
On Tue, 12 Aug 2008 list-nanog@pwns.ms wrote:
Are dups generated on traffic going over that DS3 from (rather than to) the Ocala side?
The dupes are only generated in the Orlando->Ocala direction.
Does the DS3 cross Sprint's network?
The DS3 enters an Embarq (the telco formerly known as Sprint) central office. AFAIK, the only portion of the circuit handled by Embarq is where it's handed to them in the CO where our gear is colo'd.
What would happen if you pinged the Ocala router such that the TTL was 1 when travelling over the DS3? From your traceroute it seems it travelled two IP hops that did not send ICMP error messages, but it might just be that the ICMP errors from the Ocala router are arriving first.
Based on where the dupes are coming from, I assume pinging across the DS3 with TTL tuned to expire at the Ocala side would result in TTL exceeded messages from both Ocala and the Sprint router where the packets are injected into Sprint's network. It doesn't look as if IOS gives the option to set TTL on ping...so I'd try this from a Linux machine in our data center.
traffic was actually jumping off our network and coming back in via Level3, I could see/block at least some of that using an ACL on our interface to Level3. How do you explain it, when you ping the remote end of a DS3 interface with a single echo request packet and see 5 copies of that echo request arrive at one of your transit provider interfaces?
Just clarifying: 5 duplicates were being generated for every packet that crossed the DS3, not just 1 packet that looped causing 5 duplicates?
Yes. With the ACL on our Level3 transit, I blocked 5 dupes for each echo request sent from the Orlando end of the DS3 to the Ocala end.
9 * * * 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 80.774 ms 81.030 ms 81.821 ms
Was the first visibile IP hop of the dups always that Sprint router?
No. That's one of the wild things about it. Depending on who's network you trace from (we did traces from a bunch of route servers and looking glasses. Some traces would show a pair of private IP hops before the Sprintlink IPs. Some would simply show a different Sprint router as the first off-net hop. If I break it again some night, I'll collect a few different examples.
Level3 is your circuit provider?
Yes. Originally it was a Progress Telecom circuit...but Level3 borged them. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Tue, 12 Aug 2008, Jon Lewis wrote:
What would happen if you pinged the Ocala router such that the TTL was 1 when travelling over the DS3? From your traceroute it seems it travelled two IP hops that did not send ICMP error messages, but it might just be that the ICMP errors from the Ocala router are arriving first.
Based on where the dupes are coming from, I assume pinging across the DS3 with TTL tuned to expire at the Ocala side would result in TTL exceeded messages from both Ocala and the Sprint router where the packets are injected into Sprint's network. It doesn't look as if IOS gives the option to set TTL on ping...so I'd try this from a Linux machine in our data center.
I just went ahead and "re-broke" the circuit for a bit by turning it back to hdlc to see if the issue is still there and to run some additional tests. Someone is still cross connecting our Orlando->Ocala traffic over to Sprint. I did your suggested ping with short TTL and the result was close to what I expected. $ traceroute ocalflxa-br-1 traceroute to ocalflxa-br-1.atlantic.net (209.208.6.229), 30 hops max, 38 byte packets 1 209.208.25.165 (209.208.25.165) 0.539 ms 0.426 ms 0.388 ms 2 69.28.72.162 (69.28.72.162) 0.246 ms 0.351 ms 0.223 ms 3 andc-br-3-f2-0 (209.208.9.138) 0.559 ms 0.435 ms 0.471 ms 4 ocalflxa-br-1-s1-0 (209.208.112.98) 2.735 ms * 2.656 ms So, I need a TTL of 4 to get there from this machine. $ ping -t4 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.68 ms 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.72 ms 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252 time=2.88 ms Decrease ttl by one, and I get the expected ttl exceeded from the Orlando side of the circuit. $ ping -t 3 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
From andc-br-3-f2-0.atlantic.net (209.208.9.138) icmp_seq=0 Time to live exceeded
Now, here's a mild surprise. You'll notice that in the above -t4 trace, I didn't hear back from Sprint. $ ping -t 5 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.89 ms 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=3.10 ms 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252 time=2.97 ms hmm...still no ttl exceeded from Sprint? $ ping -t 6 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.95 ms
From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=0 Time to live exceeded 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.78 ms From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=1 Time to live exceeded
$ ping -t 7 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.88 ms
From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=0 Time to live exceeded 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.84 ms From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=1 Time to live exceeded
From the traces I've seen, it seems if the first Sprint hop is sl-bb20-dc,
Is it just coincidence that there are 2 private IP hops in some traceroutes between us and Sprint? i.e. Look at this trace from cogent: Tracing the route to 209.208.33.1 1 fa0-8.na01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 0 msec 4 msec 4 msec 2 gi3-9.3507.core01.dca01.atlas.cogentco.com (66.28.67.225) 160 msec 4 msec 8 msec 3 te3-1.ccr02.dca01.atlas.cogentco.com (154.54.3.158) 0 msec 0 msec 4 msec 4 vl3493.mpd01.dca02.atlas.cogentco.com (154.54.7.230) 28 msec 4 msec te4-1.mpd01.dca02.atlas.cogentco.com (154.54.2.182) 52 msec 5 vl3494.mpd01.iad01.atlas.cogentco.com (154.54.5.42) 4 msec 4 msec vl3497.mpd01.iad01.atlas.cogentco.com (154.54.5.66) 4 msec 6 timewarner.iad01.atlas.cogentco.com (154.54.13.250) 4 msec peer-01-ge-3-1-2-13.asbn.twtelecom.net (66.192.252.217) 4 msec 12 msec 7 66-194-200-202.static.twtelecom.net (66.194.200.202) 28 msec 28 msec 32 msec 8 66-194-200-202.static.twtelecom.net (66.194.200.202) 32 msec 32 msec 28 msec 9 andc-br-3-f2-0.atlantic.net (209.208.9.138) 32 msec 32 msec 32 msec 10 172.22.122.1 32 msec 32 msec 32 msec 11 10.247.28.205 32 msec 32 msec 32 msec 12 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 32 msec 32 msec 28 msec 13 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 28 msec 32 msec 32 msec 14 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 32 msec 32 msec 28 msec 15 vlan79.csw2.Washington1.Level3.net (4.68.17.126) 28 msec vlan89.csw3.Washington1.Level3.net (4.68.17.190) 32 msec vlan79.csw2.Washington1.Level3.net (4.68.17.126) 40 msec 16 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 28 msec ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 28 msec ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 32 msec 17 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 48 msec 48 msec 56 msec 18 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 44 msec 48 msec ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52 msec 19 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 56 msec 104 msec 56 msec 20 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 52 msec 52 msec 56 msec 21 unknown.Level3.net (63.209.98.66) 52 msec 52 msec 56 msec 22 andc-br-3-f2-0.atlantic.net (209.208.9.138) 52 msec 52 msec 56 msec 23 172.22.122.1 52 msec 56 msec 52 msec 24 10.247.28.205 52 msec 52 msec 56 msec 25 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 52 msec 56 msec 52 msec 26 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 56 msec 56 msec 56 msec 27 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 52 msec 52 msec 52 msec 28 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 52 msec vlan69.csw1.Washington1.Level3.net (4.68.17.62) 56 msec vlan89.csw3.Washington1.Level3.net (4.68.17.190) 56 msec 29 ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 64 msec ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 52 msec 56 msec 30 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 76 msec 72 msec 72 msec I've seen the 172.22.122.1 & 10.247.28.205 hops before. They occasionally show up in traces when the traffic is jumping over to Sprint. Sometimes they don't show up though. i.e. Tracing from my house: traceroute to 209.208.33.1 (209.208.33.1), 30 hops max, 40 byte packets 1 172.31.0.1 (172.31.0.1) 0.336 ms 0.272 ms 0.268 ms 2 10.210.160.1 (10.210.160.1) 10.109 ms 11.719 ms 14.265 ms 3 gig7-0-4-101.orldflaabv-rtr1.cfl.rr.com (24.95.232.100) 15.302 ms 15.324 ms 16.687 ms 4 198.228.95.24.cfl.res.rr.com (24.95.228.198) 16.688 ms 18.812 ms 18.816 ms 5 te-3-3.car1.Orlando1.Level3.net (4.79.116.145) 20.084 ms 19.946 ms te-3-1.car1.Orlando1.Level3.net (4.79.116.137) 21.328 ms 6 unknown.Level3.net (63.209.98.66) 19.900 ms 14.714 ms 14.689 ms 7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 104.058 ms 11.932 ms 13.584 ms 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 15.872 ms 15.886 ms 17.238 ms 9 * * * 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 41.277 ms 41.964 ms 41.955 ms 11 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 43.360 ms 44.578 ms 35.635 ms 12 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 37.035 ms 37.062 ms 33.185 ms 13 vlan89.csw3.Washington1.Level3.net (4.68.17.190) 44.060 ms 44.057 ms vlan99.csw4.Washington1.Level3.net (4.68.17.254) 39.603 ms 14 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 38.123 ms ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) 39.546 ms ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 38.115 ms 15 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 46.284 ms 46.275 ms 46.274 ms 16 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52.523 ms ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 53.338 ms ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 53.299 ms 17 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 34.964 ms 39.582 ms 38.088 ms 18 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 36.701 ms 38.144 ms 36.949 ms 19 unknown.Level3.net (63.209.98.66) 36.902 ms 37.750 ms 37.717 ms 20 andc-br-3-f2-0.atlantic.net (209.208.9.138) 37.729 ms 35.812 ms 35.048 ms 21 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 37.485 ms 37.601 ms 36.495 ms 22 * * * 23 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 56.459 ms 56.449 ms 57.709 ms 24 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 57.694 ms 57.692 ms 60.243 ms 25 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 103.257 ms 100.829 ms 82.571 ms 26 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 70.401 ms vlan89.csw3.Washington1.Level3.net (4.68.17.190) 69.262 ms vlan99.csw4.Washington1.Level3.net (4.68.17.254) 82.700 ms 27 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 74.132 ms ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 74.135 ms ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 75.540 ms 28 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 58.656 ms 60.838 ms 54.346 ms 29 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 59.323 ms ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 59.336 ms ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 63.323 ms 30 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 127.652 ms 57.884 ms 57.851 ms the private IP hops don't show up. If the first Sprint hop is sl-crs2-dc, then the private IP hops are there. I wonder if anyone from Sprint can shed some light on that? Unfortunately, the Sprint engineer I intitially made contact with who was helpful and seemed curious about the issue seems to have vanished and isn't returning my calls or emails. Anyone else from Sprintlink care to play? ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
From the traces I've seen, it seems if the first Sprint hop is sl-bb20-dc, the private IP hops don't show up. If the first Sprint hop is sl-crs2-dc, then the private IP hops are there. I wonder if anyone from Sprint can shed some light on that?
That's an interesting correlation, but maybe they don't appear because a router in the path is dropping packets sourced from private IPs?
Is this only happening in one direction? One possibility is that the carrier has a different circuit that is provisioned up, HDLC, with no physical connection. A short-circuit in a DACS or MUX is bridging the transmit interface towards your destination with a transmit interface on the unused but active circuit. This would cause your traffic in that direction to fork both on the desired path and some rogue path that eventually gets routed to your destination. The ethernet equivalent would be a SPAN monitor port plumbed to a transmit-only interface on a different network. Definitely a strange one. If I'm correct, when the other circuit starts to get customer traffic things will probably break completely for either the new customer seeing your PPP traffic or for both of you. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
Jon, I think we can safely conclude from the information provided that you're looking at some sort of a misconfigured traffic mirroring or [un]lawful intercept. Sadly, as neither Sprint nor your loop provider will fess up, I don't think you're going to get much further on here. Probably best to order a new loop and cancel the existing one. Drive Slow, Paul - Original message - I just went ahead and "re-broke" the circuit ... On 8/17/08, Jon Lewis <jlewis@lewis.org> wrote:
On Tue, 12 Aug 2008, Jon Lewis wrote:
What would happen if you pinged the Ocala router such that the TTL was 1 when travelling over the DS3? From your traceroute it seems it travelled two IP hops that did not send ICMP error messages, but it might just be that the ICMP errors from the Ocala router are arriving first.
Based on where the dupes are coming from, I assume pinging across the DS3 with TTL tuned to expire at the Ocala side would result in TTL exceeded messages from both Ocala and the Sprint router where the packets are injected into Sprint's network. It doesn't look as if IOS gives the option to set TTL on ping...so I'd try this from a Linux machine in our data center.
I just went ahead and "re-broke" the circuit for a bit by turning it back to hdlc to see if the issue is still there and to run some additional tests. Someone is still cross connecting our Orlando->Ocala traffic over to Sprint.
I did your suggested ping with short TTL and the result was close to what I expected.
$ traceroute ocalflxa-br-1 traceroute to ocalflxa-br-1.atlantic.net (209.208.6.229), 30 hops max, 38 byte packets 1 209.208.25.165 (209.208.25.165) 0.539 ms 0.426 ms 0.388 ms 2 69.28.72.162 (69.28.72.162) 0.246 ms 0.351 ms 0.223 ms 3 andc-br-3-f2-0 (209.208.9.138) 0.559 ms 0.435 ms 0.471 ms 4 ocalflxa-br-1-s1-0 (209.208.112.98) 2.735 ms * 2.656 ms
So, I need a TTL of 4 to get there from this machine.
$ ping -t4 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.68 ms 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.72 ms 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252 time=2.88 ms
Decrease ttl by one, and I get the expected ttl exceeded from the Orlando side of the circuit.
$ ping -t 3 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
From andc-br-3-f2-0.atlantic.net (209.208.9.138) icmp_seq=0 Time to live exceeded
Now, here's a mild surprise. You'll notice that in the above -t4 trace, I didn't hear back from Sprint.
$ ping -t 5 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.89 ms 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=3.10 ms 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252 time=2.97 ms hmm...still no ttl exceeded from Sprint?
From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=0 Time to
From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=1 Time to
$ ping -t 6 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.95 ms live exceeded 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.78 ms live exceeded
From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=0 Time to
From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=1 Time to
$ ping -t 7 ocalflxa-br-1 PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.88 ms live exceeded 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.84 ms live exceeded
Is it just coincidence that there are 2 private IP hops in some traceroutes between us and Sprint? i.e. Look at this trace from cogent:
Tracing the route to 209.208.33.1
1 fa0-8.na01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 0 msec 4 msec 4 msec 2 gi3-9.3507.core01.dca01.atlas.cogentco.com (66.28.67.225) 160 msec 4 msec 8 msec 3 te3-1.ccr02.dca01.atlas.cogentco.com (154.54.3.158) 0 msec 0 msec 4 msec 4 vl3493.mpd01.dca02.atlas.cogentco.com (154.54.7.230) 28 msec 4 msec te4-1.mpd01.dca02.atlas.cogentco.com (154.54.2.182) 52 msec 5 vl3494.mpd01.iad01.atlas.cogentco.com (154.54.5.42) 4 msec 4 msec vl3497.mpd01.iad01.atlas.cogentco.com (154.54.5.66) 4 msec 6 timewarner.iad01.atlas.cogentco.com (154.54.13.250) 4 msec peer-01-ge-3-1-2-13.asbn.twtelecom.net (66.192.252.217) 4 msec 12 msec 7 66-194-200-202.static.twtelecom.net (66.194.200.202) 28 msec 28 msec 32 msec 8 66-194-200-202.static.twtelecom.net (66.194.200.202) 32 msec 32 msec 28 msec 9 andc-br-3-f2-0.atlantic.net (209.208.9.138) 32 msec 32 msec 32 msec 10 172.22.122.1 32 msec 32 msec 32 msec 11 10.247.28.205 32 msec 32 msec 32 msec 12 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 32 msec 32 msec 28 msec 13 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 28 msec 32 msec 32 msec 14 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 32 msec 32 msec 28 msec 15 vlan79.csw2.Washington1.Level3.net (4.68.17.126) 28 msec vlan89.csw3.Washington1.Level3.net (4.68.17.190) 32 msec vlan79.csw2.Washington1.Level3.net (4.68.17.126) 40 msec 16 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 28 msec ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 28 msec ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 32 msec 17 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 48 msec 48 msec 56 msec 18 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 44 msec 48 msec ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52 msec 19 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 56 msec 104 msec 56 msec 20 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 52 msec 52 msec 56 msec 21 unknown.Level3.net (63.209.98.66) 52 msec 52 msec 56 msec 22 andc-br-3-f2-0.atlantic.net (209.208.9.138) 52 msec 52 msec 56 msec 23 172.22.122.1 52 msec 56 msec 52 msec 24 10.247.28.205 52 msec 52 msec 56 msec 25 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 52 msec 56 msec 52 msec 26 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 56 msec 56 msec 56 msec 27 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 52 msec 52 msec 52 msec 28 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 52 msec vlan69.csw1.Washington1.Level3.net (4.68.17.62) 56 msec vlan89.csw3.Washington1.Level3.net (4.68.17.190) 56 msec 29 ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 64 msec ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 52 msec 56 msec 30 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 76 msec 72 msec 72 msec
I've seen the 172.22.122.1 & 10.247.28.205 hops before. They occasionally show up in traces when the traffic is jumping over to Sprint. Sometimes they don't show up though. i.e. Tracing from my house:
traceroute to 209.208.33.1 (209.208.33.1), 30 hops max, 40 byte packets 1 172.31.0.1 (172.31.0.1) 0.336 ms 0.272 ms 0.268 ms 2 10.210.160.1 (10.210.160.1) 10.109 ms 11.719 ms 14.265 ms 3 gig7-0-4-101.orldflaabv-rtr1.cfl.rr.com (24.95.232.100) 15.302 ms 15.324 ms 16.687 ms 4 198.228.95.24.cfl.res.rr.com (24.95.228.198) 16.688 ms 18.812 ms 18.816 ms 5 te-3-3.car1.Orlando1.Level3.net (4.79.116.145) 20.084 ms 19.946 ms te-3-1.car1.Orlando1.Level3.net (4.79.116.137) 21.328 ms 6 unknown.Level3.net (63.209.98.66) 19.900 ms 14.714 ms 14.689 ms 7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 104.058 ms 11.932 ms 13.584 ms 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 15.872 ms 15.886 ms 17.238 ms 9 * * * 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 41.277 ms 41.964 ms 41.955 ms 11 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 43.360 ms 44.578 ms 35.635 ms 12 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 37.035 ms 37.062 ms 33.185 ms 13 vlan89.csw3.Washington1.Level3.net (4.68.17.190) 44.060 ms 44.057 ms vlan99.csw4.Washington1.Level3.net (4.68.17.254) 39.603 ms 14 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 38.123 ms ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) 39.546 ms ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 38.115 ms 15 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 46.284 ms 46.275 ms 46.274 ms 16 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52.523 ms ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 53.338 ms ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 53.299 ms 17 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 34.964 ms 39.582 ms 38.088 ms 18 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 36.701 ms 38.144 ms 36.949 ms 19 unknown.Level3.net (63.209.98.66) 36.902 ms 37.750 ms 37.717 ms 20 andc-br-3-f2-0.atlantic.net (209.208.9.138) 37.729 ms 35.812 ms 35.048 ms 21 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 37.485 ms 37.601 ms 36.495 ms 22 * * * 23 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 56.459 ms 56.449 ms 57.709 ms 24 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 57.694 ms 57.692 ms 60.243 ms 25 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 103.257 ms 100.829 ms 82.571 ms 26 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 70.401 ms vlan89.csw3.Washington1.Level3.net (4.68.17.190) 69.262 ms vlan99.csw4.Washington1.Level3.net (4.68.17.254) 82.700 ms 27 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 74.132 ms ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 74.135 ms ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 75.540 ms 28 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 58.656 ms 60.838 ms 54.346 ms 29 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 59.323 ms ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 59.336 ms ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 63.323 ms 30 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 127.652 ms 57.884 ms 57.851 ms
From the traces I've seen, it seems if the first Sprint hop is sl-bb20-dc, the private IP hops don't show up. If the first Sprint hop is sl-crs2-dc, then the private IP hops are there. I wonder if anyone from Sprint can shed some light on that?
Unfortunately, the Sprint engineer I intitially made contact with who was helpful and seemed curious about the issue seems to have vanished and isn't returning my calls or emails. Anyone else from Sprintlink care to play?
---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
-- Sent from Gmail for mobile | mobile.google.com
The only things I can think of that might be the cause are misconfiguration in a DACS/mux somewhere along the circuit path or perhaps a mishandled lawful intercept. I don't have enough experience with either or enough access to the systems that provide the circuit to do any more than speculate. Has anyone else ever seen anything like this?
I'm not sure how a DACS/mux misconfiguration would do this. There would have to be some intelligent device grabbing those IP packets and forwarding them on to another IP router, which seems really likely. Have you noticed any unusual shifts in pattern on the internal network of your Ocala office? I wonder if somebody decided it would be clever to start up a VPN tunnel on a host inside your network, and also set the host in promiscuous mode, forwarding every packet it gets back out that tunnel. The only argument I can come up against this tunnel/host, is that when you changed line encapsulation modes, it went away. However, could have just dropped the tunnel or the host that was misbehaving decided it was a good time to stop.
This is just a WAG but what the hell. Jon Lewis wrote:
I've got this private line DS3. It connects cisco 7206 routers in Orlando (at our data center) and in Ocala (a colo rack in the Embarq CO).
According to the DLR, it's a real circuit, various portions of it ride varying sized OC circuits, and then it's handed off to us at each end the usual way (copper/coax) and plugged into PA-2T3 cards.
Are you sure that they are not crossing some channels in the middle and accidentally handing them to a different customer? You mention above that various portions of the DS3 ride different transport circuits in the middle. That always creates the potential for someone to not put it back together correctly on either end. I've seen DLCs get crossed before. I could easily see a transport provider crossing portions of a circuit, especially if they break it into pieces in the middle and have to put it back together on the ends. I think it makes sense too. Somebody's getting traffic off a T1 that isn't destined for them. Their router sees it, says WTF and sends a ICMP dest unreachable via their default route through Sprint. Same thing goes for a traceroute; it simply follows its default route to reply to your packets with the expiring TTL. Taking a path through a different provider would be expected since it doesn't have a connected route to the source of the traceroute (since it's not the far end of your T1 that you're expecting). The site getting your crossed T1 could be using the T1 as a PtP to a branch office and has Internet through a different circuit that hasn't been hosed. I would be curious to hear if Sprint is having any problems with a circuit connected to sl-bb20-dc-6-0-0.sprintlink.net, what the router is and if any directly connected customers are having T1 problems. If nothing else Sprint should be able to track down the source of the traceroute return packets and contact the customer. The T1 could be part of a bundle at their site and they may not even realize that the bundle dropped a path.
Last Tuesday, at about 2:30PM, "something bad happened." We saw a serious jump in traffic to Ocala, and in particular we noticed one customer's connection (a group of load sharing T1s) was just totally full. We quickly assumed it was a DDoS aimed at that customer, but looking at the traffic, we couldn't pinpoint anything that wasn't expected flows.
Are you sure that the traffic being received by each of the T1s is their's? Do you have any way to getting flows or packets off of individual T1s and not the bundle as a whole? Tracing through you to your upstream...
7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 47.951 ms 56.096 ms 56.154 ms 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 56.199 ms 56.320 ms 56.196 ms 9 * * *
Circuit gets crossed onto the wrong customer. Wrong site received a packet with an expiring TTL and goes to send a reply. Destination IP isn't on a connected route so the site sends the reply via it's default route on Sprint.
10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 80.774 ms 81.030 ms 81.821 ms 11 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 75.731 ms 75.902 ms 77.128 ms
Reply traverses Sprint to L3 and on to you.
12 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 46.548 ms 53.200 ms 45.736 ms 13 vlan69.csw1.Washington1.Level3.net (4.68.17.62) 42.918 ms vlan79.csw2.Washington1.Level3.net (4.68.17.126) 55.438 ms vlan69.csw1.Washington1.Level3.net (4.68.17.62) 42.693 ms 14 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 48.935 ms ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 49.317 ms ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) 48.865 ms 15 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 59.642 ms 56.278 ms 56.671 ms 16 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 47.401 ms 62.980 ms 62.640 ms 17 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 40.300 ms 40.101 ms 42.690 ms 18 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 40.959 ms 40.963 ms 41.016 ms 19 unknown.Level3.net (63.209.98.66) 246.744 ms 240.826 ms 239.758 ms 20 andc-br-3-f2-0.atlantic.net (209.208.9.138) 39.725 ms 37.751 ms 42.262 ms 21 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 43.524 ms 45.844 ms 43.392 ms 22 * * * 23 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 63.752 ms 61.648 ms 60.839 ms 24 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 66.923 ms 65.258 ms 70.609 ms 25 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 67.106 ms 93.415 ms 73.932 ms 26 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 88.919 ms 75.306 ms vlan79.csw2.Washington1.Level3.net (4.68.17.126) 75.048 ms 27 ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 69.508 ms 68.401 ms ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 79.128 ms 28 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 64.048 ms 67.764 ms 67.704 ms 29 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 68.372 ms 67.025 ms 68.162 ms 30 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 65.112 ms 65.584 ms 65.525 ms
I can't explain the continuous loop or the dupes. I'm not sure if my theory fits those symptoms or not.
Our circuit provider's support people have basically just maintained that this behavior isn't possible and so there's nothing they can do about it. i.e. that the problem has to be something other than the circuit.
Can you have them put the circuit into maintenance and have them test it end to end? They can't deny it when their TDR says that there's a problem.
I got tired of talking to their brick wall, so I contacted Sprint and was able to confirm with them that the traffic in question really was inexplicably appearing on their network...and not terribly close geographically to the Orlando/Ocala areas.
Which supports with my theory of a crossed circuit. Crossing a DS1 onto the wrong DS3 or OCx could easily make it pop up anywhere. Somewhere is another site that's having T1 problems.
So, I have a circuit that's bleeding duplicate packets onto an unrelated IP network, a circuit provider who's got their head in the sand and keeps telling me "this can't happen, we can't help you", and customers who were getting tired of receiving all their packets in triplicate (or more) saturating their connections and confusing their applications. After a while, I had to give up on finding the problem and focus on just making it stop. After trying a couple of things, the solution I found was to change the encapsulation we use at each end of the DS3. I haven't gotten confirmation of this from Sprint, but I assume they're now seeing massive input errors one the one or more circuits where our packets were/are appearing. The important thing (for me) is that this makes the packets invalid to Sprint's routers and so it keeps them from forwarding the packets to us. Cisco TAC finally got back to us the day after I "fixed" the circuit...but since it was obviously not a problem with our cisco gear, I haven't pursued it with them.
Right. By changing the encap you've basically killed the circuit. With that T1 effectively down on your end you won't be sending any packets down the problem path and aren't able to see that problem anymore with your traceroutes. However your customer with the bundle of T1s is down a circuit. It makes sense in my mind that it's simply a crossed circuit in the middle. Your transport provider for whatever reason pulled out a DS1 and sent it down a different path. They accidentally crossed DS1s in the middle and are handing your DS1 to a Sprint customer and their DS1 to your customer. That's my theory at least. Justin
On Wed, 13 Aug 2008, Justin Shore wrote:
Are you sure that they are not crossing some channels in the middle and accidentally handing them to a different customer? You mention above that
I can't be sure of anything like that. I don't have access. Those who do haven't been willing to look harder...just repeating "this can't happen, it's a private line DS3" and I'm sure thinking I'm nuts or an idiot.
I would be curious to hear if Sprint is having any problems with a circuit connected to sl-bb20-dc-6-0-0.sprintlink.net, what the router is and if any
I'm very curious too, because at this point, I expect they must be seeing lots of errors (not sure what kind you get when a line running HDLC encounters PPP encapsulated packets...but I bet they're causing input errors of some sort). I'm going to try to make contact again with the guy at Sprint I spoke with last week.
Are you sure that the traffic being received by each of the T1s is their's? Do you have any way to getting flows or packets off of individual T1s and not the bundle as a whole?
Yeah...I was looking at the output of "show ip cache flow" on the customer's router. The traffic was all expected traffic. The problem was, they were receiving several copies of every expected packet.
Can you have them put the circuit into maintenance and have them test it end to end? They can't deny it when their TDR says that there's a problem.
What problem do you think they'll see if they do intrusive testing? With the circuit taken down and test equipment at both ends, there's no way for the dupe data delivered to Sprint to get back to the circuit. All that might happen is Sprint may experience even more errors on their circuit(s)...but testing on our circuit won't reveal that to the testers.
Right. By changing the encap you've basically killed the circuit. With that T1 effectively down on your end you won't be sending any packets down the problem path and aren't able to see that problem anymore with your traceroutes. However your customer with the bundle of T1s is down a circuit.
You've pretty badly misunderstood. The customer with the multiple T1s is irrelevant. Their T1s aren't part of the problem. Their T1s being full was just one symptom of the problem. The problem circuit is our DS3 from Orlando to Ocala. Changing the encaps on that DS3 (at both ends...with no OOB access...kind of scarey) from hdlc to ppp keeps the circuit working, and keeps the dupes from coming back at us, because the ppp encapsulated packets aren't understood by Sprint's routers running [I assume] hdlc...so they can't forward the packets. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
working, and keeps the dupes from coming back at us, because the ppp encapsulated packets aren't understood by Sprint's routers running [I assume] hdlc...so they can't forward the packets.
AFAIK, routers don't just forward IP packets because they "see" them. That destination MAC address needs to be set to something assigned to the router doesn't it? Unless somehow they've crossed you into some sprint router running IRB? That might be the only exception I can think of.
On Wed, Aug 13, 2008 at 02:27:33PM -0400, Andy Johnson wrote:
working, and keeps the dupes from coming back at us, because the ppp encapsulated packets aren't understood by Sprint's routers running [I assume] hdlc...so they can't forward the packets.
AFAIK, routers don't just forward IP packets because they "see" them. That destination MAC address needs to be set to something assigned to the router doesn't it? Unless somehow they've crossed you into some sprint router running IRB? That might be the only exception I can think of.
The circuit involved in this case is a DS3, whatever you get will be processed on the other side unless it's blocked by uRPF or some sort of ACL. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
participants (10)
-
Andy Johnson -
George Carey -
Jared Mauch -
Jay Hennigan -
Jay R. Ashworth -
Jon Lewis -
Justin Shore -
Laurence F. Sheldon, Jr. -
list-nanog＠pwns.ms -
Paul Wall