XO blocking individual IP's
I'm hoping someone has had the same experiences, and is further toward a resolution on this than I am. About 6 months ago, we noticed that XO was blackholing one specific IP out of a /24. Traces to that IP stopped on XO's network, traces to anything else out of the block went through fine. XO finally admitted that they had a new security system that identifies suspicious traffic and automatically blocks the IP for 30 minutes. We had to get the IP in question "whitelisted" by their security guys. The traffic was all legit, it was just on a high port # that they considered suspicious. There have several more cases like this, and XO has not been forthcoming with information. We're either looking to be exempted from this filtering or at least get a detailed description of how the system works. I'm not sure how they think this is acceptable from a major transit provider. Anybody else had similar problems? Clayton Haydel
----- Original Message -----
From: clayton@haydel.org
There have several more cases like this, and XO has not been forthcoming with information. We're either looking to be exempted from this filtering or at least get a detailed description of how the system works. I'm not sure how they think this is acceptable from a major transit provider.
"transit provider". Is XO the end-access provider for either you or the destination site? Or are both of those on some other connection, and XO is a bystander along the way? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
"transit provider". Is XO the end-access provider for either you or the destination site? Or are both of those on some other connection, and XO is a bystander along the way?
We're a direct customer. The IP's that I've seen them block have been both on our network and on remote networks, so I suspect their filtering would affect any traffic that happened to pass over XO. Clayton Haydel
On Nov 7, 2011, at 10:06 PM, clayton@haydel.org wrote:
"transit provider". Is XO the end-access provider for either you or the destination site? Or are both of those on some other connection, and XO is a bystander along the way?
We're a direct customer. The IP's that I've seen them block have been both on our network and on remote networks, so I suspect their filtering would affect any traffic that happened to pass over XO.
While troubleshooting another issue last week, someone in the NOC at one of our ISPs mentioned that they had encountered something similar recently. "This looks suspiciously like another XO issue we ran across in the last few months where they used a network security device that blocked 'suspicious' traffic on particular ports (although it was tcp based from what I could remember)." In our case the symptoms looked like GBLX was eating traffic which hashed to a certain theoretical link (certain src-dst-srcport-dstport combinations) in a LAG or similar, but it was happening right near the XO-GBLX edge in the forward path so it's possible it was a security device at XO's edge.
----- Original Message -----
From: clayton@haydel.org
"transit provider". Is XO the end-access provider for either you or the destination site? Or are both of those on some other connection, and XO is a bystander along the way?
We're a direct customer. The IP's that I've seen them block have been both on our network and on remote networks, so I suspect their filtering would affect any traffic that happened to pass over XO.
Ah, ok. Well, that certainly gives them standing to be filtering the traffic; whether you think their reasoning is justified becomes a different level of question at that point. I concur with you that their filtering probably isn't justified, but I suspect you'd find your contract permits it. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Oh yes! Good lord I about went insane with this. I was working with a customer single homed to cBeyond. I spent 3 hours on the phone with cBeyond to figure out what was going on, it looks like a broken route. Come to find out it was an XO "security null". The engineer on the phone from cBeyond said to me "Well, I have learned 2 things today. 1, XO nulls for 'security purposes' at random. 2, I am no longer shocked by any ridiculous policy I will ever come across again." In this case majority traffic was going from cBeyond to anywhere (via XO) and being eaten, however it was VERY tough to diagnose as all parties involved assumed this would not be occurring between source and destination without good public documentation or at least any record of this happening to someone else. Also I guess we all assumed that major bandwidth players don't filter anything. I personally think its good on paper, but very bad real life until there is a way to notify the end customer of the violation quickly. This issue literally took 3 full weeks to figure out what was going on. Yes this works great in a colo datacenter as you have the customer contact info (hopefully). But in the case where my customers provider was having the IP filtered by their transit it was hell to diagnose. In my case the customer had a single infected machine that was making outbound connections on TCP3389 in the range of about 100 connections every 5 minutes and because of this was entirely being "security nulled". Blake -----Original Message----- From: clayton@haydel.org [mailto:clayton@haydel.org] Sent: Monday, November 07, 2011 7:43 PM To: nanog@nanog.org Subject: XO blocking individual IP's I'm hoping someone has had the same experiences, and is further toward a resolution on this than I am. About 6 months ago, we noticed that XO was blackholing one specific IP out of a /24. Traces to that IP stopped on XO's network, traces to anything else out of the block went through fine. XO finally admitted that they had a new security system that identifies suspicious traffic and automatically blocks the IP for 30 minutes. We had to get the IP in question "whitelisted" by their security guys. The traffic was all legit, it was just on a high port # that they considered suspicious. There have several more cases like this, and XO has not been forthcoming with information. We're either looking to be exempted from this filtering or at least get a detailed description of how the system works. I'm not sure how they think this is acceptable from a major transit provider. Anybody else had similar problems? Clayton Haydel
So if you want to launch a DoS attack against a specific IP address you spoof TCP3389 SYNs to networks single homed to XO and they will null it for you. -- Leigh On 8 Nov 2011, at 04:36, "Blake T. Pfankuch" <blake@pfankuch.me> wrote:
Oh yes! Good lord I about went insane with this. I was working with a customer single homed to cBeyond. I spent 3 hours on the phone with cBeyond to figure out what was going on, it looks like a broken route. Come to find out it was an XO "security null". The engineer on the phone from cBeyond said to me "Well, I have learned 2 things today. 1, XO nulls for 'security purposes' at random. 2, I am no longer shocked by any ridiculous policy I will ever come across again."
In this case majority traffic was going from cBeyond to anywhere (via XO) and being eaten, however it was VERY tough to diagnose as all parties involved assumed this would not be occurring between source and destination without good public documentation or at least any record of this happening to someone else. Also I guess we all assumed that major bandwidth players don't filter anything.
I personally think its good on paper, but very bad real life until there is a way to notify the end customer of the violation quickly. This issue literally took 3 full weeks to figure out what was going on. Yes this works great in a colo datacenter as you have the customer contact info (hopefully). But in the case where my customers provider was having the IP filtered by their transit it was hell to diagnose. In my case the customer had a single infected machine that was making outbound connections on TCP3389 in the range of about 100 connections every 5 minutes and because of this was entirely being "security nulled".
Blake
-----Original Message----- From: clayton@haydel.org [mailto:clayton@haydel.org] Sent: Monday, November 07, 2011 7:43 PM To: nanog@nanog.org Subject: XO blocking individual IP's
I'm hoping someone has had the same experiences, and is further toward a resolution on this than I am. About 6 months ago, we noticed that XO was blackholing one specific IP out of a /24. Traces to that IP stopped on XO's network, traces to anything else out of the block went through fine. XO finally admitted that they had a new security system that identifies suspicious traffic and automatically blocks the IP for 30 minutes. We had to get the IP in question "whitelisted" by their security guys. The traffic was all legit, it was just on a high port # that they considered suspicious.
There have several more cases like this, and XO has not been forthcoming with information. We're either looking to be exempted from this filtering or at least get a detailed description of how the system works. I'm not sure how they think this is acceptable from a major transit provider. Anybody else had similar problems?
Clayton Haydel
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
participants (5)
-
Blake T. Pfankuch -
clayton＠haydel.org -
Jay Ashworth -
Leigh Porter -
Ryan Rawdon