On Tue, Feb 23, 2010 at 02:46:54PM -0500, Paul Stewart wrote:
Hi folks...
We have a strange series of events going on in the past while.... Brief history here, looking for input from the community - especially some of the security folks on here.
We provide web hosting services - one of our hosting boxes was found a while back with root kits installed, un patched software and lots of other "goodies". With some staff changes in place (don't think I need to elaborate on that) we are trying to clean up several issues including this particular server. A new server was provisioned, patched, and deployed. User data was moved over and now the same issue is coming back....
The problem is that a user on this box appears to be launching high traffic DOS attacks from it towards other sites. These are UDP based floods that move around from time to time - most of these attacks only last a few minutes.
Counting outbound udp bytes and packets can help spot anomalies. Something like this would help but may be unwieldy if you have thousands of users on a single box: WANIF=eth0 userlist="userA userB user..." for i in ${userlist} do iptables -N ${i}_UDP iptables -I OUTPUT -m owner -o ${WANIF} -p udp --uid-owner ${i} -j ${i}_UDP done Then look at counters with: iptables -nvL OUTPUT | grep _UDP | sort....... I wouldn't leave this in place full-time for thousands of accounts though without attempting to measure the impact on network performance.
I've done tcpdumps within seconds of the attack starting and to date been unable to find the source of this attack (we know the server, just not sure which customer it is on the server that's been compromised). Several hours of scanning for php, cgi, pl type files have been wasted and come up nowhere...
It's been suggested to dump IDS in front of this box and I know I'll get some feedback positive and negative in that aspect.
What tools/practices do others use to resolve this issue? It's a Centos 5.4 box running latest Plesk control panel.
Typically we have found it easy to track down the offending script or program - this time hasn't been easy at all...
Thanks,
Paul
----------------------------------------------------------------------------
"The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you."