Howdy all, So, i'm kind of new to this so please deal with my ignorance. But, what is common practice these days for HTTP DDoS mitigation during an attack? You can of course route every offending ip address to null0 at your border. But, if it's a botnet or trojan or something, It's coming from numerous different source IPs and Null0 routes can get very cumbersome. obviously. How do you folk usually deal with this? Any input would be greatly appreciated. Cheers, Mike
On Mar 25, 2008, at 5:02 AM, Mike Lyon wrote:
Any input would be greatly appreciated.
There are devices available today from different vendors (including Cisco, full disclosure) which are intelligent DDoS-'scrubbers' and which can deal with more sophisticated types of attacks at layer-7, including HTTP and DNS. S/RTBH is also an option, keeping in mind some of the caveats you mentioned (staying mindful of attacking hosts behind proxies, botted hosts of legit customers, et. al.). ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // +66.83.266.6344 mobile It doesn't pay to dispute what you know to be true. -- Fred Reed
On Mon, Mar 24, 2008 at 5:18 PM, Roland Dobbins <rdobbins@cisco.com> wrote:
There are devices available today from different vendors (including Cisco, full disclosure) which are intelligent DDoS-'scrubbers' and which can deal with more sophisticated types of attacks at layer-7, including HTTP and DNS. S/RTBH is also an option, keeping in mind some of the caveats you mentioned (staying mindful of attacking hosts behind proxies, botted hosts of legit customers, et. al.).
Citrix (Netscaler), F5 (BIG-IP), and as Roland mentioned, Cisco, all offer varying levels of security for the content layer. If you're running Apache, you may also investigate mod_evasive, and in the case of exploits, mod_security. Naturally, your ability to filter and contain the attack with software is going to be limited by the host hardware, so it's best to take a layered approach to mitigating various attacks you face. Also important to be aware of your network architecture lest you find yourself with DDoS bits clogging the pipes just before your (expensive) defenses. :-) - Tim
On Mar 25, 2008, at 6:18 AM, Tim Yocum wrote:
If you're running Apache, you may also investigate mod_evasive, and in the case of exploits, mod_security.
mod_evasive and mod_security are definitely recommended, good point. And a good relationship with your peers/upstreams/customers/vendors is also key, so that you can get assistance when you need it. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // +66.83.266.6344 mobile It doesn't pay to dispute what you know to be true. -- Fred Reed
mike.lyon@gmail.com ("Mike Lyon") writes:
So, i'm kind of new to this so please deal with my ignorance.
:-). on the internet, everybody's new to everything since it's all changing every day. if anybody grumps at you for your ignorance, or says "i can't type that into an IOS prompt" then the fault is theirs.
But, what is common practice these days for HTTP DDoS mitigation during an attack? You can of course route every offending ip address to null0 at your border. But, if it's a botnet or trojan or something, It's coming from numerous different source IPs and Null0 routes can get very cumbersome. obviously. How do you folk usually deal with this?
i only use or recommend operating systems that have their own host based firewalls. soon that will mean pf (from openbsd but available on freebsd) but right now that means ipfw. ipfw has a "table" construct which uses a data structure similar to the kernel's routing table. with a little bit of tuning, and using X86_64 to get more kernel memory map space than I386, i've listed every member of 60K-node botnets in a table whose only use is "if a SYN comes from here, silently drop it with no ICMP response". with more tuning work, a 200K-node botnet would pose no problem. we populate these tables with a perl script that watches the apache server's logfiles. -- Paul Vixie
Paul Vixie wrote:
i only use or recommend operating systems that have their own host based firewalls. soon that will mean pf (from openbsd but available on freebsd)
pf's tables are nifty too btw :) pfsense, which is FreeBSD + pf, also has a port of snort IDS available. Provided the OP has a signature of the attack he can match on, there's a wholly open-source solution (I know snort can be configured inline to drop packets on a filtering bridge, but of course you've got the problems of half-open connections accumulating as well as the potential for migration to https).
On Mon, Mar 24, 2008 at 11:34:58PM +0000, Paul Vixie wrote:
i only use or recommend operating systems that have their own host based firewalls. soon that will mean pf (from openbsd but available on freebsd) but right now that means ipfw. ipfw has a "table" construct which uses a data structure similar to the kernel's routing table. with a little bit of tuning, and using X86_64 to get more kernel memory map space than I386, i've listed every member of 60K-node botnets in a table whose only use is "if a SYN comes from here, silently drop it with no ICMP response". with more tuning work, a 200K-node botnet would pose no problem. we populate these tables with a perl script that watches the apache server's logfiles.
Even on an untuned fbsd i386, I had success with an ipfw table with well over 1e6 entries. What finally broke was doing a table list, possibly because the command prints in sorted order. No performance problems were observed at my limited volume of perhaps 30000 hits per day. -- Barney Wolff I never met a computer I didn't like.
On Mon, Mar 24, 2008 at 11:34:58PM +0000, Paul Vixie wrote:
i only use or recommend operating systems that have their own host based firewalls.
That was exactly my problem. Barney Wolff wrote:
What finally broke was doing a table list, possibly because the command prints in sorted order.
Happened to me too. First step: Borrowed "sort.c" from Minix. Next step: Large swap file. Finally: changed the distribution. sort is one the biggest hidden problems. There are broken sorts around, I guess some of the problems are character set specific. There is no more EBCDIC but UTF-8 and UTF-16 are even worse. Related to sort, you may have more than enough memory or swap but your process wont get it. You can avoid sorting by looking into the "/proc" files. proc2pl might get you ideas, from the ISAON tools on http://iason.site.voila.fr/ You might even sort or grep the output and you can always do that on a machine that is not your router. Kind regards Peter -- Peter and Karin Dambier Cesidian Root - Radice Cesidiana Rimbacher Strasse 16 D-69509 Moerlenbach-Bonsweiher +49(6209)795-816 (Telekom) +49(6252)750-308 (VoIP: sipgate.de) mail: peter@peter-dambier.de http://iason.site.voila.fr/ https://sourceforge.net/projects/iason/ http://www.cesidianroot.com/
On 3/25/08, Peter Dambier <peter@peter-dambier.de> wrote:
proc2pl might get you ideas, from the ISAON tools on
You know, for the last year or two I've heard you go on and on about IASON. A few months ago I actually did download it and the only thing I can find in it is an assortment of scripts to manage DNS zone files. I don't see anything in there about auto detecting the network, automatically blocking DDoS or any of the other artificial intelligence you purport it has. Peter and Karin Dambier I'm not sure how to interpret this. Are Peter & Karin the same person? You be the judge. http://wiki.piratenpartei.de/images/3/39/KarinPeter.jpg Cesidian Root - Radice Cesidiana Google searches on Cesidian Root reveal a rather scruffy man running this alternative root out of Long Island, fighting for secession from the United States. I'm sure Most Rev. Dr. Cesidio Tallini, BS, PhD hc, CPC, RH-INHA, APP, AMBCS, MMPR, OEMTDV will get there, one day. http://www.cesidianroot.com/
You might want to have the reverend doctor contact customer care, as the website suggests. It seems the website is down.
There are also companies with whom you can contract for this service. It's my understanding that if you have a problem they will help you mitigate it. I'm not sure if they require some specific DDoS gear or if they are able to take advantage of their customer's gear to address the issue. In any case, it's reactive. Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Mike Lyon Sent: Monday, March 24, 2008 5:02 PM To: NANOG Subject: Mitigating HTTP DDoS attacks? Howdy all, So, i'm kind of new to this so please deal with my ignorance. But, what is common practice these days for HTTP DDoS mitigation during an attack? You can of course route every offending ip address to null0 at your border. But, if it's a botnet or trojan or something, It's coming from numerous different source IPs and Null0 routes can get very cumbersome. obviously. How do you folk usually deal with this? Any input would be greatly appreciated. Cheers, Mike
On Mar 25, 2008, at 8:10 AM, Frank Bulk - iNAME wrote:
In any case, it's reactive.
Several SPs (quite a few, actually) are offering DDoS mitigation services based upon a variety of tools and techniques, and with various pricing models. Some provide the service for their own transit/hosting/colo customers, and some provide it as an OTT/overlay service. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // +66.83.266.6344 mobile It doesn't pay to dispute what you know to be true. -- Fred Reed
On Mon, Mar 24, 2008 at 6:02 PM, Mike Lyon <mike.lyon@gmail.com> wrote:
Howdy all,
So, i'm kind of new to this so please deal with my ignorance. But, what is common practice these days for HTTP DDoS mitigation during an attack? You can of course route every offending ip address to null0 at your border. But, if it's a botnet or trojan or something, It's coming from numerous different source IPs and Null0 routes can get very cumbersome. obviously. How do you folk usually deal with this?
Any input would be greatly appreciated.
Cheers, Mike
They're a few companies that specialize in "DDOS protection type services" one company that comes to mind is Prolexic and their IPN infrastructure protection service. Prolexic will basically absorbs all attacks filter out the bad data and then deliver clean traffic back to your network. Its completly transparent to you're clients. Its not cheap but i've worked with a few internet based trading companies who used this service to litigate DDOS attacks on their network infrastructure. -- [ Rodrick R. Brown ] http://www.rodrickbrown.com http://www.linkedin.com/in/rodrickbrown
On Mon, 24 Mar 2008 23:13:25 -0400 "Rodrick Brown" <rodrick.brown@gmail.com> wrote:
They're a few companies that specialize in "DDOS protection type services" one company that comes to mind is Prolexic and their IPN infrastructure protection service. Prolexic will basically absorbs all attacks filter out the bad data and then deliver clean traffic back to your network. Its completly transparent to you're clients. Its not cheap but i've worked with a few internet based trading companies who used this service to litigate DDOS attacks on their network infrastructure.
Prolexic was indicted about 1.5 years ago for aiding gambling sites: http://www.infoworld.com/article/06/11/15/HNnyillegalonlinegambling_1.html http://www.firstamendment.com/media/NYQCIndictment.pdf Does anyone know if the indictment has been dropped? (It should be.) A quick poke around their site didn't show any news items saying that. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Hi Mike, Depending upon the type of DDOS, there are five things you should do in order: 1. immediate response: set your host based security to mitigate the attack. E.g. mod_security for Apache web server, IPTables for host firewall. This will keep the hard drives from filling up, the cpu from smoking, etc. 2. second response: gateway router or border firewall. Filter that stuff out if you can. This will keep your internal network clean so it won't affect your other systems. One quickie *temporary* fix would be to block whole networks of DSL/Cable modems. There are lists out there specifically for this--always-on broadband home PCs are a often the compromised sources of attacks. 3. third response: contact your upstream providers and ask them to take action. They can apply filters, and apply pressure to their colos. 4. make sure you have done your part: secure your network so it cannot be used for DOS attacks by applying egress filtration etc. ( http://www.sans.org/dosstep/ ); secure your hosts against future DOS attacks using things like mod_security and mod_evasive for Apache, tcplimit for IPTables, or etc. One caveat: bandwidth flooding effects can be mitigated, but you can't really do anything about it other than contacting your upstream provider. Until your provider does something, the bottleneck here is your uplink. --Patrick Darden -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Mike Lyon Sent: Monday, March 24, 2008 6:02 PM To: NANOG Subject: Mitigating HTTP DDoS attacks? Howdy all, So, i'm kind of new to this so please deal with my ignorance. But, what is common practice these days for HTTP DDoS mitigation during an attack? You can of course route every offending ip address to null0 at your border. But, if it's a botnet or trojan or something, It's coming from numerous different source IPs and Null0 routes can get very cumbersome. obviously. How do you folk usually deal with this? Any input would be greatly appreciated. Cheers, Mike
participants (12)
-
Barney Wolff -
Darden, Patrick S. -
Frank Bulk - iNAME -
Mike Lewinski -
Mike Lyon -
Paul Vixie -
Paul Wall -
Peter Dambier -
Rodrick Brown -
Roland Dobbins -
Steven M. Bellovin -
Tim Yocum