I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc. But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1). Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it? Thanks, -mark
I too would be interested if someone could point a good white paper for cisco DDOS protection mechanisms and best practices in general. On Thu, May 20, 2004 at 11:52:01AM -0700, Mark Kent wrote:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
Thanks, -mark
--- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/
At 12:00 PM 20-05-04 -0700, Wayne E. Bouchard wrote:
I too would be interested if someone could point a good white paper for cisco DDOS protection mechanisms and best practices in general.
For Cisco specific ideas try: http://www.ripe.net/ripe/meetings/archive/ripe-41/tutorials/eof-ddos.pdf specifically slides 86-92 and 105-127. -Hank
On Thu, May 20, 2004 at 11:52:01AM -0700, Mark Kent wrote:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
Thanks, -mark
--- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/
On Thu, May 20, 2004 at 11:52:01AM -0700, Mark Kent wrote:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
or acl it. some providers offer blackhole services where you can inject a route to them via bgp over the same session (with communities) or over a different session that just takes blackhole routes.. that can be used by you to cause them to null0/discard the traffic within their network automatically.. with junipers being used commonly these days, and their ability to write long, complex firewall filters, I think you're seeing more people do fancier things.. I've placed filters for at least one customer (for the duration of a DoS) that match on specific packet sizes or packet ranges of a specific type. The more you know about the profile of the attack you have going on, the better others can help you mitigate it.. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
jared@puck.nether.net disait :
On Thu, May 20, 2004 at 11:52:01AM -0700, Mark Kent wrote:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
or acl it.
some providers offer blackhole services where you can inject a route to them via bgp over the same session (with communities) or over a different session that just takes blackhole routes..
that can be used by you to cause them to null0/discard the traffic within their network automatically..
At last Ripe meeting, i made a presentation about the way France Telecom is handling DDOS attack : http://www.ripe.net/ripe/meetings/ripe-48/eof.html#nocexp Slides at http://www.ripe.net/ripe/meetings/ripe-48/presentations/ripe48-eof-gillet.pd... We presented our practice from a NOC perspective (ACL, blackhole, sinkhole, netflow, sample, ... etc) and our next steps. We proposed to give this presentation at coming Nanog, but we were not so succesfull. Next nanog meeting maybe ... Vincent.
On Thursday, May 20, 2004 2:52 PM, Mark Kent wrote:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
This depends entirely on your definition of handling. To some people this means shutting down the victim to save the network as a whole. To others this means keeping everyone running smoothly, including the victim. The latter is preferred of course, but it is not for those who aren't willing to pay for it.
But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
It sounds like you're willing to blackhole the victim. In that case, yes, netflow is highly useful in finding out just who is getting attacked. Once you have that information, you can either manually contact your upstreams to have them null route the destination IP, or better yet, arrange ahead of time for a way to send properly tagged BGP announcements to them to blackhole /32s anytime you want. The alternative is to get bigger links, bigger routers, and protect the host. For bigger links and bigger routers, keep PPS in mind. Some attacks are large packets and large bandwidth, with low PPS. Other attacks are low bandwidth, but high PPS. I get hit pretty regularly with 500k-600k PPS of SYNs. While this only adds up to a few hundred megabits of traffic, that is a lot of PPS for many routers, firewalls, servers, or whatever else they might hit. Junipers, for example, have no problem with high PPS. Second, you have to figure out how to protect the host(s). We've gone with Riverhead (recently bought by Cisco) and they work quite well. I've seen attacks as high as around 650k PPS of spoofed SYNs, and the site running on a single (relatively weak) server remains up and generally unaffected by the attack.
A paper based on a presentation I did at the PAIX peering forum in December is here: http://www.stevegibbard.com/ddos-talk.htm I should probably update it a bit, but that may not happen any time soon. Slides from another presentation at the same conference are here: http://www.prostructure.com/content/research/presentations/ddos_intro/ -Steve On Thu, 20 May 2004, Mark Kent wrote:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
Thanks, -mark
Is there any quantification on what qualifies as a Large DDOS attack and perhaps a comparison of what type of routers can/can't handle such a load? Typical DDOS's that I've seen are 10-20X the normal incoming packet rate, upto and over 1Mpps. Having to switch that amount of additonal load has a tremendous impact on linecard CPU and any amount of additional features to try and protect your customer will sometimes result in a degradation to *everyone* not just the target. In my experience calling the upstream provider and having it blocked is still the only thing that can be done. When working on the backbone I've spent hours tracking the majority of flows back to one or more peering points and blocking it there where the attack isn't as concentrated and thus safer to filter. -Brent At 11:52 AM 5/20/2004, Mark Kent wrote:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
Thanks, -mark
The dearth of comprehensive BCP asserting the end-all-be-all for DDoS is likely and largely due to the lack of an end-all-be-all DDoS. The range of variants, strains, chewy fillings and flavors of fuxor out there beg different techniques for alleviation, so prescribing a single poultice for blanket application does not seem to be in wide practice outside marketing stratagem and other blustering. The resources requiring protection and receiving priority, as well as the trade-off in exacting reactive measures, also have a say in how things are managed. In general, however, yeah...identifying the source or target is a must. Or a source port or destination port or protocol type or packet size or point of ingress/egress...the list of signature-worthy candidates is significant and also determines how a DDoS is triaged. The only thing that can be said for certain is that *some* unifying factor must be discovered. :P Furthermore, how you do that and what you do with that is a fluid thing, and further refinement or definition of the type of DDoS you are seeking to relieve may be required before you will be able to root out an attack management template that is worth its salt. Blackhole servers, sinkhole routers, IDS, extrusion detection, heuristic baselining, and definitely bigger routers never hurt this effort either. ;) If you are able to elaborate on what you might be seeking to accomplish on- or off-list, I will try to proffer any appropriate resources I have available. Good luck. --ra -- Rachael Treu-Gomes, CISSP rara@navigo.com ..quis costodiet ipsos custodes?.. On Thu, May 20, 2004 at 11:52:01AM -0700, Mark Kent said something to the effect of:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
Thanks, -mark
On May 20, 2004, at 12:52 PM, Mark Kent wrote:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about how to be a good net.citizen (we already are), how to tune a kernel to better withstand a syn flood, router stuff you can do to protect hosts behind it, how to track the attack back to the source, how to determine the nature of the traffic, etc.
There's lots and lots of really-useful-very-often-multi-vendor stuff here: ftp://ftp-eng.cisco.com/cons/isp/security/ In particular, under the bootcamp and CPN-summit stuff. Though it may seem vendor-specific per logos and the like, I know of several (more than three) vendors that have contributed to this content, most of which is very practical and generally informative, and should be applicable to most deployed vendors. There's also some VOD stuff here that expands some areas of the content: http://www.getitmm.com/bootcampflash/launch.html HTH, -danny
mark@noc.mainstreet.net (Mark Kent) writes:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about ... But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
that seems hardly worthwhile. ddos is astonishingly easier to launch than to defend against. if you stop a flow the attacker *might* get bored and decide to do something else, but they could also decide to attack you from a different direction, or wait two days and do it all over again, and every time they attack and you defend it's 10 minutes of their time and 10 hours of yours. far better to involve law enforcement and get some bad guys arrested, if you possibly can. this changes your costs from 10 hours to 15 hours but it actually puts some chips on the table and makes the game worthwhile. -- Paul Vixie
----- Original Message ----- From: "Paul Vixie" <vixie@vix.com> To: <nanog@merit.edu> Sent: Thursday, May 20, 2004 9:48 PM Subject: Re: handling ddos attacks
mark@noc.mainstreet.net (Mark Kent) writes:
I've been trying to find out what the current BCP is for handling ddos attacks. Mostly what I find is material about ... But I don't care about most of that. I care that a gazillion pps are crushing our border routers (7206/npe-g1).
Other than getting bigger routers, is it still the case that the best we can do is identify the target IP (with netflow, for example) and have upstreams blackhole it?
that seems hardly worthwhile. ddos is astonishingly easier to launch than to defend against. if you stop a flow the attacker *might* get bored and decide to do something else, but they could also decide to attack you from a different direction, or wait two days and do it all over again, and
every
time they attack and you defend it's 10 minutes of their time and 10 hours of yours.
far better to involve law enforcement and get some bad guys arrested, if you possibly can. this changes your costs from 10 hours to 15 hours but it actually puts some chips on the table and makes the game worthwhile. -- Paul Vixie
Hey Paul ! Ok, I 'll buy that right now; we have a DDoS Attack on our core nameservers from 66.165.10.24. Where do we start, do I call the police in Bellingham or Washington State Police. We have blocked their ips but, we know they will come in another way. Peter OrgName: Western Washington University OrgID: WWU Address: Computer Center Address: 516 High Street City: Bellingham StateProv: WA PostalCode: 98225 Country: US NetRange: 66.165.0.0 - 66.165.31.255 CIDR: 66.165.0.0/19 NetName: WWU-RESIDENT-1 NetHandle: NET-66-165-0-0-2 Parent: NET-66-165-0-0-1 NetType: Reassigned NameServer: VIKING.WWU.EDU NameServer: HENSON.CC.WWU.EDU Comment: RegDate: 2002-08-15 Updated: 2002-08-15 TechHandle: JSW12-ARIN TechName: Williams, J. Scott TechPhone: +1-360-650-2868 TechEmail: scott@cc.wwu.edu
On Thu, 20 May 2004, P.Schroebel wrote:
Ok, I 'll buy that right now; we have a DDoS Attack on our core nameservers from 66.165.10.24. Where do we start, do I call the police in Bellingham or Washington State Police. We have blocked their ips but, we know they will come in another way.
Call your local branch of the US Secret Service, if you're in the states, and ask for their electronic crimes division. If you're not in the states, contact your comprable local authority. They can work with you to coordinate with other jurisdictions, etc. You may have some luck directly with the local police at the point of origin, but it generally helps to have a broader agency involved to coordinate matters. -- Tim Wilde twilde@dyndns.org Systems Administrator Dynamic Network Services, Inc. http://www.dyndns.org/
On May 20, 2004, at 8:10 PM, Tim Wilde wrote:
Call your local branch of the US Secret Service, if you're in the states, and ask for their electronic crimes division. If you're not in the states, contact your comprable local authority. They can work with you to coordinate with other jurisdictions, etc.
You may have some luck directly with the local police at the point of origin, but it generally helps to have a broader agency involved to coordinate matters.
I'd love to hear from anyone who has actually been successful prosecuting an attacker for launching a "distributed" DOS attack. I suspect it occurs very INfrequently (with the recent trend in extortion aside, as it often results in "paper trails") -- unfortunately. Any pointers? -danny
participants (13)
-
Brent Van Dussen
-
Danny McPherson
-
Hank Nussbacher
-
Jared Mauch
-
Mark Kent
-
Matt Buford
-
P.Schroebel
-
Paul Vixie
-
Rachael Treu-Gomes
-
Steve Gibbard
-
Tim Wilde
-
Vincent Gillet - Opentransit
-
Wayne E. Bouchard