Effective ways to deal with DDoS attacks?
There's been plenty of discussion about DDoS attacks, and my IDS system is darn good at identifying them. But what are effective methods for large service-provider networks (ie ones where a firewall at the front would not be possible) to deal with DDoS attacks? Current method of updating ACLs with the source and/or destination are slow and error-prone and hard to maintain (especially when the target of the attack is a site that users would like to access). A rather extensive survey of DDoS papers has not resulted in much on this topic. What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks? Thanks. Pete.
http://www.secsup.org/Tracking/ UUNet uses that...others might as well, Shrug. Quick, simple, effective tracking of DDoS attacks. As for identifying attacks, quite honestly large ISP's are typically still relying on customer notification. I know that's how we do it. On Wed, 1 May 2002, Pete Kruckenberg wrote:
There's been plenty of discussion about DDoS attacks, and my IDS system is darn good at identifying them. But what are effective methods for large service-provider networks (ie ones where a firewall at the front would not be possible) to deal with DDoS attacks?
Current method of updating ACLs with the source and/or destination are slow and error-prone and hard to maintain (especially when the target of the attack is a site that users would like to access).
A rather extensive survey of DDoS papers has not resulted in much on this topic.
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
Thanks. Pete.
On Wed, 1 May 2002, Pete Kruckenberg wrote:
A rather extensive survey of DDoS papers has not resulted in much on this topic.
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
Hazaa.. something I know a little about. DDoS attacks by their very nature, are distributed. The primary purpose of more DDoS attacks is to flood the target's upstream connection to the point of saturation. As time goes by, tools are being developed (in fact they're used now) that completely randomize the TCP or UDP ports attacked, or use a variety of icmp types in the attack. So cuurrently the only way you can 'block' such attacks is to block all packets for the offending protocol as far upstream as you possibly can, but this is not ideal. If you're being attacked by a SYN flood, you can ask try to rate-limit the flood at your border (possible on Cisco IOS 12.0 and higher, and probably other routers too?) If you're being smurfed, you can block ICMP Echo Reply's inbound to the target IP. It all depends on the TYPE of attack. Having said that, it's only a matter of time before somebody releases a tool that saturates a line by spooofing the source, randomizing the protocol, and ports, and maybe even atacking other hosts on the same subnet, etc etc. The only thing you can try and do is work with your upstream provider and try to trace the source of the attacks back, but that's incredibly difficult. As a side note, does anyone know the status of the ICMP Traceback proposal? The ieft draft expired yesterday: http://www.ietf.org/internet-drafts/draft-ietf-itrace-01.txt
On Thu, May 02, 2002 at 01:49:40AM +0100, Avleen Vig wrote:
DDoS attacks by their very nature, are distributed. The primary purpose of more DDoS attacks is to flood the target's upstream connection to the point of saturation.
Actually the original goal (and probably still the primary benefit) of DDoS attacks was to evade detection by using so many hosts that any specific one either went unnoticed or unreported. Having multiple sources to bypass any potential congestion (since a DoS is only as effective as the weakest link along the path) and filtering is still what I would rank as a secondary effect. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Thu, 2 May 2002, Avleen Vig wrote:
On Wed, 1 May 2002, Pete Kruckenberg wrote:
A rather extensive survey of DDoS papers has not resulted in much on this topic.
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
Hazaa.. something I know a little about.
DDoS attacks by their very nature, are distributed. The primary purpose of more DDoS attacks is to flood the target's upstream connection to the point of saturation.
As time goes by, tools are being developed (in fact they're used now) that completely randomize the TCP or UDP ports attacked, or use a variety of icmp types in the attack. So cuurrently the only way you can 'block' such attacks is to block all packets for the offending protocol as far upstream as you possibly can, but this is not ideal.
If you're being attacked by a SYN flood, you can ask try to rate-limit the flood at your border (possible on Cisco IOS 12.0 and higher, and probably other routers too?)
Let me say this one more time... "RATE LIMITS DON'T DO SHIT TO STOP ATTACKS" for the victim atleast, all they do is make the job of the attacker that much easier. For instance: 1) I synflood www.avleen.org 2) you rate-limit syns to 1MB 3) I now only flood 1MB and I still win So, don't rely on a rate-limit as its not going to help.
If you're being smurfed, you can block ICMP Echo Reply's inbound to the target IP.
It all depends on the TYPE of attack.
This point is very good, depending on the attack your ISP (or you) might have to take different counter measures to stop the attack... You (or your ISP) will need to know as much as possible about the attack, the defenses available on the hardware/software in the network, and actually be available when there is a problem.
Having said that, it's only a matter of time before somebody releases a tool that saturates a line by spooofing the source, randomizing the protocol, and ports, and maybe even atacking other hosts on the same subnet, etc etc.
The only thing you can try and do is work with your upstream provider and try to trace the source of the attacks back, but that's incredibly difficult.
This depends :) Call us, if you are our customer, and I guarantee that someone will be there to resolve your issue, most times in 5 minutes or less. Perhaps other ISP's should start having some folks on staff and available for these tasks????? (hint, Hint, HINT!!!)
As a side note, does anyone know the status of the ICMP Traceback proposal? The ieft draft expired yesterday: http://www.ietf.org/internet-drafts/draft-ietf-itrace-01.txt
This is a wasted effort. It's not feasible, nor is it reasonable. There are tools in place today that perform this function without the required changes to protocols/functionality. Why would you want to make things incrementally more difficult when the technology exists today to do this? -Chris
On Thu, May 02, 2002 at 04:28:44AM +0000, Christopher L. Morrow wrote:
Let me say this one more time... "RATE LIMITS DON'T DO SHIT TO STOP ATTACKS" for the victim atleast, all they do is make the job of the attacker that much easier. For instance:
1) I synflood www.avleen.org 2) you rate-limit syns to 1MB 3) I now only flood 1MB and I still win
So, don't rely on a rate-limit as its not going to help.
Thank you, I can't make this point enough and people still say "we'll just rate limit!". Filtering is only as good as your ability to DETERMINE WHAT TO FILTER. The only time you can get anything from this is when you admit defeat on keeping your services responding to new connection but want to keep existing connections and/or the end servers from failing completely. Depending on the service in question this may or may not be a good goal. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Thu, 2 May 2002, Christopher L. Morrow wrote: > On Thu, 2 May 2002, Avleen Vig wrote: > > If you're being attacked by a SYN flood, you can ask try to rate-limit the > > flood at your border (possible on Cisco IOS 12.0 and higher, and probably > > other routers too?) > Let me say this one more time... "RATE LIMITS DON'T DO SHIT TO STOP > ATTACKS" for the victim atleast, all they do is make the job of the > attacker that much easier. For instance: > 1) I synflood www.avleen.org > 2) you rate-limit syns to 1MB > 3) I now only flood 1MB and I still win > So, don't rely on a rate-limit as its not going to help. Actually it's avleen.com :) But joking aside you make a valid point. I should have clarified my statement by saying that I was thinking of the whole network getting attacked rather than the single host. Yes, one host may be the target, but when your bandwidth is saturates, your entire network is effectively offline. I have seen some 'clever' handling of DoS / DDoS from the attackers front where they don't often like to waste bandwidth during an attack. If a 1Mb flood will take you offline, they won't bother using 100Mb. Maybe 2Mb but not 100Mb :) This can be a Good Thing(tm) if you're willing to temporarily let one host suffer so that the rest of your network can stay alive. > > The only thing you can try and do is work with your upstream provider and > > try to trace the source of the attacks back, but that's incredibly > > difficult. > This depends :) Call us, if you are our customer, and I guarantee that > someone will be there to resolve your issue, most times in 5 minutes or > less. Perhaps other ISP's should start having some folks on staff and > available for these tasks????? (hint, Hint, HINT!!!) I wish other ISPs would start doing this.
At 01:49 AM 02-05-02 +0100, Avleen Vig wrote:
As time goes by, tools are being developed (in fact they're used now) that completely randomize the TCP or UDP ports attacked, or use a variety of icmp types in the attack. So cuurrently the only way you can 'block' such attacks is to block all packets for the offending protocol as far upstream as you possibly can, but this is not ideal.
If you're being attacked by a SYN flood, you can ask try to rate-limit the flood at your border (possible on Cisco IOS 12.0 and higher, and probably other routers too?)
ACLs have been a good tool for the past number of years to stop DOS attacks but they suffer one very bad feature - they throw away the good packets along with the bad packets. The same goes for CAR. The same goes for taking a /32 and null routing it. Consider Amazon being hit with a DDOS attack from random spoofed IPs to their web site. You can't block on source IP since it is random. If you block on destination IP - you end up taking Amazon off the network (the ultimate aim of the attacker) at a daily revenue loss of over $1M. Therefore, the solutions in the future will be mechanisms that can filter and sieve the bad packets out and let the good packets thru. Disclosure: I consult to an anti-DDOS company with this philosophy. Hank Consultant Riverhead Networks (formerly Wanwall Networks) www.riverhead.com
If you're being smurfed, you can block ICMP Echo Reply's inbound to the target IP.
It all depends on the TYPE of attack.
Having said that, it's only a matter of time before somebody releases a tool that saturates a line by spooofing the source, randomizing the protocol, and ports, and maybe even atacking other hosts on the same subnet, etc etc.
The only thing you can try and do is work with your upstream provider and try to trace the source of the attacks back, but that's incredibly difficult.
As a side note, does anyone know the status of the ICMP Traceback proposal? The ieft draft expired yesterday: http://www.ietf.org/internet-drafts/draft-ietf-itrace-01.txt
There is one more usefull policy to decrease effectiveness of attacks such as DDOS. This is _refusal_ policy. In case of SYN attack, if system ALWAYS accept SYN packets, dropping old waiting half-open connections if there is not enougph room, SYN attack became much less dangerous - if 90% traffic is DDOS and 10% traffic is usefull, most _right_ connections will compleet succesfully (because if room for the half-open connections is big enougph, DDOS packets will drop mainly other DDOS related connections). It's a common approach - NEVER refuse new requests for the resource, if there is not enougph resource, drop some of the old users of the resource... In a lot of cases, it will derevent deadlock because you will be dropping the users who exhausted resource more than _correct_ users. It relay to the half connections, memory, etc etc... If case of _random_ IP addresses - ok, what's happen if you'll drop (always) FIRST packet from any new IP address? For the good SYN packet, you will receive a second request in a second; for a false one, you just filter out DDOS itself. This is not universal, but for the simple DDOS it will work. It's not too bad to slow down all connections (by requesting _repeated request_, for example) instead of blocking them. So, I think things are not so bad. ----- Original Message ----- From: "Hank Nussbacher" <hank@att.net.il> To: "Avleen Vig" <lists-nanog@silverwraith.com>; "Pete Kruckenberg" <pete@kruckenberg.com> Cc: <nanog@merit.edu> Sent: Thursday, May 02, 2002 1:51 AM Subject: Re: Effective ways to deal with DDoS attacks?
At 01:49 AM 02-05-02 +0100, Avleen Vig wrote:
As time goes by, tools are being developed (in fact they're used now) that completely randomize the TCP or UDP ports attacked, or use a variety of icmp types in the attack. So cuurrently the only way you can 'block' such attacks is to block all packets for the offending protocol as far upstream as you possibly can, but this is not ideal.
If you're being attacked by a SYN flood, you can ask try to rate-limit the flood at your border (possible on Cisco IOS 12.0 and higher, and probably other routers too?)
ACLs have been a good tool for the past number of years to stop DOS attacks but they suffer one very bad feature - they throw away the good packets along with the bad packets. The same goes for CAR. The same goes for taking a /32 and null routing it. Consider Amazon being hit with a DDOS attack from random spoofed IPs to their web site. You can't block on source IP since it is random. If you block on destination IP - you end up taking Amazon off the network (the ultimate aim of the attacker) at a daily revenue loss of over $1M.
Therefore, the solutions in the future will be mechanisms that can filter and sieve the bad packets out and let the good packets thru.
Disclosure: I consult to an anti-DDOS company with this philosophy.
Hank Consultant Riverhead Networks (formerly Wanwall Networks) www.riverhead.com
If you're being smurfed, you can block ICMP Echo Reply's inbound to the target IP.
It all depends on the TYPE of attack.
Having said that, it's only a matter of time before somebody releases a tool that saturates a line by spooofing the source, randomizing the protocol, and ports, and maybe even atacking other hosts on the same subnet, etc etc.
The only thing you can try and do is work with your upstream provider and try to trace the source of the attacks back, but that's incredibly difficult.
As a side note, does anyone know the status of the ICMP Traceback proposal? The ieft draft expired yesterday: http://www.ietf.org/internet-drafts/draft-ietf-itrace-01.txt
On Thu, May 02, 2002 at 01:42:03AM -0700, Alexei Roudnev wrote:
It's a common approach - NEVER refuse new requests for the resource, if there is not enougph resource, drop some of the old users of the resource... In a lot of cases, it will derevent deadlock because you will be dropping the users who exhausted resource more than _correct_ users. It relay to the half connections, memory, etc etc...
If case of _random_ IP addresses - ok, what's happen if you'll drop (always) FIRST packet from any new IP address? For the good SYN packet, you will receive a second request in a second; for a false one, you just filter out DDOS itself. This is not universal, but for the simple DDOS it will work.
It all depends on *what* is being DoS'd. The application? The TCP listen queue? The number of interrupts/sec that box can handle? The pipe on that box? The switch? The router? The providers router? The pipe between any of the previous 3? Any of these are potentially valid targets. Given a network which doesn't break, one can very easily expect a FreeBSD -STABLE box on a p3 1GHz to survive at least 100kpps of SYN flood. Past 144kpps you clog FastE completely, and need to go to GigE. I've seen well configured servers eatting 250kpps of SYN floods while still providing uninterrupted service, which is probably more then your router will be able to handle unless its a GSR or Juniper. But if you are on a DS3, or even if you have an OC48 from a provider who either doesn't want to or doesn't know how to protect their infrastructure from attacks, all of that means absolutily NOTHING. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Thu, 2 May 2002, Hank Nussbacher wrote:
At 01:49 AM 02-05-02 +0100, Avleen Vig wrote:
As time goes by, tools are being developed (in fact they're used now) that completely randomize the TCP or UDP ports attacked, or use a variety of icmp types in the attack. So cuurrently the only way you can 'block' such attacks is to block all packets for the offending protocol as far upstream as you possibly can, but this is not ideal.
If you're being attacked by a SYN flood, you can ask try to rate-limit the flood at your border (possible on Cisco IOS 12.0 and higher, and probably other routers too?)
ACLs have been a good tool for the past number of years to stop DOS attacks but they suffer one very bad feature - they throw away the good packets along with the bad packets. The same goes for CAR. The same goes for taking a /32 and null routing it. Consider Amazon being hit with a DDOS attack from random spoofed IPs to their web site. You can't block on source IP since it is random. If you block on destination IP - you end up taking Amazon off the network (the ultimate aim of the attacker) at a daily revenue loss of over $1M.
So, just filter and track quickly... move the block as far back as you can. Have the customer remain agile also. :)
On Wed, May 01, 2002 at 05:18:24PM -0600, Pete Kruckenberg wrote:
A rather extensive survey of DDoS papers has not resulted in much on this topic.
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
"DDoS attacks" is such a generic term. There are a wide variety of attacks which each need to be handled in their own way, the extra "D" is just one possible twist. Can you explain what kind of attack you're interested in? I've tried to compile a list of the *practical* things everyone needs to know (but usually doesn't) to handle DoS effectively, try reading: http://www.e-gerbil.net/ras/projects/dos/dos.txt -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Wed, 1 May 2002, Richard A Steenbergen wrote:
"DDoS attacks" is such a generic term. There are a wide variety of attacks which each need to be handled in their own way, the extra "D" is just one possible twist. Can you explain what kind of attack you're interested in?
We experience a lot of types of attacks ("education/research network" = "easy hacker target"). With DDoS incidents, it seems we are more often an unknowing/unwilling participant than the target, partly due to owning big chunks of IP address space. We most frequently are the zombie/reflector participants in an attack that originates outside our network, to a target outside our network. As many as 8,000 hosts on our network are reflecting SYN floods in the current attacks. Identification doesn't seem to be a problem. Snort is doing far too well notifying us. Responding and managing all of the defenses is becoming a lot of pain-staking work, and error-prone (why can't Cisco make ACLs easier to manage). Our approach so far has been temporary blocks (via ACL) of the target address. Blocking 8,000 internal addresses, many legitimate (secured) Web servers, generates more complaints. I'm thinking about a scripted Zebra feed where route injections are triggered by Snort. Routes for the target and/or SYN flood reflector hosts could be injected temporarily during the attack to border routers, which would route-map those routes to Null0. Script periodically withdraws routes to see if the attack is over (some of these last weeks, some only last a few seconds), to minimize the impact on those otherwise legitimate hosts. Has anyone tried this kind of an approach or any other type of automated/efficient approach to dampen the "zombie" side of the DDoS attack? Pete.
On Wed, 1 May 2002, Pete Kruckenberg wrote:
We experience a lot of types of attacks ("education/research network" = "easy hacker target"). With DDoS incidents, it seems we are more often an unknowing/unwilling participant than the target, partly due to owning big chunks of IP address space.
Universities are hacker training grounds, but also have much better network security response than most corporate networks. Whatever problems you have, the rest of us will have soon enough it may just take us longer to notice it.
Has anyone tried this kind of an approach or any other type of automated/efficient approach to dampen the "zombie" side of the DDoS attack?
Has anyone implemented Bellovin's Pushback in a production network yet?
On Wed, 1 May 2002, Pete Kruckenberg wrote:
On Wed, 1 May 2002, Richard A Steenbergen wrote:
"DDoS attacks" is such a generic term. There are a wide variety of attacks which each need to be handled in their own way, the extra "D" is just one possible twist. Can you explain what kind of attack you're interested in?
We experience a lot of types of attacks ("education/research network" = "easy hacker target"). With DDoS incidents, it seems we are more often an unknowing/unwilling participant than the target, partly due to owning big chunks of IP address space.
We most frequently are the zombie/reflector participants in an attack that originates outside our network, to a target outside our network. As many as 8,000 hosts on our network are reflecting SYN floods in the current attacks.
Sounds like its time for a firewall on your network :)
Identification doesn't seem to be a problem. Snort is doing far too well notifying us. Responding and managing all of the defenses is becoming a lot of pain-staking work, and error-prone (why can't Cisco make ACLs easier to manage).
they aren't tough to 'manage' they are sometimes tough to live with though :(
Our approach so far has been temporary blocks (via ACL) of the target address. Blocking 8,000 internal addresses, many legitimate (secured) Web servers, generates more complaints.
I'm thinking about a scripted Zebra feed where route injections are triggered by Snort. Routes for the target and/or SYN flood reflector hosts could be injected temporarily during the attack to border routers, which would route-map those routes to Null0. Script periodically withdraws routes to see if the attack is over (some of these last weeks, some only last a few seconds), to minimize the impact on those otherwise legitimate hosts.
This is a nice idea, anything 'scripted' is prone to abuse though ;( all of a sudden www.your.edu is dead.. on class registration day no less.
Has anyone tried this kind of an approach or any other type of automated/efficient approach to dampen the "zombie" side of the DDoS attack?
On Wed, 1 May 2002, Pete Kruckenberg wrote:
We experience a lot of types of attacks ("education/research network" = "easy hacker target"). With DDoS incidents, it seems we are more often an unknowing/unwilling participant than the target, partly due to owning big chunks of IP address space.
We most frequently are the zombie/reflector participants in an attack that originates outside our network, to a target outside our network. As many as 8,000 hosts on our network are reflecting SYN floods in the current attacks.
I finally found a paper on this type of attack. http://grc.com/files/drdos.pdf and http://grc.com/dos/grcdos.htm describe the attack and a few possible defenses, though they are about as ineffective as most other DDoS defenses. Pete.
On Wed, 1 May 2002, Pete Kruckenberg wrote:
I finally found a paper on this type of attack. http://grc.com/files/drdos.pdf and http://grc.com/dos/grcdos.htm describe the attack and a few possible defenses, though they are about as ineffective as most other DDoS defenses.
Has NANOG stooped to quoting Steve Gibson as an expert on DDoS attacks? -Ralph
On Mon, May 06, 2002 at 05:15:25PM -0600, Pete Kruckenberg wrote:
I finally found a paper on this type of attack. http://grc.com/files/drdos.pdf and http://grc.com/dos/grcdos.htm describe the attack and a few possible defenses, though they are about as ineffective as most other DDoS defenses.
Don't confuse the rantings of a nutcase and his T1 with useful information about DoS. I have to admit I like the direction the made up acronyms are going though, can we have MS-DOS next? :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
Once upon a time, Richard A Steenbergen <ras@e-gerbil.net> said:
Don't confuse the rantings of a nutcase and his T1 with useful information about DoS. I have to admit I like the direction the made up acronyms are going though, can we have MS-DOS next? :)
You mean MicroSoft Denial Of Service? I think it is more commonly spelled O-U-T-L-O-O-K. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On Wed, 1 May 2002, Pete Kruckenberg wrote:
There's been plenty of discussion about DDoS attacks,
and then again, there has been much discussion on simple DoS attacks, where the term DDoS is erroneously used... I am very much not trying to imply that this is the case here, but it's important that the two be thoroughly distinguished from each other - they are totally different things to deal with.
and my IDS system is darn good at identifying them.
Chances are your IDS is detecting simple DoS, or maybe tiny scale DDoS. Full DDoS attacks do not require and IDS to detect ;-) In fact, if your IDS doesn't tip over under the load of a full blown DDoS, I'd sure like to know what it's using for an engine...
But what are effective methods for large service-provider networks (ie ones where a firewall at the front would not be possible) to deal with DDoS attacks?
True DDoS attacks, fortunately, are rarer than most people believe. If they were not, the Internet as we know it would look a lot more like a telephone system in USSR-at-it's-worst-days. For example, of the two recent DDoS's I have been on the receiving end of, the first was generating a little over 300mbit/sec (steady for a prolonged time), and the second went over that by a fair bit. In both cases, we had core equipment (M20's and BSN5000's) fall over and die trying to "work" the events. Additionally, our upstream peers also had core equipment fall over, and we all came the [now obvious] conclusion that the only way to stop these attacks was to completely null route ourselves at our upstreams (they tried filter-fishing for specific data which may have helped our investigation, but when their routers started wheezing, we gave them the OK to just send us straight into the bit bucket till it was over...
Current method of updating ACLs with the source and/or destination are slow and error-prone and hard to maintain (especially when the target of the attack is a site that users would like to access).
We captured several seconds of the last DDoS and came up with over 700 participating hosts...
A rather extensive survey of DDoS papers has not resulted in much on this topic.
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
A great deal of thought is being expended on this question, I am certain, however, how many of these thought campaings have born significant fruit yet, I do not know.
Thanks. Pete.
-- Yours, J.A. Terranson sysadmin@mfn.org If Governments really want us to behave like civilized human beings, they should give serious consideration towards setting a better example: Ruling by force, rather than consensus; the unrestrained application of unjust laws (which the victim-populations were never allowed input on in the first place); the State policy of justice only for the rich and elected; the intentional abuse and occassionally destruction of entire populations merely to distract an already apathetic and numb electorate... This type of demogoguery must surely wipe out the fascist United States as surely as it wiped out the fascist Union of Soviet Socialist Republics. The views expressed here are mine, and NOT those of my employers, associates, or others. Besides, if it *were* the opinion of all of those people, I doubt there would be a problem to bitch about in the first place... --------------------------------------------------------------------
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
A great deal of thought is being expended on this question, I am certain, however, how many of these thought campaings have born significant fruit yet, I do not know.
How about the following : We develop a new community , being fully transitive (666 would be appropriate ) and either build into router code or create a route map to null route anything that contains this community. The effect of this being the distribution of the force of the attack. This aside, how effective would be using a no export community with ones peers (being non transitive, it would still distribute the force of the attack).
Then you are pushing out /32's and peers would need to accept them. Then someone will want to blackhole /30's, /29's, etc. Route bloat. Yum! Additionally you are creating a way to basically destroy the Internet as a whole. One kiddie gets ahold of a router, say of a large backbone provider, takes one of their aggregate blocks (/16? /10? /8?) and splits it into /32 announcements. Anyways, some providers already allow you to set a community on a route, and they will inturn "blackhole" it for you. I believe Teleglobe does this for some customers and I know UUNet does this for all customers. On Wed, 1 May 2002, Wojtek Zlobicki wrote:
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
A great deal of thought is being expended on this question, I am certain, however, how many of these thought campaings have born significant fruit yet, I do not know.
How about the following :
We develop a new community , being fully transitive (666 would be appropriate ) and either build into router code or create a route map to null route anything that contains this community. The effect of this being the distribution of the force of the attack.
This aside, how effective would be using a no export community with ones peers (being non transitive, it would still distribute the force of the attack).
Then you are pushing out /32's and peers would need to accept them. Then someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!
I am in no way proposing discounting current filtering rules. There are alway two different intersts one must consider, one that of the customer and two that of the service provider. If a large block must be filtered so be it. Where are providers drawing the line ? Anyone have somewhat detailed published policies as to what a provider can do in order to protect their nework as a whole. At what point (strength of the attack) does a customers netblock (assuming a /24 for example) get null routed by whichever party.
Anyways, some providers already allow you to set a community on a route, and they will inturn "blackhole" it for you. I believe Teleglobe does this for some customers and I know UUNet does this for all customers.
When the attack is distributed, having one or two providers (even if they are UUNET or Teleglobe) is just not enough. Must private routing policy be developed in order to make my suggestion work. The reason that so many methods likely fail are the difficulty of implementation and low implementation.
On Wed, 1 May 2002, Wojtek Zlobicki wrote:
Where are providers drawing the line ? Anyone have somewhat detailed published policies as to what a provider can do in order to protect their nework as a whole. At what point (strength of the attack) does a customers netblock (assuming a /24 for example) get null routed by whichever party.
Most providers likely have a policy similar to: "I can't sacrafice 1 my network for 1 customer". So, if the attack is sufficient to degrade service on the ISP network most likely the customer under attack will get null routed.
Anyways, some providers already allow you to set a community on a route, and they will inturn "blackhole" it for you. I believe Teleglobe does this for some customers and I know UUNet does this for all customers.
When the attack is distributed, having one or two providers (even if they are UUNET or Teleglobe) is just not enough. Must private routing policy be developed in order to make my suggestion work. The reason that so many methods likely fail are the difficulty of implementation and low implementation.
Hmm, perhaps FIRST customers should insist that their ISP have some 24/7 security contact that can actually help in the case of an attack. Today there are very few that have this capability. I'd say from personal experience that the number is way too small, even in the 'large' ISP arena :( More pressure from customers for real security would be a good start.
On Thu, May 02, 2002 at 04:45:43AM +0000, Christopher L. Morrow wrote:
On Wed, 1 May 2002, Wojtek Zlobicki wrote:
Where are providers drawing the line ? Anyone have somewhat detailed published policies as to what a provider can do in order to protect their nework as a whole. At what point (strength of the attack) does a customers netblock (assuming a /24 for example) get null routed by whichever party.
Most providers likely have a policy similar to: "I can't sacrafice 1 my network for 1 customer". So, if the attack is sufficient to degrade service on the ISP network most likely the customer under attack will get null routed.
Are you saying UUnet, assuming for a sec that I am a customer of UUnet (just for the sake of the argument), UU will not null route my ircd if it it gets attacked on regular basis, say *daily* ? Furthermore you are going to consistently place filters on your routers, take them out within the 24h (or whatever then-current policy of UUnet is) and track attacks back to their sources within the boundaries of your backbone on a daily basis? ;) Will you do that for say a regular T1 customer or do I need more "commitment" as sales droids like to put it, to even consider such a service ? ;)
Hmm, perhaps FIRST customers should insist that their ISP have some 24/7 security contact that can actually help in the case of an attack. Today there are very few that have this capability. I'd say from personal experience that the number is way too small, even in the 'large' ISP arena :(
More pressure from customers for real security would be a good start.
sigh, tried and failed, miserably I might add. -Basil
On Wed, 1 May 2002, Basil Kruglov wrote:
On Thu, May 02, 2002 at 04:45:43AM +0000, Christopher L. Morrow wrote:
On Wed, 1 May 2002, Wojtek Zlobicki wrote:
Where are providers drawing the line ? Anyone have somewhat detailed published policies as to what a provider can do in order to protect their nework as a whole. At what point (strength of the attack) does a customers netblock (assuming a /24 for example) get null routed by whichever party.
Most providers likely have a policy similar to: "I can't sacrafice 1 my network for 1 customer". So, if the attack is sufficient to degrade service on the ISP network most likely the customer under attack will get null routed.
Are you saying UUnet, assuming for a sec that I am a customer of UUnet (just for the sake of the argument), UU will not null route my ircd if it it gets attacked on regular basis, say *daily* ?
I did not say that.
Furthermore you are going to consistently place filters on your routers, take them out within the 24h (or whatever then-current policy of UUnet is) and track attacks back to their sources within the boundaries of your backbone on a daily basis? ;)
uhm... sure, we do this now... or have you not been paying attention?
Will you do that for say a regular T1 customer or do I need more "commitment" as sales droids like to put it, to even consider such a service ? ;)
read above.
Hmm, perhaps FIRST customers should insist that their ISP have some 24/7 security contact that can actually help in the case of an attack. Today there are very few that have this capability. I'd say from personal experience that the number is way too small, even in the 'large' ISP arena :(
More pressure from customers for real security would be a good start.
sigh, tried and failed, miserably I might add.
Then become a UUNET customer cause we already do this... Perhaps other providers with 24/7 security teams will pipe up to give potential customers a heads-up on options other than UUNET? If you go with UUNET please tell the sales driod I sent you cause then I get 50 bucks :) (my only raise thanks to bernie)
At 09:58 PM 01-05-02 -0400, Wojtek Zlobicki wrote: The ultimate goal of the DDOS attack is to take a specific user/site down. Blackholing is a way to help the attacker along. If the user is a small site, we say "screw it" and do the null0 in order to save the ISP backbone links. If the user is large (think eBay or any other major e-commerce site), you wouldn't easily blackhole them in order to save the rest of your network. You would try to find a better solution. Hank Consultant Riverhead Networks (formerly Wanwall Networks) www.riverhead.com
Then you are pushing out /32's and peers would need to accept them. Then someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!
I am in no way proposing discounting current filtering rules. There are alway two different intersts one must consider, one that of the customer and two that of the service provider. If a large block must be filtered so be it.
Where are providers drawing the line ? Anyone have somewhat detailed published policies as to what a provider can do in order to protect their nework as a whole. At what point (strength of the attack) does a customers netblock (assuming a /24 for example) get null routed by whichever party.
Anyways, some providers already allow you to set a community on a route, and they will inturn "blackhole" it for you. I believe Teleglobe does this for some customers and I know UUNet does this for all customers.
When the attack is distributed, having one or two providers (even if they are UUNET or Teleglobe) is just not enough. Must private routing policy be developed in order to make my suggestion work. The reason that so many methods likely fail are the difficulty of implementation and low implementation.
In a message written on Wed, May 01, 2002 at 08:17:04PM -0500, dies wrote:
Then you are pushing out /32's and peers would need to accept them. Then someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!
I'm not sure what form this would take, but I have long wished route processing could be sent into a "programming language". For this specific example it would be nice to set a maximum number of route limit for the total number of routes on the session, as well as /per community/. That is, community xxxx:666 == blackhole me, and I could limit each peer to say, 6 of these at a time. More would not take down the session, but simply be ignored. I can carry 6 /32's for every peer I have, and if they only have 6, they will probably use them for the most abusive target. There are, of course, approximately an infinitate number more applications for a more flexible mechanism. Of course, it would require more human smarts, which might be why vendors don't do it. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On Wed, May 01, 2002 at 10:15:44PM -0400, Leo Bicknell wrote:
In a message written on Wed, May 01, 2002 at 08:17:04PM -0500, dies wrote:
Then you are pushing out /32's and peers would need to accept them. Then someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!
I'm not sure what form this would take, but I have long wished route processing could be sent into a "programming language". For this specific example it would be nice to set a maximum number of route limit for the total number of routes on the session, as well as /per community/.
Agreed wholeheartedly. But then you'd have to have network engineers who could program (and no perl doesn't count). :)
That is, community xxxx:666 == blackhole me, and I could limit each peer to say, 6 of these at a time. More would not take down the session, but simply be ignored.
I can carry 6 /32's for every peer I have, and if they only have 6, they will probably use them for the most abusive target.
I give it 2 months, then they'll start hitting random dst IPs in a target prefix (say a common /24 going through the same path). -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Wed, 1 May 2002, dies wrote:
Then you are pushing out /32's and peers would need to accept them. Then someone will want to blackhole /30's, /29's, etc. Route bloat. Yum!
Yes.
Additionally you are creating a way to basically destroy the Internet as a whole. One kiddie gets ahold of a router, say of a large backbone provider, takes one of their aggregate blocks (/16? /10? /8?) and splits it into /32 announcements.
Or, blackhole the /16 :) more fun! (assuming no other smaller announcements inside that /16 of course)
Anyways, some providers already allow you to set a community on a route, and they will inturn "blackhole" it for you. I believe Teleglobe does this for some customers and I know UUNet does this for all customers.
Hmm, Mr. 'dies' is almost correct... if you are a UUNET customer and you would like to do this please call the customer service center and they will help you to configure this 'service'. Thanks though Mr. 'dies' :)
On Wed, 1 May 2002, Wojtek Zlobicki wrote:
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
A great deal of thought is being expended on this question, I am certain, however, how many of these thought campaings have born significant fruit yet, I do not know.
How about the following :
We develop a new community , being fully transitive (666 would be appropriate ) and either build into router code or create a route map to null route anything that contains this community. The effect of this being the distribution of the force of the attack.
This aside, how effective would be using a no export community with ones peers (being non transitive, it would still distribute the force of the attack).
On Wed, May 01, 2002 at 09:38:52PM -0400, Wojtek Zlobicki wrote:
How about the following :
We develop a new community , being fully transitive (666 would be appropriate ) and either build into router code or create a route map to null route anything that contains this community. The effect of this being the distribution of the force of the attack.
This has been proposed a dozen times over, and I agree that there should be a well known community for discarding packets. Go try and get the IETF to add it, let me know how it goes. :)
This aside, how effective would be using a no export community with ones peers (being non transitive, it would still distribute the force of the attack).
Many people do this already. If you're looking to purchase transit and you think this is something you'll care about, ask for it or vote with your wallet. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Wed, 1 May 2002, Wojtek Zlobicki wrote:
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
A great deal of thought is being expended on this question, I am certain, however, how many of these thought campaings have born significant fruit yet, I do not know.
How about the following :
We develop a new community , being fully transitive (666 would be appropriate ) and either build into router code or create a route map to null route anything that contains this community. The effect of this being the distribution of the force of the attack.
How about no. How about you do this inside YOUR network, perhaps get an agreement with your peers to accept a /32 route from you and you can do it with your peers also in times of need... There is something ominous about 'automagically propogating' a blackhole route. 1) I hack connected ISP X 2) I inject www.ebay.com /32 blackhole route 3) no more ebay I use ebay as an example of course, I wouldn't want them harmed cause how would I be able to buy all that nice routing gear at bargain basement prices without them? :)
This aside, how effective would be using a no export community with ones peers (being non transitive, it would still distribute the force of the attack).
For YOUR PEERS this is a fine idea, provided this fits with your peer's edge policies and doesn't step on his already-used community.
On Thu, 2 May 2002, Christopher L. Morrow wrote:
1) I hack connected ISP X 2) I inject www.ebay.com /32 blackhole route 3) no more ebay
I use ebay as an example of course, I wouldn't want them harmed cause how would I be able to buy all that nice routing gear at bargain basement prices without them? :)
Replace steps 2 and 3 with: 2) I route all packets going to Ebay to my box 3) I have my box to connect to real Ebay using passwords folks connecting to my man-in-the-middle box (how many of them have a clue to carefully look to the "SSL in use" icon anyway?) 4) I have the mershandise they bought shipped to me; and steal their CC numbers in the process. There are endless variations on the theme. Access to the routing infrastructure _MUST_ be tightly controlled. Intercepting traffic to root NSes is even more fun :) And, Satan bless the folks who want to let Unicode into DNS names, having many visually indistinguishable "ebay.com"s is a breeze, so one can get valid X.509 certificates for those undistinguishable "ebays", too. --vadim
Stoned koalas drooled eucalyptus spit in awe as Christopher L. Morrow exclaimed:
I use ebay as an example of course, I wouldn't want them harmed cause how would I be able to buy all that nice routing gear at bargain basement ^^^ Shouldn't this be s/I/WCOM/? :)
prices without them? :)
I suspect that they'll be buying all the Cicso memory they can find on there now, to support the route boat they've introduced over the past two days. WCOM 9:55am 2.425 +0.215 +9.73% 20,522,300 /me scrounges around in the couch cushions, looking for change to buy some WCOM. -J -- Jeff Workman | jworkman@pimpworks.org | http://www.pimpworks.org
On Wed, 1 May 2002 measl@mfn.org wrote:
and then again, there has been much discussion on simple DoS attacks, where the term DDoS is erroneously used... I am very much not trying to imply that this is the case here, but it's important that the two be thoroughly distinguished from each other - they are totally different things to deal with.
Sorry, I should have been more clear. My issue (currently) is not being the target of the DDoS attack, but being a (unwilling) participant. People outside our network are launching DDoS attacks (distributed SYN floods) against destinations outside our network, using about 8,000 Web server hosts on our network as reflectors. These are not zombies. They are secured, uncompromised Web servers. The attack spoofs the target address as the source, and one of our machines as a destination, port 80. Getting everyone to implement defenses (SYN cookies) on their Web servers is nearly impossible (most don't even have a defense--printers and routers with Web interfaces). SYN packet comes in, one of these machines responses with a RST to the "source", which is actually the target of the attack. Unfortunately, the target is often a site that people would like to get to, as is the reflector, so permanent filters on the target or reflector create lots of complaints.
We captured several seconds of the last DDoS and came up with over 700 participating hosts...
Some of them probably appear to be from our network... Pete.
On Wed, May 01, 2002 at 08:56:16PM -0600, Pete Kruckenberg wrote:
Sorry, I should have been more clear.
My issue (currently) is not being the target of the DDoS attack, but being a (unwilling) participant. People outside our network are launching DDoS attacks (distributed SYN floods) against destinations outside our network, using about 8,000 Web server hosts on our network as reflectors.
Neat, and totally not what people expect when you say "victim of a DDoS attack".
These are not zombies. They are secured, uncompromised Web servers. The attack spoofs the target address as the source, and one of our machines as a destination, port 80. Getting everyone to implement defenses (SYN cookies) on their Web servers is nearly impossible (most don't even have a defense--printers and routers with Web interfaces).
Thats not a defense anyways, SYN cookies still send replies (which is what the attacker wants), they just don't store state information (which is probably not an issue anyways, unless their stack is REALLY bad or old it's probably not going to care that much).
SYN packet comes in, one of these machines responses with a RST to the "source", which is actually the target of the attack. Unfortunately, the target is often a site that people would like to get to, as is the reflector, so permanent filters on the target or reflector create lots of complaints.
You have an interesting situation. I think rate limiting outbound RSTs would be the least offensive thing you could do, off the top of my head. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Thu, 2 May 2002, Richard A Steenbergen wrote:
SYN packet comes in, one of these machines responses with a RST to the "source", which is actually the target of the
You have an interesting situation. I think rate limiting outbound RSTs would be the least offensive thing you could do, off the top of my head.
What about just blocking out-going RSTs altogether from our borders? While this interferes with "proper" TCP functionality, would it actually interfere enough to cause noticeable problems? Would certainly be less of a burden on routers than rate-limiting. Pete.
On Wed, 1 May 2002, Pete Kruckenberg wrote:
On Thu, 2 May 2002, Richard A Steenbergen wrote:
SYN packet comes in, one of these machines responses with a RST to the "source", which is actually the target of the
You have an interesting situation. I think rate limiting outbound RSTs would be the least offensive thing you could do, off the top of my head.
What about just blocking out-going RSTs altogether from our borders? While this interferes with "proper" TCP functionality, would it actually interfere enough to cause noticeable problems? Would certainly be less of a burden on routers than rate-limiting.
Aren't the initial packets in the 'gibson syn amp attack' syn-ack's?
On Wed, May 01, 2002 at 11:56:07PM -0600, Pete Kruckenberg wrote:
On Thu, 2 May 2002, Richard A Steenbergen wrote:
You have an interesting situation. I think rate limiting outbound RSTs would be the least offensive thing you could do, off the top of my head.
What about just blocking out-going RSTs altogether from our borders? While this interferes with "proper" TCP functionality, would it actually interfere enough to cause noticeable problems? Would certainly be less of a burden on routers than rate-limiting.
If you really wanted to try you could probably get away with it, but you'll probably get complaints about broken behavior during "peacetime". I'd still advise a rate limit, say something on the order of 512Kbps or less depending on your pipe, and outbound TCP RST. If this makes your routers fall over, you need new routers. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Wed, 1 May 2002, Pete Kruckenberg wrote:
On Wed, 1 May 2002 measl@mfn.org wrote:
and then again, there has been much discussion on simple DoS attacks, where the term DDoS is erroneously used... I am very much not trying to imply that this is the case here, but it's important that the two be thoroughly distinguished from each other - they are totally different things to deal with.
Sorry, I should have been more clear.
My issue (currently) is not being the target of the DDoS attack, but being a (unwilling) participant. People outside our network are launching DDoS attacks (distributed SYN floods) against destinations outside our network, using about 8,000 Web server hosts on our network as reflectors.
Funny, you say 'secured' here...
These are not zombies. They are secured, uncompromised Web servers. The attack spoofs the target address as the source, and one of our machines as a destination, port 80. Getting everyone to implement defenses (SYN cookies) on their Web servers is nearly impossible (most don't even have a defense--printers and routers with Web interfaces).
and here you say: "printers and routers" Since when did they need to be accessible off campus? Additionally, why does a router need a web interface?? Printers are on the cusp, but they certainly don't need to be accesible from out of your LAN.
On Thu, 2 May 2002, Christopher L. Morrow wrote:
Funny, you say 'secured' here...
These are not zombies. They are secured, uncompromised Web servers. The attack spoofs the target address as the source, [snip] and here you say: "printers and routers" Since when did they need to be accessible off campus? Additionally, why does a router need a web interface?? Printers are on the cusp, but they certainly don't need to be accesible from out of your LAN.
More clarification needed. We are not a campus network. We are a state-wide research/education network, as in we are the service provider to the various K-12 and higher-ed institutions in the state (there is a network, not a purchasing cooperative like many other state "networks"). We are large in the sense that there are some 1,000 end sites (each comparable in size to a mid- to large-size enterprise) and a network that looks like many national networks, but condensed into a single state. We tend to design and operate our network, and experience problems similar to a national-scale network. Like almost every other service provider, we do not have the luxury of simply putting a firewall at the border of our network, since we do not have the ability to enforce security policies any more than other service providers do. We also have the ability to suggest security policy and block hosts or networks that interfere with network operations, but it's not our business whether someone uses a Web interface to their printer or router any more than it's UUNet's business. We do have a fairly aggressive security group that identifies compromised machines and assists customers in properly securing them. We can be fairly certain that the way these hosts are responding to this DoS attack is not as a result of being compromised, but a "normal" IP stack implementation. As such, though we are a state education service provider, it seems that these kinds of attacks are most likely pervasive on all networks, and probably are going on all the time. One advantage we have is a close relationship with our customers, which allows us to use tools such as IDS and Netflow in conjunction with information about the customer implementation to identify what is a bonafide attack. Pete.
On Wed, May 01, 2002 at 11:29:46PM -0600, Pete Kruckenberg wrote:
We do have a fairly aggressive security group that identifies compromised machines and assists customers in properly securing them. We can be fairly certain that the way these hosts are responding to this DoS attack is not as a result of being compromised, but a "normal" IP stack implementation.
Please please please please please tell me you are doing ingress filtering so the compromised boxes you host aren't spewing totally random source addresses on the internet. Not that it matters though, it's still pretty difficult to find the box in question. DDoS programs have been "auto-probing" for the best src address method to use for some time now (almost since their birth). For example, say a box is compromised on a network which does ingress filtering. The packet program detects this, and instead of randomizing the IP with every packet, it picks a single random IP by spoofing the last octet. In the interesting environments (like a college dorm network) this gets past most peoples ingress filters, since they're usually not exactly providing layer 3 all the way to the student. So when you send in a DoS complaint about 1.2.3.182, the campus computer nerd looks it up, and goes to knock on that persons door. Little do they know that the actual compromised machine is 1.2.3.97 spoofing it. You ever tried explaining this to the campus nerd? Not pretty! -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Wed, 1 May 2002, Pete Kruckenberg wrote:
On Thu, 2 May 2002, Christopher L. Morrow wrote:
Funny, you say 'secured' here...
These are not zombies. They are secured, uncompromised Web servers. The attack spoofs the target address as the source, [snip] and here you say: "printers and routers" Since when did they need to be accessible off campus? Additionally, why does a router need a web interface?? Printers are on the cusp, but they certainly don't need to be accesible from out of your LAN.
More clarification needed. We are not a campus network. We are a state-wide research/education network, as in we are the service provider to the various K-12 and higher-ed institutions in the state (there is a network, not a purchasing cooperative like many other state "networks").
This does complicate things, what about adding in some security provisions to your 'contract' ?? Or providing managed firewall services? Or better yet, reselling managed firewall services to your customers? :) There are ways, most times it just comes down to people at the far end not knowing enough to protect themselves, or not having the man power to fix it :(
We are large in the sense that there are some 1,000 end sites (each comparable in size to a mid- to large-size enterprise) and a network that looks like many national networks, but condensed into a single state. We tend to design and operate our network, and experience problems similar to a national-scale network.
Like almost every other service provider, we do not have the luxury of simply putting a firewall at the border of our network, since we do not have the ability to enforce security policies any more than other service providers do. We also have the ability to suggest security policy and block hosts or networks that interfere with network operations, but it's not our business whether someone uses a Web interface to their printer or router any more than it's UUNet's business.
Agreed, which is why we have resale and managed firewall businesses, so the customer can say what their security policy should be.
We do have a fairly aggressive security group that identifies compromised machines and assists customers in properly securing them. We can be fairly certain that the way these hosts are responding to this DoS attack is not as a result of being compromised, but a "normal" IP stack implementation.
'normal' to something that really has no business being accessible ;( but I agree with your point.
As such, though we are a state education service provider, it seems that these kinds of attacks are most likely pervasive on all networks, and probably are going on all the time. One advantage we have is a close relationship with our customers, which allows us to use tools such as IDS and Netflow in conjunction with information about the customer implementation to identify what is a bonafide attack.
On Wed, 1 May 2002 measl@mfn.org wrote:
True DDoS attacks, fortunately, are rarer than most people believe. If they were not, the Internet as we know it would look a lot more like a telephone system in USSR-at-it's-worst-days. For example, of the two recent DDoS's I have been on the receiving end of, the first was generating a little over 300mbit/sec (steady for a prolonged time), and the second went over that by a fair bit. In both cases, we had core equipment (M20's and BSN5000's) fall over and die trying to "work" the events. Additionally, our upstream peers
Your M20 tipped over?? What were you doing? We regularly stop large (+100Mb->800Mb) attacks with less horsepower than this. Truthfully, a cisco is even capable of filtering (done right) at +200kpps...
also had core equipment fall over, and we all came the [now obvious] conclusion that the only way to stop these attacks was to completely null route ourselves at our upstreams (they tried filter-fishing for specific data which may have helped our investigation, but when their routers started wheezing, we gave them the OK to just send us straight into the bit bucket till it was over...
Hmm, this highlights the need to learn how to use the equipment, learn its boundaries and learn defenses inside these boundaries... -Chris
chris@UU.NET disait :
have been on the receiving end of, the first was generating a little over 300mbit/sec (steady for a prolonged time), and the second went over that by a fair bit. In both cases, we had core equipment (M20's and BSN5000's) fall over and die trying to "work" the events. Additionally, our upstream peers
Your M20 tipped over?? What were you doing? We regularly stop large (+100Mb->800Mb) attacks with less horsepower than this. Truthfully, a cisco is even capable of filtering (done right) at +200kpps...
On Cisco boxes, it depends too much on Interface type, LC Engine, IOS, ... etc ... Beside, some features cannot run concurently (i remumber an ACL on GSR that make my netflow export stop .... it tooks days to figure this out !!!) ACL Implement on GSR is too a nightmare. We are operating more than 70 GSRs with very different interface, LC engine and IOS ... _some_ IOS with _some_ LC might truthfully filter (turbo, extended, vanilla, in, out ACLs ?!) .... but there is too many variable in the equation to get ops people use it for massive anti-DOS purpose ! Vincent.
On Thu, 2 May 2002, Vincent Gillet wrote:
chris@UU.NET disait :
have been on the receiving end of, the first was generating a little over 300mbit/sec (steady for a prolonged time), and the second went over that by a fair bit. In both cases, we had core equipment (M20's and BSN5000's) fall over and die trying to "work" the events. Additionally, our upstream peers
Your M20 tipped over?? What were you doing? We regularly stop large (+100Mb->800Mb) attacks with less horsepower than this. Truthfully, a cisco is even capable of filtering (done right) at +200kpps...
On Cisco boxes, it depends too much on Interface type, LC Engine, IOS, ... etc ...
In this you have my whole-hearted agreement :( But, this goes back to 'know you systems, know their boundaries'. All of the people that work here (on our team) know what you can and can't do, we are effective in our jobs because of this. Sure your random NOC worker or even level3/4 NOC worker isn't going to know all the ins and outs of security thingy's on your backbone equipemt, that's why you pay 5-7 people to learn it :)
Beside, some features cannot run concurently (i remumber an ACL on GSR that make my netflow export stop .... it tooks days to figure this out !!!)
Ha! :) try acl's on engine-2 cards with sub-interfaces! (like the triton gig card... cause no one would ever sub-interface a gig interface, right?)
ACL Implement on GSR is too a nightmare. We are operating more than 70 GSRs with very different interface, LC engine and IOS ...
Just 70? your live is easy then :) Really though, this is, in my opinion, the larges problem Cisco has to over come. They need to have the 'luxury' that Juniper has: One IOS, One implementation of commands, same commands everywhere... consistency I believe its called. Its not, obviously, going to happen overnight, but to their credit folks at cisco ARE working to make the security problem less of a problem. If you are having trouble getting your sales folk from cisco to listen/understand/pass-along-input you can look for their 'ISP Group' which I'm sure Barry Greene will be happy to properly name and provide contacts for, or perhaps they are in the sites he posted here before?
On Thu, 2 May 2002, Christopher L. Morrow wrote:
On Wed, 1 May 2002 measl@mfn.org wrote:
True DDoS attacks, fortunately, are rarer than most people believe. If they were not, the Internet as we know it would look a lot more like a telephone system in USSR-at-it's-worst-days. For example, of the two recent DDoS's I have been on the receiving end of, the first was generating a little over 300mbit/sec (steady for a prolonged time), and the second went over that by a fair bit. In both cases, we had core equipment (M20's and BSN5000's) fall over and die trying to "work" the events. Additionally, our upstream peers
Your M20 tipped over?? What were you doing? We regularly stop large (+100Mb->800Mb) attacks with less horsepower than this. Truthfully, a cisco is even capable of filtering (done right) at +200kpps...
I'm sorry, I was not clear here... The M20 does great at simply pushing this load to discard, but the overhead of what we were trying to do (extensive filter lists to try and begin backtracing the actual skr1pt k1dd13 origin) was too much. There is simply no good way to get back to the ultimate source of truly distributed DoS attacks, which is, IMHO, the reason these attacks are so prevalent - no fear of prosecution, no matter how much collateral damage is inflicted.
also had core equipment fall over, and we all came the [now obvious] conclusion that the only way to stop these attacks was to completely null route ourselves at our upstreams (they tried filter-fishing for specific data which may have helped our investigation, but when their routers started wheezing, we gave them the OK to just send us straight into the bit bucket till it was over...
Hmm, this highlights the need to learn how to use the equipment, learn its boundaries and learn defenses inside these boundaries...
In the larger picture, my concern is with finding the source, so I can prevent recurrence - a paradoxical problem considering that the short term goal is to just stop the attack...
-Chris
-- Yours, J.A. Terranson sysadmin@mfn.org If Governments really want us to behave like civilized human beings, they should give serious consideration towards setting a better example: Ruling by force, rather than consensus; the unrestrained application of unjust laws (which the victim-populations were never allowed input on in the first place); the State policy of justice only for the rich and elected; the intentional abuse and occassionally destruction of entire populations merely to distract an already apathetic and numb electorate... This type of demogoguery must surely wipe out the fascist United States as surely as it wiped out the fascist Union of Soviet Socialist Republics. The views expressed here are mine, and NOT those of my employers, associates, or others. Besides, if it *were* the opinion of all of those people, I doubt there would be a problem to bitch about in the first place... --------------------------------------------------------------------
What we use and we're a 'largeish' network: http://www.secsup.org/Tracking/ (shameless plug #1) Among other things this is a tool we use... there was a great set of slides and presentation given at NANOG23: http://www.nanog.org/mtg-0110/greene.html (shameless plug #2) There is also a set of papers Barry Greene from Cisco has available on the Cisco website... I'm positive he'll respond to this with the link, if he doesn't search the NANOG mailing list archive for the link it should be obvious in posts from Barry. If you want more pointers I'd be glad to chat on the phone with you, numbers included below. --Chris (chris@uu.net) ####################################################### ## UUNET Technologies, Inc. ## ## Manager ## ## Customer Router Security Engineering Team ## ## (W)703-886-3823 (C)703-338-7319 ## ####################################################### On Wed, 1 May 2002, Pete Kruckenberg wrote:
There's been plenty of discussion about DDoS attacks, and my IDS system is darn good at identifying them. But what are effective methods for large service-provider networks (ie ones where a firewall at the front would not be possible) to deal with DDoS attacks?
Current method of updating ACLs with the source and/or destination are slow and error-prone and hard to maintain (especially when the target of the attack is a site that users would like to access).
A rather extensive survey of DDoS papers has not resulted in much on this topic.
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
Thanks. Pete.
On Wed, May 01, 2002 at 05:18:24PM -0600, pete@kruckenberg.com said: [snip]
A rather extensive survey of DDoS papers has not resulted in much on this topic.
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
It seems to me that the real issue in defending against an attack of this type of differentiating between legitimate traffic and zombie traffic. This seems to be self-evident, but on a distributed scale, how _would_ one tell the difference between a host/netblock that's making a lot of requests to a busy site (amazon.com, say) and a host/netblock that's sending a lot of zombie requests, especially when both sets of requests are bound for the same ports (80/443 in this case) on the same IP/set of IPs? The more D the DoS, the more difficult it becomes to tell what's legit and what's not. (Stating the obvious again, I know, but it helps me think. :) ) -- Scott Francis darkuncle@ [home:] d a r k u n c l e . n e t Systems/Network Manager sfrancis@ [work:] t o n o s . c o m GPG public key 0xCB33CCA7 illum oportet crescere me autem minui
On Wed, 1 May 2002, Pete Kruckenberg wrote:
There's been plenty of discussion about DDoS attacks, and my IDS system is darn good at identifying them. But what are effective methods for large service-provider networks (ie ones where a firewall at the front would not be possible) to deal with DDoS attacks?
I'm working on something that should provide a solution to this for at least some subset of all attacks. Basically, it works like this: when you identify the target of the attack, you have traffic for those target addresses rerouted to a "filter box". This filter box then contains source address based filters to get rid of the attacking traffic. The idea is that a service provider could install one or more of those filter boxes (standard routers or multilayer switches) and have customers use standard BGP mechanisms to get the filter boxes to clean up the traffic. This should work as long as the number of source addresses is relatively limited, say below 20,000. If anyone is interested in testing such a setup in a real network, contact me off-list. My goal is to evaluate how well this works and then write up an article for the benefit of the networking community. Iljitsch van Beijnum
On Thu, 2 May 2002, Iljitsch van Beijnum wrote:
Basically, it works like this: when you identify the target of the attack, you have traffic for those target addresses rerouted to a "filter box". This filter box then contains source address based filters to get rid of the attacking traffic.
Two questions: 1) How do you plan on determining what an allowed src address and what isn't? 2) Secondly, how would you deal with spoofed src addresses where the src address is rarely repeated in the attack?
On Thu, 2 May 2002, Avleen Vig wrote:
Basically, it works like this: when you identify the target of the attack, you have traffic for those target addresses rerouted to a "filter box". This filter box then contains source address based filters to get rid of the attacking traffic.
Two questions: 1) How do you plan on determining what an allowed src address and what isn't?
"allowed"?
2) Secondly, how would you deal with spoofed src addresses where the src address is rarely repeated in the attack?
If that is the case, this solution won't help. Unfortunately, it is impossilbe to prevent traffic with spoofed source addresses to come in over transit connections. However, it is doable to make sure traffic coming in from peers uses source addresses that belong to peers. So for networks large enough to have a major part of their traffic coming in over peering rather than transit, there are possibilities.
On Thu, 2 May 2002, Iljitsch van Beijnum wrote:
On Wed, 1 May 2002, Pete Kruckenberg wrote:
There's been plenty of discussion about DDoS attacks, and my IDS system is darn good at identifying them. But what are effective methods for large service-provider networks (ie ones where a firewall at the front would not be possible) to deal with DDoS attacks?
I'm working on something that should provide a solution to this for at least some subset of all attacks.
Basically, it works like this: when you identify the target of the attack, you have traffic for those target addresses rerouted to a "filter box". This filter box then contains source address based filters to get rid of the attacking traffic.
The idea is that a service provider could install one or more of those filter boxes (standard routers or multilayer switches) and have customers use standard BGP mechanisms to get the filter boxes to clean up the traffic. This should work as long as the number of source addresses is relatively limited, say below 20,000.
Congrats on re-inventing the wheel :( This is what mazuu/arbor/wanwall(riverhead now?) all do... this is also the way CenterTrack(tm robert stone) was kind of supposed to work. As near as I can tell this doesn't scale too well in a large network. This is a shame, but its a reality. Additionally 20k sources max? that's not nearly enough, how many addresses are in 0/0 ? you should atleast plan for this contingency...
On Thu, 2 May 2002, Christopher L. Morrow wrote:
Congrats on re-inventing the wheel :( This is what mazuu/arbor/wanwall(riverhead now?) all do... this is also the way CenterTrack(tm robert stone) was kind of supposed to work.
Thanks for the kind works. Just to be clear: I'm not working on a _product_, just on a paper explaining how to do this using standard components and protocols.
As near as I can tell this doesn't scale too well in a large network.
If you have a router that can forward 10 Gbps into the right direction, you can also have a router forward 10 Gbps in the wrong direction. That's pretty much all it takes.
This is a shame, but its a reality. Additionally 20k sources max? that's not nearly enough, how many addresses are in 0/0 ? you should atleast plan for this contingency...
The idea is to use unicast RPF. So you're only limited by the number of routes a Cisco can hold. 20k per customer under attack should be doable without too much effort, more should be possible, but filtering 0/0 defeats the purpose. Also, it can be done using a single line, so no problem there.
On the subject of uRPF, I thought I should point out that Juniper just added support for it in JunOS 5.3. The news seems to have been obscured by the T640, but I think its a pretty big deal. One less excuse for the people who still aren't RPF filtering their customers (you know who you are). Go forth and be filterful. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks?
What we are using is matching of a specific community on all of our edge routers. A route matching this specific community will be blackholed on the edge. All that is then needed is by our NOC or one of our customers to announce the host under attack as a /32 with the right community and they will not suffer under the attack. Problem then is to get the router to drop all the packets.... - kurtis -
participants (20)
-
Alexei Roudnev -
Avleen Vig -
Basil Kruglov -
Chris Adams -
Christopher L. Morrow -
dies -
Hank Nussbacher -
Iljitsch van Beijnum -
Jeff Workman -
Kurt Erik Lindqvist -
Leo Bicknell -
measl＠mfn.org -
Pete Kruckenberg -
Ralph Doncaster -
Richard A Steenbergen -
Scott Francis -
Sean Donelan -
Vadim Antonov -
Vincent Gillet -
Wojtek Zlobicki