Re: Over a decade of DDOS--any progress yet?
----Mensaje original---- De: rdobbins@arbor.net Fecha: 08/12/2010 10:53 Para: "North American Operators' Group"<nanog@nanog.org> Asunto: Re: Over a decade of DDOS--any progress yet?
On Dec 8, 2010, at 7:28 PM, Arturo Servin wrote:
One big problem (IMHO) of DDoS is that sources (the host of botnets) may be completely unaware that they are part of a DDoS. I do not mean the bot machine, I mean the ISP connecting those.
The technology exists to detect and classify this attack traffic, and is deployed in production networks today.
And of course, the legitimate owners of the botted hosts are generally unaware that their machine is being used for nefarious
A very common action is to blackhole ddos traffic upstream by sending a bgp route to the next AS with a preestablished community indicating the traffic must be sent to Null0. The route may be very specific, in order to impact as less as possible. This needs previous coordination between providers. Regards. purposes.
In the other hand the target of a DDoS cannot do anything to stop
to attack besides adding more BW or contacting one by one the whole path of providers to try to minimize the effect.
Actually, there're lots of things they can do.
I know that this has many security concerns, but would it be good
a signalling protocol between ISPs to inform the sources of a DDoS attack in order to take semiautomatic actions to rate-limit the traffic as close as the source? Of course that this is more complex that these three or two lines, but I wonder if this has been considerer in the past.
It already exists.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
----Mensaje original---- De: rdobbins@arbor.net Fecha: 08/12/2010 10:53 Para: "North American Operators' Group"<nanog@nanog.org> Asunto: Re: Over a decade of DDOS--any progress yet?
On Dec 8, 2010, at 7:28 PM, Arturo Servin wrote:
One big problem (IMHO) of DDoS is that sources (the host of botnets) may be completely unaware that they are part of a DDoS. I do not mean the bot machine, I mean the ISP connecting those.
The technology exists to detect and classify this attack traffic, and is deployed in production networks today.
And of course, the legitimate owners of the botted hosts are generally unaware that their machine is being used for nefarious
Yes, but this obviously completes the 'DDoS attack' and sends the signal that the bully will win. -Drew -----Original Message----- From: alvaro.sanchez@adinet.com.uy [mailto:alvaro.sanchez@adinet.com.uy] Sent: Wednesday, December 08, 2010 8:46 AM To: rdobbins@arbor.net; North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet? A very common action is to blackhole ddos traffic upstream by sending a bgp route to the next AS with a preestablished community indicating the traffic must be sent to Null0. The route may be very specific, in order to impact as less as possible. This needs previous coordination between providers. Regards. purposes.
In the other hand the target of a DDoS cannot do anything to stop
to attack besides adding more BW or contacting one by one the whole path of providers to try to minimize the effect.
Actually, there're lots of things they can do.
I know that this has many security concerns, but would it be good
a signalling protocol between ISPs to inform the sources of a DDoS attack in order to take semiautomatic actions to rate-limit the traffic as close as the source? Of course that this is more complex that these three or two lines, but I wonder if this has been considerer in the past.
It already exists.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
+1 On Wed, Dec 8, 2010 at 10:30 AM, Drew Weaver <drew.weaver@thenap.com> wrote:
Yes, but this obviously completes the 'DDoS attack' and sends the signal that the bully will win.
-Drew
-----Original Message----- From: alvaro.sanchez@adinet.com.uy [mailto:alvaro.sanchez@adinet.com.uy] Sent: Wednesday, December 08, 2010 8:46 AM To: rdobbins@arbor.net; North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet?
A very common action is to blackhole ddos traffic upstream by sending a bgp route to the next AS with a preestablished community indicating the traffic must be sent to Null0. The route may be very specific, in order to impact as less as possible. This needs previous coordination between providers. Regards.
----Mensaje original---- De: rdobbins@arbor.net Fecha: 08/12/2010 10:53 Para: "North American Operators' Group"<nanog@nanog.org> Asunto: Re: Over a decade of DDOS--any progress yet?
On Dec 8, 2010, at 7:28 PM, Arturo Servin wrote:
One big problem (IMHO) of DDoS is that sources (the host of botnets) may be completely unaware that they are part of a DDoS. I do not mean the bot machine, I mean the ISP connecting those.
The technology exists to detect and classify this attack traffic, and is deployed in production networks today.
And of course, the legitimate owners of the botted hosts are generally unaware that their machine is being used for nefarious purposes.
In the other hand the target of a DDoS cannot do anything to stop to attack besides adding more BW or contacting one by one the whole path of providers to try to minimize the effect.
Actually, there're lots of things they can do.
I know that this has many security concerns, but would it be good a signalling protocol between ISPs to inform the sources of a DDoS attack in order to take semiautomatic actions to rate-limit the traffic as close as the source? Of course that this is more complex that these three or two lines, but I wonder if this has been considerer in the past.
It already exists.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
On 12/8/10 6:30 AM, Drew Weaver wrote:
Yes, but this obviously completes the 'DDoS attack' and sends the signal that the bully will win.
it's part of a valid mitigation strategy. shifting the target out from underneath the blackholed address is also part of the activity. that's easier in some cases than others. the bots will move and you play whack a rat with your upstreams. joel
-Drew
From: alvaro.sanchez@adinet.com.uy [mailto:alvaro.sanchez@adinet.com.uy] Sent: Wednesday, December 08, 2010 8:46 AM To: rdobbins@arbor.net; North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet?
A very common action is to blackhole ddos traffic upstream by sending a bgp route to the next AS with a preestablished community indicating the traffic must be sent to Null0. The route may be very specific, in order to impact as less as possible. This needs previous coordination between providers. Regards.
The NIST has proposed a framework for operators to notify botnet victims. The call for comments and article discussing it are described here: https://www.infosecisland.com/blogview/17021-Government-Proposes-ISPs-Notif y-Victims-of-Botnets.html#.TotXA6C-16Q.twitter "Comments on the proposed Code of Conduct and botnet reporting initiative are due on or before 5 p.m. EDT, November 4, 2011. Written comments on the proposal may be submitted by mail to the National Institute of Standards and Technology at the U.S. Department of Commerce, 1401 Constitution Avenue, NW., Room 4822, Washington, DC 20230. Submissions may be in any of the following formats: HTML, ASCII, Word, rtf, or pdf. Online comment submissions in electronic form may be sent to Consumer_Notice_RFI@nist.gov. Paper submissions should include a compact disc (CD). CDs should be labeled with the name and organizational affiliation of the filer and the name of the word processing program used to create the document. Comments will be posted at http://www.nist.gov/itl/. A list of questions are included in the Request for Information, and can be accessed at the source link below: Source: http://www.federalregister.gov/articles/2011/09/21/2011-24180/models-to-adv ance-voluntary-corporate-notification-to-consumers-regarding-the-illicit-us e-of#p-3 <http://www.federalregister.gov/articles/2011/09/21/2011-24180/models-to-ad vance-voluntary-corporate-notification-to-consumers-regarding-the-illicit-u se-of#p-3> " IMHO this would go a long way to addressing the underlying root cause (botted machines). Regards, Zachary On 12/14/10 5:34 PM, "Joel Jaeggli" <joelja@bogus.com> wrote:
On 12/8/10 6:30 AM, Drew Weaver wrote:
Yes, but this obviously completes the 'DDoS attack' and sends the signal that the bully will win.
it's part of a valid mitigation strategy. shifting the target out from underneath the blackholed address is also part of the activity. that's easier in some cases than others. the bots will move and you play whack a rat with your upstreams.
joel
-Drew
From: alvaro.sanchez@adinet.com.uy [mailto:alvaro.sanchez@adinet.com.uy] Sent: Wednesday, December 08, 2010 8:46 AM To: rdobbins@arbor.net; North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet?
A very common action is to blackhole ddos traffic upstream by sending a bgp route to the next AS with a preestablished community indicating the traffic must be sent to Null0. The route may be very specific, in order to impact as less as possible. This needs previous coordination between providers. Regards.
A less common action is to use flowspec (if you have some Juniper gear) to drop only the attack and hopefully not any legitimate traffic. What is really missing atm is a way to filter flowspec announcements (limit the number and make sure they are for routes the peer is announcing). Until this is sorted I believe flowspec will be a marginal solution. Thomas PLUG: http://code.google.com/p/exabgp/ On 8 Dec 2010, at 13:46, alvaro.sanchez@adinet.com.uy wrote:
A very common action is to blackhole ddos traffic upstream by sending a bgp route to the next AS with a preestablished community indicating the traffic must be sent to Null0. The route may be very specific, in order to impact as less as possible. This needs previous coordination between providers. Regards.
----Mensaje original---- De: rdobbins@arbor.net Fecha: 08/12/2010 10:53 Para: "North American Operators' Group"<nanog@nanog.org> Asunto: Re: Over a decade of DDOS--any progress yet?
On Dec 8, 2010, at 7:28 PM, Arturo Servin wrote:
One big problem (IMHO) of DDoS is that sources (the host of botnets) may be completely unaware that they are part of a DDoS. I do not mean the bot machine, I mean the ISP connecting those.
The technology exists to detect and classify this attack traffic, and is deployed in production networks today.
And of course, the legitimate owners of the botted hosts are generally unaware that their machine is being used for nefarious purposes.
In the other hand the target of a DDoS cannot do anything to stop to attack besides adding more BW or contacting one by one the whole path of providers to try to minimize the effect.
Actually, there're lots of things they can do.
I know that this has many security concerns, but would it be good a signalling protocol between ISPs to inform the sources of a DDoS attack in order to take semiautomatic actions to rate-limit the traffic as close as the source? Of course that this is more complex that these three or two lines, but I wonder if this has been considerer in the past.
It already exists.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
On Dec 8, 2010, at 10:10 PM, Thomas Mangin wrote:
Until this is sorted I believe flowspec will be a marginal solution.
We're seeing a significant uptick in flowspec interest, actually, and S/RTBH has been around for ages. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
On 8 Dec 2010, at 15:12, Dobbins, Roland wrote:
On Dec 8, 2010, at 10:10 PM, Thomas Mangin wrote:
Until this is sorted I believe flowspec will be a marginal solution.
We're seeing a significant uptick in flowspec interest, actually, and S/RTBH has been around for ages.
Great to hear :) But my point is still valid, Flowspec is great if you are are a backbone and are performing the filtering, or if you want to filter outgoing traffic. If you are a smaller network, you need the filtering to be performed by your transit provider, as your uplink will otherwise be congested. So I will stand by my comment that flowspec would see a bigger uptake if T1 could accept the flowspec routes, which they will only do once they can filter them (to insure correctness and resource protection). Thomas PS : Someone need to add IPv6 support to the RFC :p
On Dec 8, 2010, at 10:36 PM, Thomas Mangin wrote:
If you are a smaller network, you need the filtering to be performed by your transit provider, as your uplink will otherwise be congested.
Actually, most DDoS attacks aren't link-flooding attacks - this hasn't been true for the last ~7 years or so. I'm not saying it doesn't happen, because it does, and sometimes quite spectacularly - but in most cases, the attackers don't have to flood the link to achieve their desired goal. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
I would say that > 99% of the attacks that we see are 'link fillers' with < 1% being an application attack. thanks, -Drew -----Original Message----- From: Dobbins, Roland [mailto:rdobbins@arbor.net] Sent: Wednesday, December 08, 2010 10:41 AM To: North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet? On Dec 8, 2010, at 10:36 PM, Thomas Mangin wrote:
If you are a smaller network, you need the filtering to be performed by your transit provider, as your uplink will otherwise be congested.
Actually, most DDoS attacks aren't link-flooding attacks - this hasn't been true for the last ~7 years or so. I'm not saying it doesn't happen, because it does, and sometimes quite spectacularly - but in most cases, the attackers don't have to flood the link to achieve their desired goal. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
On Dec 8, 2010, at 11:14 PM, Drew Weaver wrote:
I would say that > 99% of the attacks that we see are 'link fillers' with < 1% being an application attack.
Application-layer attacks aside, most packet-flooding attacks these days don't completely fill links, as there's no need for the attacker to do so. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
On 12/8/2010 10:28 AM, Dobbins, Roland wrote:
Application-layer attacks aside, most packet-flooding attacks these days don't completely fill links, as there's no need for the attacker to do so.
I think the difference here is scale. packet-flooding attacks often do fill links; if the links drop to 155mb/s or below. I've seen some gig+ DOS, but that is less common. The DOS I posted a flow capture link for wasn't that large, but enough to flood out the little DS3 going to the small town where the target DSL customers was. Jack
< 1 Gbps attacks used to be standard issue but as of the past 90 days we have been seeing 2 - 8 Gbps a lot more frequently. Jeff On Wed, Dec 8, 2010 at 11:38 AM, Jack Bates <jbates@brightok.net> wrote:
On 12/8/2010 10:28 AM, Dobbins, Roland wrote:
Application-layer attacks aside, most packet-flooding attacks these days don't completely fill links, as there's no need for the attacker to do so.
I think the difference here is scale. packet-flooding attacks often do fill links; if the links drop to 155mb/s or below. I've seen some gig+ DOS, but that is less common. The DOS I posted a flow capture link for wasn't that large, but enough to flood out the little DS3 going to the small town where the target DSL customers was.
Jack
-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions
On 12/8/2010 10:41 AM, Jeffrey Lyon wrote:
< 1 Gbps attacks used to be standard issue but as of the past 90 days we have been seeing 2 - 8 Gbps a lot more frequently.
That may well be true. I'm an eyeball network and I can usually point at a user pissing someone off on IRC/Forums for DOS instigating. I probably deal with 1 large scale attack per year at most, though most likely my attacks are from smaller botnet owners. Jack
You can get a dedicated server for $80 with a 1Gbps connection to the Internet without looking that hard. It is pretty easy/cheap to kill a 1Gbps connection now a days. Soon several providers will begin offering dedicated servers with a 10Gbps connection to a single machine. -Drew -----Original Message----- From: Jeffrey Lyon [mailto:jeffrey.lyon@blacklotus.net] Sent: Wednesday, December 08, 2010 11:42 AM To: Jack Bates Cc: North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet? < 1 Gbps attacks used to be standard issue but as of the past 90 days we have been seeing 2 - 8 Gbps a lot more frequently. Jeff On Wed, Dec 8, 2010 at 11:38 AM, Jack Bates <jbates@brightok.net> wrote:
On 12/8/2010 10:28 AM, Dobbins, Roland wrote:
Application-layer attacks aside, most packet-flooding attacks these days don't completely fill links, as there's no need for the attacker to do so.
I think the difference here is scale. packet-flooding attacks often do fill links; if the links drop to 155mb/s or below. I've seen some gig+ DOS, but that is less common. The DOS I posted a flow capture link for wasn't that large, but enough to flood out the little DS3 going to the small town where the target DSL customers was.
Jack
-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions
Yes, and I have no problem with this in theory, I just wish that some of the larger ones could proactively monitor their networks to avoid crushing the smaller ones but maybe this is intentional. I have seen a huge increase in the number of attacks originating from other "hosting" companies recently. Previously it had mainly been cable modems, etc. It must be much easier to just target IaaS providers to build botnets because each machine there has 1Gbps than to worry about collecting 100 10Mbps cable modem customers. -Drew -----Original Message----- From: Randy McAnally [mailto:rsm@fast-serv.com] Sent: Wednesday, December 08, 2010 11:59 AM To: Drew Weaver; 'Jeffrey Lyon'; Jack Bates Cc: North American Operators' Group Subject: RE: Over a decade of DDOS--any progress yet?
Soon several providers will begin offering dedicated servers with a 10Gbps connection to a single machine.
-Drew
Several already do. -Randy
On Dec 8, 2010, at 11:38 PM, Jack Bates wrote:
I think the difference here is scale. packet-flooding attacks often do fill links; if the links drop to 155mb/s or below.
I'm not saying that link-flooding attacks don't happen; they certainly do, and on very big links, sometimes. But in the scheme of things, they don't happen nearly as often as they used to, as the attackers simply don't need to fill the links in order to accomplish their goals, in most cases. It's also important to note that a lot of DDoS isn't directly perpetrated by those who wish the DDoS performed, but rather is hired out to botmasters who're paid to execute the attacks. Even if the person who is the motivating force behind the attack is paying in stolen credit cards or whatever, he doesn't want to pay for more than is needed to accomplish his goal. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
On 08/12/2010 16:14, Drew Weaver wrote:
I would say that > 99% of the attacks that we see are 'link fillers' with < 1% being an application attack.
thanks, -Drew
This has been our recent experience as well. There are some pure app attacks, to be sure, but we many blended attacks also. Bandwidth (UDP/ICMP/SYN Flood) attack to distract with a app attack (GET/PUSH floods) attempting to run underneath the radar. We regularly see SYN floods these days > 20 Gb/s. The thing to bear in mind is that app attacks *are* difficult to detect as they are low bandwidth and make a full TCP connection. As a result many IDS/Firewalls etc regularly miss these attacks. Lastly there is usually always someone at the other end of these attacks watching what is working and what is not. If the attack doesn't work they will simply round up more bots to increase the attack bandwidth or change the attack vector. Best, --J --- Jay Coley Prolexic Technologies
On Dec 8, 2010, at 11:47 PM, Jay Coley wrote:
This has been our recent experience as well.
I see a link-filling attacks with some regularity; but again, what I'm saying is simply that they aren't as prevalent as they used to be, because the attackers don't *need* to fill links in order to achieve their goals, in many cases. That being said, high-bandwidth DNS reflection/amplification attacks tip the scales, every time.
Lastly there is usually always someone at the other end of these attacks watching what is working and what is not
This is a very important point - determined attackers will observe and react in order to try and defeat successful countermeasures, so the defenders must watch for shifting attack vectors. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
Nobody has really driven the point home that yes you can purchase a system from Arbor, RioRey, make your own mitigation system; what-have you, but you still have to pay for the transit to digest the attack, which is probably the main cost right now. -Drew -----Original Message----- From: Dobbins, Roland [mailto:rdobbins@arbor.net] Sent: Wednesday, December 08, 2010 11:54 AM To: North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet? On Dec 8, 2010, at 11:47 PM, Jay Coley wrote:
This has been our recent experience as well.
I see a link-filling attacks with some regularity; but again, what I'm saying is simply that they aren't as prevalent as they used to be, because the attackers don't *need* to fill links in order to achieve their goals, in many cases. That being said, high-bandwidth DNS reflection/amplification attacks tip the scales, every time.
Lastly there is usually always someone at the other end of these attacks watching what is working and what is not
This is a very important point - determined attackers will observe and react in order to try and defeat successful countermeasures, so the defenders must watch for shifting attack vectors. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
On 12/10/10 12:33 PM, Drew Weaver wrote:
Nobody has really driven the point home that yes you can purchase a system from Arbor, RioRey, make your own mitigation system; what-have you, but you still have to pay for the transit to digest the attack, which is probably the main cost right now.
or you outsource it and it's still costlier. Paying for DOS mitigation you rarely if ever use is quite expensive. If you use it a lot it's even more expensive, but can at least be rationalized on the basis of known costs e.g. npv calculation on the number and duration of outages...
-Drew
-----Original Message----- From: Dobbins, Roland [mailto:rdobbins@arbor.net] Sent: Wednesday, December 08, 2010 11:54 AM To: North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet?
On Dec 8, 2010, at 11:47 PM, Jay Coley wrote:
This has been our recent experience as well.
I see a link-filling attacks with some regularity; but again, what I'm saying is simply that they aren't as prevalent as they used to be, because the attackers don't *need* to fill links in order to achieve their goals, in many cases.
That being said, high-bandwidth DNS reflection/amplification attacks tip the scales, every time.
Lastly there is usually always someone at the other end of these attacks watching what is working and what is not
This is a very important point - determined attackers will observe and react in order to try and defeat successful countermeasures, so the defenders must watch for shifting attack vectors.
-----------------------------------------------------------------------
Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
On Dec 11, 2010, at 5:51 AM, Joel Jaeggli wrote:
Paying for DOS mitigation you rarely if ever use is quite expensive.
Some operators offer 'Clean Pipes' commercial DDoS mitigation services; they have various fee models, and they charge their end-customers for it. It's positioned as a form of insurance, for the end-customer. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
On Fri, Dec 10, 2010 at 5:51 PM, Joel Jaeggli <joelja@bogus.com> wrote:
On 12/10/10 12:33 PM, Drew Weaver wrote:
Nobody has really driven the point home that yes you can purchase a system from Arbor, RioRey, make your own mitigation system; what-have you, but you still have to pay for the transit to digest the attack, which is probably the main cost right now.
or you outsource it and it's still costlier.
Paying for DOS mitigation you rarely if ever use is quite expensive. If you use it a lot it's even more expensive, but can at least be rationalized on the basis of known costs e.g. npv calculation on the number and duration of outages...
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts) -chris
-Drew
-----Original Message----- From: Dobbins, Roland [mailto:rdobbins@arbor.net] Sent: Wednesday, December 08, 2010 11:54 AM To: North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet?
On Dec 8, 2010, at 11:47 PM, Jay Coley wrote:
This has been our recent experience as well.
I see a link-filling attacks with some regularity; but again, what I'm saying is simply that they aren't as prevalent as they used to be, because the attackers don't *need* to fill links in order to achieve their goals, in many cases.
That being said, high-bandwidth DNS reflection/amplification attacks tip the scales, every time.
Lastly there is usually always someone at the other end of these attacks watching what is working and what is not
This is a very important point - determined attackers will observe and react in order to try and defeat successful countermeasures, so the defenders must watch for shifting attack vectors.
-----------------------------------------------------------------------
Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
I'm certain there are thresholds to that. Carrier grade mitigation solutions will start low and ramp up to 5, 6, 7, etc. figures depending on the attack and amount of bandwidth to be filtered among other variables. Jeff On Sun, Dec 12, 2010 at 12:05 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Fri, Dec 10, 2010 at 5:51 PM, Joel Jaeggli <joelja@bogus.com> wrote:
On 12/10/10 12:33 PM, Drew Weaver wrote:
Nobody has really driven the point home that yes you can purchase a system from Arbor, RioRey, make your own mitigation system; what-have you, but you still have to pay for the transit to digest the attack, which is probably the main cost right now.
or you outsource it and it's still costlier.
Paying for DOS mitigation you rarely if ever use is quite expensive. If you use it a lot it's even more expensive, but can at least be rationalized on the basis of known costs e.g. npv calculation on the number and duration of outages...
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts)
-chris
-Drew
-----Original Message----- From: Dobbins, Roland [mailto:rdobbins@arbor.net] Sent: Wednesday, December 08, 2010 11:54 AM To: North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet?
On Dec 8, 2010, at 11:47 PM, Jay Coley wrote:
This has been our recent experience as well.
I see a link-filling attacks with some regularity; but again, what I'm saying is simply that they aren't as prevalent as they used to be, because the attackers don't *need* to fill links in order to achieve their goals, in many cases.
That being said, high-bandwidth DNS reflection/amplification attacks tip the scales, every time.
Lastly there is usually always someone at the other end of these attacks watching what is working and what is not
This is a very important point - determined attackers will observe and react in order to try and defeat successful countermeasures, so the defenders must watch for shifting attack vectors.
-----------------------------------------------------------------------
Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions
On Sun, Dec 12, 2010 at 12:20 AM, Jeffrey Lyon <jeffrey.lyon@blacklotus.net> wrote:
I'm certain there are thresholds to that. Carrier grade mitigation solutions will start low and ramp up to 5, 6, 7, etc. figures depending on the attack and amount of bandwidth to be filtered among other variables.
nope, the pricing (when I was there, and I don't think it's changed much) is 3250/month for 500mbps or mitigation, though there was ~12gbps available easily before any work had to be done by the ISP... If the plan I/sfouant put in place was followed you could had scaled the capacity to much higher than that. If a customer continuously abused the 'limit' they may have been boosted to the next tier, but... I'd not ever seen that done. 3250/month... easy, peasy. -chris
Jeff
On Sun, Dec 12, 2010 at 12:05 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Fri, Dec 10, 2010 at 5:51 PM, Joel Jaeggli <joelja@bogus.com> wrote:
On 12/10/10 12:33 PM, Drew Weaver wrote:
Nobody has really driven the point home that yes you can purchase a system from Arbor, RioRey, make your own mitigation system; what-have you, but you still have to pay for the transit to digest the attack, which is probably the main cost right now.
or you outsource it and it's still costlier.
Paying for DOS mitigation you rarely if ever use is quite expensive. If you use it a lot it's even more expensive, but can at least be rationalized on the basis of known costs e.g. npv calculation on the number and duration of outages...
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts)
-chris
-Drew
-----Original Message----- From: Dobbins, Roland [mailto:rdobbins@arbor.net] Sent: Wednesday, December 08, 2010 11:54 AM To: North American Operators' Group Subject: Re: Over a decade of DDOS--any progress yet?
On Dec 8, 2010, at 11:47 PM, Jay Coley wrote:
This has been our recent experience as well.
I see a link-filling attacks with some regularity; but again, what I'm saying is simply that they aren't as prevalent as they used to be, because the attackers don't *need* to fill links in order to achieve their goals, in many cases.
That being said, high-bandwidth DNS reflection/amplification attacks tip the scales, every time.
Lastly there is usually always someone at the other end of these attacks watching what is working and what is not
This is a very important point - determined attackers will observe and react in order to try and defeat successful countermeasures, so the defenders must watch for shifting attack vectors.
-----------------------------------------------------------------------
Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions
I'm certain there are thresholds to that. Carrier grade mitigation solutions will start low and ramp up to 5, 6, 7, etc. figures depending on the attack and amount of bandwidth to be filtered among other variables.
My point was, if you "mitigate" the attack vs. null routing the target you have to pay for the transit that the attack consumes between your network and the upstream network(s). thanks, -Drew
On Mon, Dec 13, 2010 at 8:52 AM, Drew Weaver <drew.weaver@thenap.com> wrote:
I'm certain there are thresholds to that. Carrier grade mitigation solutions will start low and ramp up to 5, 6, 7, etc. figures depending on the attack and amount of bandwidth to be filtered among other variables.
My point was, if you "mitigate" the attack vs. null routing the target you have to pay for the transit that the attack consumes between your network and the upstream network(s).
so... with a carrier managed solution (or the one ATT/Sprint/VZB sold) the transit of the attack happens inside their networks and isn't charged to the end-customer (the destination, obviously contributing customers get charged :) ) -chris
On Sun, Dec 12, 2010 at 12:05 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts)
reasonable, but 'completely self-service' ? how much to have an engineer pump my gas for me (full service)? does that include a windshield wipe down, tire pressure and oil check (old timey full service extras)?
On Sun, Dec 12, 2010 at 12:42 AM, Aaron Glenn <aaron.glenn@gmail.com> wrote:
On Sun, Dec 12, 2010 at 12:05 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts)
reasonable, but 'completely self-service' ? how much to have an engineer pump my gas for me (full service)? does that include a windshield wipe down, tire pressure and oil check (old timey full service extras)?
end customer sends the right community and mitigation happens... remove the community it stops. no need to call someone and make it happen, just have the NOC/etc at your network follow a simple procedure. you are funny though :) (and I think you can call for free, 1-800 number, and get an engineer to make things happen for you as well...) -Chris
The thread made it to both NetworkWorld: http://www.networkworld.com/news/2010/120910-wikileaks-ddos-attacks.html and Slashdot: http://tech.slashdot.org/story/10/12/12/2120254/Has-Progress-Been-Made-In-Fi... with the usual set of comments :) -Lorand Jakab On 12/12/2010 08:58 AM, Christopher Morrow wrote:
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts) reasonable, but 'completely self-service' ? how much to have an engineer pump my gas for me (full service)? does
On Sun, Dec 12, 2010 at 12:05 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote: that include a windshield wipe down, tire pressure and oil check (old timey full service extras)? end customer sends the right community and mitigation happens... remove the community it stops. no need to call someone and make it happen, just have the NOC/etc at your network follow a simple
On Sun, Dec 12, 2010 at 12:42 AM, Aaron Glenn <aaron.glenn@gmail.com> wrote: procedure.
you are funny though :) (and I think you can call for free, 1-800 number, and get an engineer to make things happen for you as well...)
-Chris
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts) -chris
That doesn't sound too unreasonable as long as you are in a market Verizon services and you can find the right Verizon rep who isn't trying to sell transit at $25/mbps. thanks, -Drew
On Mon, Dec 13, 2010 at 8:49 AM, Drew Weaver <drew.weaver@thenap.com> wrote:
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts)
-chris
That doesn't sound too unreasonable as long as you are in a market Verizon services and you can find the right Verizon rep who isn't trying to sell transit at $25/mbps.
if you find that guy, maybe they'll also be the mythical unicorn of a sales person who will sell you ipv6 transit too? -chris
Date: Mon, 13 Dec 2010 10:09:16 -0500 From: Christopher Morrow <morrowc.lists@gmail.com>
On Mon, Dec 13, 2010 at 8:49 AM, Drew Weaver <drew.weaver@thenap.com> wrote:
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts)
-chris
That doesn't sound too unreasonable as long as you are in a market Verizon services and you can find the right Verizon rep who isn't trying to sell transit at $25/mbps.
if you find that guy, maybe they'll also be the mythical unicorn of a sales person who will sell you ipv6 transit too?
Unless VZB has started accepting prefixes longer than /32, they really don't have real IPv6 transit to sell. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
On Mon, Dec 13, 2010 at 3:29 PM, Kevin Oberman <oberman@es.net> wrote:
Date: Mon, 13 Dec 2010 10:09:16 -0500 From: Christopher Morrow <morrowc.lists@gmail.com> if you find that guy, maybe they'll also be the mythical unicorn of a sales person who will sell you ipv6 transit too?
Unless VZB has started accepting prefixes longer than /32, they really don't have real IPv6 transit to sell.
I did say 'mythical unicorn of a sales person' didn't I? :) -chris
On Dec 12, 2010, at 12:05 AM, Christopher Morrow wrote:
verizon's ddos service was/is 3250/month flat... not extra if there was some sort of incident, and completely self-service for the customer(s). Is 3250/month a reasonable insurance against loss? (40k/yr or there abouts)
Or just buy a gig-e from cogent at 3$/meg/mo (or is it $4 this month?) to burn for ddos. The problem I've found is that some of the vendors of ddos gear still have significant problems they are working to address. The Cisco (riverhead) guard would have a 1 second delay (for example) for each configuration line one would add. If you dealt with a wildcard rule, it would be 1 second per underlying rule to make the configuration change. The ability to 'paste' something in to a device and have a predictable output seemed to be too high of a bar for them to solve, this could be one of the reasons the product went to the wayside. I'm also not sure that anyone else is much better in this regard. Of course everyone is willing to sell you a seven-figure "solution" for your problems, but once you actually start talking about the usability, ease of provisioning, and the customer education about the caveats most people start to glaze quickly. Even with the right gear, technology, etc.. the vendors don't make it easy to deliver these solutions. - Jared
On 12/13/2010 8:32 AM, Jared Mauch wrote:
Or just buy a gig-e from cogent at 3$/meg/mo (or is it $4 this month?) to burn for ddos.
*cough* 10G burstable with 1-2G commit. Still cheaper than anything else I have or can get, and more likely to handle those large DDOS cases, where you can just reroute the effected network through the 10G and mitigate with whatever hardware you have.
Of course everyone is willing to sell you a seven-figure "solution" for your problems, but once you actually start talking about the usability, ease of provisioning, and the customer education about the caveats most people start to glaze quickly.
Even with the right gear, technology, etc.. the vendors don't make it easy to deliver these solutions.
True, but they often will dedicate some time and effort during an attack to make things work. There are many in-house custom solutions you can use, and we've seen public blacklists use many of them over the years. If you want the extra support during the crisis, you pay the 3rd party for their product to get it. Jack
On Dec 13, 2010, at 11:15 AM, Jack Bates wrote:
On 12/13/2010 8:32 AM, Jared Mauch wrote:
Or just buy a gig-e from cogent at 3$/meg/mo (or is it $4 this month?) to burn for ddos.
*cough* 10G burstable with 1-2G commit. Still cheaper than anything else I have or can get, and more likely to handle those large DDOS cases, where you can just reroute the effected network through the 10G and mitigate with whatever hardware you have.
my point is, there is this 'middle' space where it's hard to justify spending money on something that isn't used. Of course it's easy to view as "insurance" and easier to justify *after* an attack (or loss). it is hard to proactively justify this type of expense. If for every 10g of capacity, you had a 40k/year "Security" surcharge, at what point do you factor this in as part of your regular bandwidth costs vs the current "down and to the right" pricing trend. Delivering these services is something I have observed it is difficult to ask someone to pay for unless they have experience with it. Most are willing to start off with the "self-insure" premise until it is too much to bear, then immediately they are willing to pay 'something' to allow capital cost recovery.
Of course everyone is willing to sell you a seven-figure "solution" for your problems, but once you actually start talking about the usability, ease of provisioning, and the customer education about the caveats most people start to glaze quickly.
Even with the right gear, technology, etc.. the vendors don't make it easy to deliver these solutions.
True, but they often will dedicate some time and effort during an attack to make things work. There are many in-house custom solutions you can use, and we've seen public blacklists use many of them over the years. If you want the extra support during the crisis, you pay the 3rd party for their product to get it.
I am talking about those purporting to offer ddos solution hardware either past, present or future. If it's 2010 or 2011 and you experience flow-control like issues with your CLI interface, either slow interactive response or garbled processing (over telnet/ssh) there is something not quite right IMHO. Then again, I'm known for being a bit of an odd character. - Jared
FYI, A single data point on current DDOS traffic levels. An Akamai press release says they handled DDOS attacks peaking at 14Gbps in the Nov. 30 to Dec 2nd time frame. http://finance.yahoo.com/news/Akamai-Shields-Leading-prnews-2768453391.html "The majority of attack traffic against the five retailers initiated from distributed IP addresses out of Thailand, Mexico, Philippines, and Brazil and reached peeks of up to 14 Gbps, with some websites experiencing up to 10,000 times above normal daily traffic. " Bill Bogstad
On Dec 14, 2010, at 2:04 AM, Bill Bogstad wrote:
A single data point on current DDOS traffic levels.
In the 2009 Arbor WWISR, the largest attack reported was 49gb/sec. We're currently wrapping up the 2010 WWISR, and the largest attack report was considerably larger. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
The largest attacks we have solid proof on are 20+ Gbps. The only larger ones that i've seen were in company's marketing collateral vs. real life. Jeff On Mon, Dec 13, 2010 at 2:11 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Dec 14, 2010, at 2:04 AM, Bill Bogstad wrote:
A single data point on current DDOS traffic levels.
In the 2009 Arbor WWISR, the largest attack reported was 49gb/sec. We're currently wrapping up the 2010 WWISR, and the largest attack report was considerably larger.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Sell your computer and buy a guitar.
-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions
On Dec 14, 2010, at 2:40 AM, "Jeffrey Lyon" <jeffrey.lyon@blacklotus.net> wrote:
The only larger ones that i've seen were in company's marketing collateral vs. real life.
Here's a link to last year's Report (previous editions may be downloaded, as well): <http://www.arbornetworks.com/report> The WWISR is the result of a survey we perform every year of network operators; survey participants fill in their own answers, & we collect the data, collate it, analyze it, & publish it. We've observed packet-flooding attacks which are considerably larger than what's reported in the WWISR via ATLAS; but as the WWISR is about what operators see and share, we vet, relay & comment upon the observations of survey respondents. --------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
On Wed, Dec 8, 2010 at 8:47 AM, Jay Coley <jay@prolexic.com> wrote:
On 08/12/2010 16:14, Drew Weaver wrote:
I would say that > 99% of the attacks that we see are 'link fillers' with < 1% being an application attack.
thanks, -Drew
This has been our recent experience as well. There are some pure app attacks, to be sure, but we many blended attacks also. Bandwidth (UDP/ICMP/SYN Flood) attack to distract with a app attack (GET/PUSH floods) attempting to run underneath the radar. We regularly see SYN floods these days > 20 Gb/s.
Another thing to be aware of--when you get hit with what seems to be a "simple" flooding attack aimed at one point of your infrastructure... start checking your logs at _other_ places in your network very, VERY carefully. There seems to be a trend of using larger-scale flooding, or other simple types of attacks to get all the network people at an organization rushing over to throw resources and energy at it...while the real target of the attack is something completely different, on a different subnet, in a different part of the company; and that attack is small, carefully focused at its target, and is designed to be relatively quiet. The "big" attack is used simply to ensure all the human energy is focused on the wrong place, increasing the chance that what otherwise might caused raised eyebrows and double-checking of logs/IDS alerts, etc. gets missed while everyone is focusing on the"big" attack.
The thing to bear in mind is that app attacks *are* difficult to detect as they are low bandwidth and make a full TCP connection. As a result many IDS/Firewalls etc regularly miss these attacks.
Lastly there is usually always someone at the other end of these attacks watching what is working and what is not. If the attack doesn't work they will simply round up more bots to increase the attack bandwidth or change the attack vector.
And, in what seems to be an increasing trend, what they are watching for is *not* necessarily the result of the large botnet attack; they're checking on the results of their targeted probes elsewhere in the network, or on the outbound set of connections from a compromised machine within an organization; after all, during a huge DDoS attack, with everyone focusing on a set of uplinks being flooded with _inbound_ traffic, who is going to notice the (relatively smaller) outbound spike of traffic as the compromised machine sends out a copy of your internal intellectual property to the miscreant recipients? Matt (speaking purely hypothetically, of course, and definitely not on behalf of any institution or entity other than myself)
On Dec 9, 2010, at 1:34 AM, Matthew Petach wrote:
There seems to be a trend of using larger-scale flooding, or other simple types of attacks to get all the network people at an organization rushing over to throw resources and energy at it.
Concur, the more serious attackers use diversionary attacks or 'demonstrations' like this from time to time, absolutely. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
Ah, Honestly we can usually point to the exact cause of the attacks once we have time to triage the situation. Recently it has been stuff like: -Made someone in Asia angry. -Running a runescape server and made someone angry -Made someone on IRC angry It has been pretty rare to see an attack that wasn't just the end result of a pissing contest. and like I said most of the ones I have seen recently are either UDP 80 floods which is probably the result of one of the UDP.PL variants or fragments (UDP DST 0) attacks which kind of indicates at least in part that the 'attacker' simply downloaded the first thing they could find that said 'DDoS' on it and didn't spend too much time worrying about it. This is probably mainly because of how easy it is now to acquire dedicated servers (that arent properly monitored) and have 1Gbps (and now) 10Gbps connections to the Internet. How many organizations are using 10G connections to the Internet these days? -Drew -----Original Message----- From: Matthew Petach [mailto:mpetach@netflight.com] Sent: Wednesday, December 08, 2010 1:35 PM To: jay@prolexic.com Cc: nanog@nanog.org Subject: Re: Over a decade of DDOS--any progress yet? On Wed, Dec 8, 2010 at 8:47 AM, Jay Coley <jay@prolexic.com> wrote:
On 08/12/2010 16:14, Drew Weaver wrote:
I would say that > 99% of the attacks that we see are 'link fillers' with < 1% being an application attack.
thanks, -Drew
This has been our recent experience as well. There are some pure app attacks, to be sure, but we many blended attacks also. Bandwidth (UDP/ICMP/SYN Flood) attack to distract with a app attack (GET/PUSH floods) attempting to run underneath the radar. We regularly see SYN floods these days > 20 Gb/s.
Another thing to be aware of--when you get hit with what seems to be a "simple" flooding attack aimed at one point of your infrastructure... start checking your logs at _other_ places in your network very, VERY carefully. There seems to be a trend of using larger-scale flooding, or other simple types of attacks to get all the network people at an organization rushing over to throw resources and energy at it...while the real target of the attack is something completely different, on a different subnet, in a different part of the company; and that attack is small, carefully focused at its target, and is designed to be relatively quiet. The "big" attack is used simply to ensure all the human energy is focused on the wrong place, increasing the chance that what otherwise might caused raised eyebrows and double-checking of logs/IDS alerts, etc. gets missed while everyone is focusing on the"big" attack.
The thing to bear in mind is that app attacks *are* difficult to detect as they are low bandwidth and make a full TCP connection. As a result many IDS/Firewalls etc regularly miss these attacks.
Lastly there is usually always someone at the other end of these attacks watching what is working and what is not. If the attack doesn't work they will simply round up more bots to increase the attack bandwidth or change the attack vector.
And, in what seems to be an increasing trend, what they are watching for is *not* necessarily the result of the large botnet attack; they're checking on the results of their targeted probes elsewhere in the network, or on the outbound set of connections from a compromised machine within an organization; after all, during a huge DDoS attack, with everyone focusing on a set of uplinks being flooded with _inbound_ traffic, who is going to notice the (relatively smaller) outbound spike of traffic as the compromised machine sends out a copy of your internal intellectual property to the miscreant recipients? Matt (speaking purely hypothetically, of course, and definitely not on behalf of any institution or entity other than myself)
On 8 Dec 2010, at 15:40, Dobbins, Roland wrote:
On Dec 8, 2010, at 10:36 PM, Thomas Mangin wrote:
If you are a smaller network, you need the filtering to be performed by your transit provider, as your uplink will otherwise be congested.
Actually, most DDoS attacks aren't link-flooding attacks - this hasn't been true for the last ~7 years or so.
I'm not saying it doesn't happen, because it does, and sometimes quite spectacularly - but in most cases, the attackers don't have to flood the link to achieve their desired goal.
Fair point. I never had to face any intelligent type of DDOS ... lucky me :) Thomas
On Dec 8, 2010, at 10:10 PM, Thomas Mangin wrote:
Until this is sorted I believe flowspec will be a marginal solution.
We're seeing a significant uptick in flowspec interest, actually, and S/RTBH has been around for ages.
Great to hear :)
But my point is still valid [...]
After some offline discussion with Pedro Marques, I now realise that I misunderstood the flow rule validation process, which mean that my "complain" is really irrelevant, which is good news as it mean that inter ISP flow route exchange really have no technical obstacle that I can now think off. Thomas
participants (18)
-
Aaron Glenn -
alvaro.sanchez＠adinet.com.uy -
Bill Bogstad -
Christopher Morrow -
Dobbins, Roland -
Drew Weaver -
Jack Bates -
Jared Mauch -
Jay Coley -
Jeffrey Lyon -
jim deleskie -
Joel Jaeggli -
Kevin Oberman -
Loránd Jakab -
Matthew Petach -
Randy McAnally -
Thomas Mangin -
Zachary Hanna