Pros and Cons of Cloud Computing in dealing with DDoS
I'm working on an article on the Pros and Cons of Cloud Computing as an effective strategy for dealing with DDoS. I'd like to open this up for debate and get some perspectives from folks on the list. In a recent article in ITWire titled "DDoS, the biggest threat to Cloud Computing", Roland Dobbins states that "DDoS attacks are one of the most under-rated and ill-guarded against security threats to corporate IT, and in particular the biggest threat facing cloud computing." To a certain extent, I agree with Roland, however, I also believe this perspective is inconsistent with the view that the elasticity of cloud computing and ability to scale resources on demand is a good way of dealing with the problem. The counterpoint to this is that I can also envision the cloud computing model causing a shift from that of a DDoS to what some are calling EDoS (Economic Denial of Sustainability). In an EDoS, the elasticity of the cloud and surplus of available resources might be used in such a way that large botnets generating seemingly legitimate "targeted" requests for service causing the victim to cloudburst in order to keep pace with the scale of the requests. Even though the victim can sustain business operations, the cost of doing so may be so exorbitantly expensive that to do so threatens economic sustainability. Roland also states "The cloud providers emerging as leaders don't tend to talk much about their resiliency to DDoS attacks". Which brings about another point - are there any cloud providers taking a proactive look at dealing with this problem and deploying effective countermeasures for dealing with this in their environments? What motivation would cloud providers have to deploy DDoS mitigation services and/or services which can distinguish between legitimate resource consumption vs. targeted resource consumption, especially if their revenues are driven from service availability and potential expansion of resource utilization? Stefan Fouant GPG Key ID: 0xB5E3803D
DDoS is a threat to the cloud just as DDoS is a threat to any other service when you fail to implement protection. Our company recently put out a DDoS mitigated cloud product specifically for high risk clients. Best regards, Jeff On Thu, Nov 5, 2009 at 1:06 PM, Stefan Fouant <sfouant@shortestpathfirst.com> wrote:
I'm working on an article on the Pros and Cons of Cloud Computing as an effective strategy for dealing with DDoS. I'd like to open this up for debate and get some perspectives from folks on the list.
In a recent article in ITWire titled "DDoS, the biggest threat to Cloud Computing", Roland Dobbins states that "DDoS attacks are one of the most under-rated and ill-guarded against security threats to corporate IT, and in particular the biggest threat facing cloud computing." To a certain extent, I agree with Roland, however, I also believe this perspective is inconsistent with the view that the elasticity of cloud computing and ability to scale resources on demand is a good way of dealing with the problem. The counterpoint to this is that I can also envision the cloud computing model causing a shift from that of a DDoS to what some are calling EDoS (Economic Denial of Sustainability). In an EDoS, the elasticity of the cloud and surplus of available resources might be used in such a way that large botnets generating seemingly legitimate "targeted" requests for service causing the victim to cloudburst in order to keep pace with the scale of the requests. Even though the victim can sustain business operations, the cost of doing so may be so exorbitantly expensive that to do so threatens economic sustainability.
Roland also states "The cloud providers emerging as leaders don't tend to talk much about their resiliency to DDoS attacks". Which brings about another point - are there any cloud providers taking a proactive look at dealing with this problem and deploying effective countermeasures for dealing with this in their environments? What motivation would cloud providers have to deploy DDoS mitigation services and/or services which can distinguish between legitimate resource consumption vs. targeted resource consumption, especially if their revenues are driven from service availability and potential expansion of resource utilization?
Stefan Fouant
GPG Key ID: 0xB5E3803D
-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Platinum sponsor of HostingCon 2010. Come to Austin, TX on July 19 - 21 to find out how to "protect your booty."
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, Nov 5, 2009 at 10:27 AM, Seth Mattinen <sethm@rollernet.us> wrote:
Jeffrey Lyon wrote:
DDoS is a threat to the cloud just as DDoS is a threat to any other service when you fail to implement protection. Our company recently put out a DDoS mitigated cloud product specifically for high risk clients.
How about the cloud as a DDoS source?
It's called a botnet. - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.5.3 (Build 5003) wj8DBQFK8yGZq1pz9mNUZTMRAsipAJ0ZuFMPjGDhK8RjZD8yerlNJZ4sLQCeLrbp vDQqPsasoKduDA2m+rj2h1Q= =PLip -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thu, Nov 5, 2009 at 10:27 AM, Seth Mattinen <sethm@rollernet.us> wrote:
Jeffrey Lyon wrote:
DDoS is a threat to the cloud just as DDoS is a threat to any other service when you fail to implement protection. Our company recently put out a DDoS mitigated cloud product specifically for high risk clients.
How about the cloud as a DDoS source?
It's called a botnet.
Everything's called a botnet. How about 'clotnet'? :-) ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
-----Original Message----- From: jeffrey.lyon@gmail.com [mailto:jeffrey.lyon@gmail.com] On Behalf Of Jeffrey Lyon Sent: Thursday, November 05, 2009 1:20 PM To: Stefan Fouant Cc: NANOG list Subject: Re: Pros and Cons of Cloud Computing in dealing with DDoS
DDoS is a threat to the cloud just as DDoS is a threat to any other service when you fail to implement protection. Our company recently put out a DDoS mitigated cloud product specifically for high risk clients.
Best regards, Jeff
Obviously the cloud is no different than any other infrastructure insofar as implementing protection mechanisms. Ample bandwidth (typically more so than in the enterprise) should make it easier to absorb larger amounts of the bad stuff. What I'm really wondering is what steps cloud providers are taking to be able to differentiate between the legitimate vs. targeted resource consumption, what are their motivations if the main thing driving revenue is expansion of resource utilization, or do most cloud providers simply think this is a non-issue if they can just overengineer compute, storage, and network resources such that they can sustain even the heaviest loads, legitimate or not. I'd also like to get perspectives from some of the heavy hitters (ahem... Danny, Roland, etc.) and understand why they think DDoS is the single biggest threat to the cloud computing model, again this is counter to a lot of evidence which points to the corollary - think DNS Root Servers and you'll have an idea what I'm talking about... BTW - the BlackLotus offering using RioRey is pretty cool (those are good boxes and I've used them before for specific point applications), but I'm really trying to discuss the relevance to cloud based services, not hosted services (I don't generally group them into the same category). Stefan Fouant GPG Key ID: 0xB5E3803D
On Nov 6, 2009, at 2:11 AM, Stefan Fouant wrote:
Obviously the cloud is no different than any other infrastructure insofar as implementing protection mechanisms. Ample bandwidth (typically more so than in the enterprise) should make it easier to absorb larger amounts of the bad stuff.
Actually, no - the miscreants are always going to have more bandwidth at their disposal, plus they utilize attack vectors which provide a great deal of amplification (including at layer-7) which make bandwidth largely irrelevant.
why they think DDoS is the single biggest threat to the cloud computing model,
Availability is the one thing which *must* be guaranteed at all costs in order for the cloud model to work, and by definition, most cloud infrastructure isn't going to be within the span of control of the end- customer. Look at all the apps/services we all use and depend upon every day - Webmail, IM, various Web 2.0ish AJAXy things, Skype, SIP, et al. When these things are DDoSed either deliberately or inadvertently, directly or indirectly (i.e., zorching authoritative DNS a la Baofeng), lots and lots of folks end up getting hosed. Now, expand this to your back-end line-of-business apps, your IP PBXes, your customer databases, your ERP software, your CAM/CAM system, your basic file/print services, and the picture becomes much clearer. The movement towards 'cloud' - starting with things like VPS and VPDC and SaaS for specific applications - largely consists of end-customer organizations jettisoning their internal data centers/WAN links/ops staff and subscribing to these apps/services on a recurring basis, with said apps/services either residing within a public-facing IDC or in a multitenanted IDC made available to the end-customer via an MPLS NGN. It entails shutting down locally-/internally-owned-and-operated DCs and moving into
again this is counter to a lot of evidence which points to the corollary
Which evidence is that? [You meant 'contrary', yes?]
- think DNS Root Servers and you'll have an idea what I'm talking about...
There's a heck of a lot of engineering which has gone into protecting the roots - I'm sure you'll recall the high-visibility DDoS attacks which affected multiple roots in the past. The root operators learned from that experience and took proactive measures to ensure that they can continue to maintain availability in the face of constant onslaughts. The bottom line is that it's easy to achieve perfect confidentiality and integrity if availability is lacking, heh. All three legs of the classical information security triad are of import, but it's always been my view that availability is the first among equals, which translates into the need for robust, scalable architecture and the willingness to devote time and resources to the operational security art. Paul's comment about botnets being 'cloud' services is dead-on; and of course, miscreants using stolen credit-cards to purchase IaaS for spamming/phishing purposes has already been seen in the wild, just as they do so with their nonsense domains for botnet C&C. IaaS abused to launch DDoS won't be far behind. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sorry, sometimes I mistake your existential crises for technical insights. -- xkcd #625
-----Original Message----- From: Roland Dobbins [mailto:rdobbins@arbor.net] Sent: Thursday, November 05, 2009 4:35 PM
On Nov 6, 2009, at 2:11 AM, Stefan Fouant wrote:
Obviously the cloud is no different than any other infrastructure insofar as implementing protection mechanisms. Ample bandwidth (typically more so than in the enterprise) should make it easier to absorb larger amounts of the bad stuff.
Actually, no - the miscreants are always going to have more bandwidth at their disposal, plus they utilize attack vectors which provide a great deal of amplification (including at layer-7) which make bandwidth largely irrelevant.
So if I'm hearing you correctly, you're saying that no matter how much infrastructure you have to potentially absorb the problem, there is nothing you can do because the bad guys are always going to have more bandwidth at their disposal. Man, that's a pretty bad position to be in for a vendor who's fundamental premise is to sell boxes to deal with these sorts of problems. ;) I've built quite a few of these solutions now, and the designs usually entail having enough bandwidth and other resources at your disposal so as to be able to scrub the traffic with purpose-built mitigation equipment. I'd also like to point out that according to the 4th edition of Arbor's Worldwide Infrastructure Security Report, only about 1% of all attacks observed via ATLAS were in the 10+ Gbps range. So while there are certainly larger attacks exhibited in the wild, I'm pretty certain that most of the cloud providers today have at least enough bandwidth to deal with the other 99% of attacks, assuming they have the appropriate countermeasures in place to scrub the traffic. To your point however with regards to various attack vectors, I am in agreement that this doesn't provide any tangible benefit to those low-level attacks which require surgical mitigation to deal with.
why they think DDoS is the single biggest threat to the cloud computing model,
Availability is the one thing which *must* be guaranteed at all costs in order for the cloud model to work, and by definition, most cloud infrastructure isn't going to be within the span of control of the end- customer. Look at all the apps/services we all use and depend upon every day - Webmail, IM, various Web 2.0ish AJAXy things, Skype, SIP, et al. When these things are DDoSed either deliberately or inadvertently, directly or indirectly (i.e., zorching authoritative DNS a la Baofeng), lots and lots of folks end up getting hosed.
Now, expand this to your back-end line-of-business apps, your IP PBXes, your customer databases, your ERP software, your CAM/CAM system, your basic file/print services, and the picture becomes much clearer.
The movement towards 'cloud' - starting with things like VPS and VPDC and SaaS for specific applications - largely consists of end-customer organizations jettisoning their internal data centers/WAN links/ops staff and subscribing to these apps/services on a recurring basis, with said apps/services either residing within a public-facing IDC or in a multitenanted IDC made available to the end-customer via an MPLS NGN. It entails shutting down locally-/internally-owned-and-operated DCs and moving into
again this is counter to a lot of evidence which points to the corollary
Which evidence is that? [You meant 'contrary', yes?]
Yep, brainfart. ;)
- think DNS Root Servers and you'll have an idea what I'm talking about...
There's a heck of a lot of engineering which has gone into protecting the roots - I'm sure you'll recall the high-visibility DDoS attacks which affected multiple roots in the past. The root operators learned from that experience and took proactive measures to ensure that they can continue to maintain availability in the face of constant onslaughts.
My point exactly - similar measures can and should be done to ensure that cloud computing models are similarly robust.
The bottom line is that it's easy to achieve perfect confidentiality and integrity if availability is lacking, heh. All three legs of the classical information security triad are of import, but it's always been my view that availability is the first among equals, which translates into the need for robust, scalable architecture and the willingness to devote time and resources to the operational security art.
Paul's comment about botnets being 'cloud' services is dead-on; and of course, miscreants using stolen credit-cards to purchase IaaS for spamming/phishing purposes has already been seen in the wild, just as they do so with their nonsense domains for botnet C&C. IaaS abused to launch DDoS won't be far behind.
This is really scary to think about... if we look at how Service Providers typically respond to hosts on their network behaving badly, it doesn't bode well for the Internet as a whole. Stefan Fouant GPG Key ID: 0xB5E3803D
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, Nov 5, 2009 at 4:46 PM, Stefan Fouant <sfouant@shortestpathfirst.com> wrote:
Actually, no - the miscreants are always going to have more bandwidth at their disposal, plus they utilize attack vectors which provide a great deal of amplification (including at layer-7) which make bandwidth largely irrelevant.
So if I'm hearing you correctly, you're saying that no matter how much infrastructure you have to potentially absorb the problem, there is nothing you can do because the bad guys are always going to have more bandwidth at their disposal. Man, that's a pretty bad position to be in for a vendor who's fundamental premise is to sell boxes to deal with these sorts of problems. ;)
Well, the fact of the matter is that you can't put 10 lb. of [expletive] in a 5 lb. bag, so to speak. :-) - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.5.3 (Build 5003) wj8DBQFK83sEq1pz9mNUZTMRAnvkAJ9XcDIi7XTE32nMtwJfwCflq6hcdgCfXmPT OkqNIuL8OH+BN6S4UxlfdSc= =kqaC -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/
-----Original Message----- From: Paul Ferguson [mailto:fergdawgster@gmail.com] Sent: Thursday, November 05, 2009 8:26 PM
On Thu, Nov 5, 2009 at 4:46 PM, Stefan Fouant <sfouant@shortestpathfirst.com> wrote:
Actually, no - the miscreants are always going to have more
bandwidth
at their disposal, plus they utilize attack vectors which provide a great deal of amplification (including at layer-7) which make bandwidth largely irrelevant.
So if I'm hearing you correctly, you're saying that no matter how much infrastructure you have to potentially absorb the problem, there is nothing you can do because the bad guys are always going to have more bandwidth at their disposal. Man, that's a pretty bad position to be in for a vendor who's fundamental premise is to sell boxes to deal with these sorts of problems. ;)
Well, the fact of the matter is that you can't put 10 lb. of [expletive] in a 5 lb. bag, so to speak. :-)
Which is why vendors selling DDoS mitigation equipment will always tell you to get a 15lb. bag first. ;) Their solutions work, but only if you got a bag big enough to store a lot of crap. Stefan Fouant GPG Key ID: 0xB5E3803D
* Stefan Fouant:
Which is why vendors selling DDoS mitigation equipment will always tell you to get a 15lb. bag first. ;) Their solutions work, but only if you got a bag big enough to store a lot of crap.
Not all attacks involve saturated pipes. There used to be anti-DDoS vendors whose boxes didn't even have WAN links. Part of the problem is that operating systems come with TCP stacks and web servers which are not very robust, so it's pretty easy to create something which behaves spectacularly better under certain attacks. -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
-----Original Message----- From: Florian Weimer [mailto:fweimer@bfk.de] Sent: Friday, November 06, 2009 4:55 AM
Not all attacks involve saturated pipes.
There used to be anti-DDoS vendors whose boxes didn't even have WAN links. Part of the problem is that operating systems come with TCP stacks and web servers which are not very robust, so it's pretty easy to create something which behaves spectacularly better under certain attacks.
I am in complete agreement with you here. And I don't think the things I've said are generally inconsistent with the views held by others. The original point I was trying to make before the discussion got so eloquently hijacked towards a discussion on flooding and its impact on availability is that with regards to cloud computing, if the discussion hasn't shifted from that of DDoS to EDoS, it should. Just take one look at Amazon's usage-based pricing model, and one can envision that a surplus of resources could actually be just as bad as a lack thereof. How long do you think it will take for the bad guys to realize that they don't need to cause an outage to cause havoc to the victim. A slow trickle of seemingly legitimate requests from just a few thousand hosts performed over several days or weeks might give some organizations pause and that $50k extortion might start looking pretty enticing. I second Roland's comments with regard to the CIA triad, and his opinion that availability of resources is the first among equals is spot on. But I'm willing to bet that if the attackers exploit the so-called elasticity of the cloud and the subsequent associated financial costs, integrity is going to take on a whole new level of importance. BTW, heuristic/behavioral based analysis has benefit here, it just needs to start happening on more granular level... Getting back to the original discussion, I'd still like to hear what some of you think are the Pros vs. the Cons of Cloud Computing in dealing with this situation. We've heard a few and now I'd like to hear what others have to say. BTW, I realize this is a sensitive subject and I can understand why some of you might not want to respond on-list (security through obscurity eh' ;). To those of you who have taken the time to respond to me off-list, I appreciate your feedback and promise to keep your identities confidential. Regards, Stefan Fouant GPG Key ID: 0xB5E3803D
On Nov 8, 2009, at 2:33 AM, Stefan Fouant wrote:
if the discussion hasn't shifted from that of DDoS to EDoS, it should.
All DDoS is 'EDoS' - it's a distinction without a difference, IMHO. DDoS costs opex, can cost direct revenue, can induce capex spends - it's all about economics at bottom, always has been, or nobody would care in the first place. And look at click-fraud attacks in which the miscreants either a) are committing fraud by causing botnets to make fake clicks so that they can be paid for same or b) wish to exhaust a rival's advertising budget when he's paying per-impression. Plain old packet-flooding DDoSes can cost victims/unwitting sources big money in transit costs, can cost SPs in transit and/or violating peering agreements, etc. There's no need or justification for a separate term; Chris Hoff bounced 'EDoS' around earlier this year, and the same arguments apply. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Sun, 8 Nov 2009, Dobbins, Roland wrote:
if the discussion hasn't shifted from that of DDoS to EDoS, it should.
All DDoS is 'EDoS' - it's a distinction without a difference, IMHO.
DDoS costs opex, can cost direct revenue, can induce capex spends - it's all about economics at bottom, always has been, or nobody would care in the first place. And look at click-fraud attacks in which the miscreants either a) are committing fraud by causing botnets to make fake clicks so that they can be paid for same or b) wish to exhaust a rival's advertising budget when he's paying per-impression. Plain old packet-flooding DDoSes can cost victims/unwitting sources big money in transit costs, can cost SPs in transit and/or violating peering agreements, etc.
There's no need or justification for a separate term; Chris Hoff bounced 'EDoS' around earlier this year, and the same arguments apply.
The so-called "E"DOS is easy to solve. Just re-negotiate your contract with the cloud service provider to exclude that traffic from your bill. After all, if the cloud security provider's security is great, they shouldn't have a problem giving their customers credit for those problems which the cloud solves. No more "E" problems for thec customer, the DOS risk is shifted to the service provider. But now the service provider still needs to solve the same problem. Oh, the cloud service provider won't negotiate, won't give you unlimited service credits, want to charge extra for that protection, don't want to make promises it will work, and so on :-) The same unsolved problems from the 1970's mainframe/timesharing era still haven't been solved with virtualization/cloud/etc.
Sean Donelan wrote:
Oh, the cloud service provider won't negotiate, won't give you unlimited service credits, want to charge extra for that protection, don't want to make promises it will work, and so on :-)
The same unsolved problems from the 1970's mainframe/timesharing era still haven't been solved with virtualization/cloud/etc.
I'm sorry, you must have not received the memo on how cloud computing is the bee's knees and solves all those silly problems that only affect non-cloud services. We'll get you a copy. ~Seth
On Nov 6, 2009, at 7:46 AM, Stefan Fouant wrote:
So if I'm hearing you correctly, you're saying that no matter how much infrastructure you have to potentially absorb the problem, there is nothing you can do because the bad guys are always going to have more bandwidth at their disposal.
What I'm saying is that one can't simply rely on bandwidth capacity/ connection capacity/tps scaling/etc. on their own to always 'eat' the problem traffic; rather that there's a full spectrum of things one must do in order to be able to maintain availability in the face of attack, starting with fundamental architecture at layer-7 and moving down the model, taking special care to try and avoid design choices which lead to blocking behaviors and/or open up amplification vectors (some of these simply can't be avoided due to protocol semantics, of course). I'm also saying that threats to availability aren't something one can always assume one will be able to handle alone; engaging with the larger opsec community is key. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sorry, sometimes I mistake your existential crises for technical insights. -- xkcd #625
* Stefan Fouant:
Obviously the cloud is no different than any other infrastructure insofar as implementing protection mechanisms.
It's different in one aspect, though: you don't know with whom you're sharing your toothbrush. To some extent, this is true for other infrastructure as well (even your dedicated Internet connectivity eventually joins shared infrastructure, which is precisely the point, of course). But virtualization makes those risks very difficult to estimate. Some companies have already suffered from this because they completely outsourced their authoritative DNS service to dedicated DNS service providers. Only very few customers of those providers were attacked, but the impact was felt across larger parts of their customer base. (The obvious thing to do is to use both external DNS and DNS on your network, so you stay up even if your external DNS goes down. I suppose a similar model could be used for many in-the-cloud services.) -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
On 11/6/09 10:52 AM, Florian Weimer wrote:
* Stefan Fouant:
Obviously the cloud is no different than any other infrastructure insofar as implementing protection mechanisms.
It's different in one aspect, though: you don't know with whom you're sharing your toothbrush.
I love the analogy, though I feel the need to replace my toothbrush. I also think that clouds have a tendency to do usage based billing, and hence a DDOS can run your meter, which perhaps is where Stefan Fouant gets to the idea of EDOS. Eliot
-----Original Message----- From: Florian Weimer [mailto:fweimer@bfk.de] Sent: Friday, November 06, 2009 4:52 AM To: Stefan Fouant Cc: 'Jeffrey Lyon'; 'NANOG list' Subject: Re: Pros and Cons of Cloud Computing in dealing with DDoS
Some companies have already suffered from this because they completely outsourced their authoritative DNS service to dedicated DNS service providers. Only very few customers of those providers were attacked, but the impact was felt across larger parts of their customer base.
Been there, done that. I used to work for Ultra. 'Nuff said ;) Stefan Fouant GPG Key ID: 0xB5E3803D
participants (10)
-
Dobbins, Roland -
Eliot Lear -
Florian Weimer -
Jeffrey Lyon -
Joe Greco -
Paul Ferguson -
Roland Dobbins
-
Sean Donelan -
Seth Mattinen -
Stefan Fouant