Re: D/DoS mitigation hardware/software needed.
1. We have multiple nodes conducting DDoS scrubbing, one failing would not be catastrophic. 2. Indeed. 3. Sort of, such devices are downstream for extremely valid reasons I won't get into now. 4. Indeed, were equipped to handle substantially higher than 150kpps. I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I don't expect an employee of the vendor themselves to attest to this though. Best regards, Jeff Best regards, Jeff On Jan 4, 2010 10:14 PM, "Suresh Ramasubramanian" <ops.lists@gmail.com> wrote: On Tue, Jan 5, 2010 at 8:36 AM, Jeffrey Lyon <jeffrey.lyon@blacklotus.net> wrote: > We have such a c... So .. this is interesting. The firewall would have to frontend your mail / web / whatever application .. and if something goes beyond the firewall's rated capacity (100k ++ - maybe nearly 150..175k connections per second for a high end firewall), the firewall falls over. And even before that, there's the risk of whatever application you're protecting getting pounded flat if your firewall passes even a small percentage of this traffic. Do you - 1. Have (say) two firewalls in HA config? 2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff] 3. Simply back the firewall with a netflow based device? 4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people] --srs -- Suresh Ramasubramanian (ops.lists@gmail.com)
With these safeguards in place - and with flow devices being part of the mix somewhere .. what you propose is quite reasonable. There's still the question of whether an application that receives a lot of new / untrusted traffic - a mail or web server - would benefit from having a stateful firewall in front .. Roland seems to think not. --srs On Tue, Jan 5, 2010 at 9:35 AM, Jeffrey Lyon <jeffrey.lyon@blacklotus.net> wrote:
1. We have multiple nodes conducting DDoS scrubbing, one failing would not be catastrophic.
2. Indeed.
3. Sort of, such devices are downstream for extremely valid reasons I won't get into now.
4. Indeed, were equipped to handle substantially higher than 150kpps.
I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I don't expect an employee of the vendor themselves to attest to this though.
-- Suresh Ramasubramanian (ops.lists@gmail.com)
On Jan 5, 2010, at 11:05 AM, Jeffrey Lyon wrote:
I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution.
I disagree with this proposition, too. S/RTBH and/or flow-spec are great DDoS mitigation tools which don't require any further investment beyond the network infrastructure an operator has already purchased and deployed. These should be the first mitigation tools anyone deploys; in many cases, they're all that's needed.
I don't expect an employee of the vendor themselves to attest to this though.
Wrong again. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Mon, Jan 4, 2010 at 11:20 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Jan 5, 2010, at 11:05 AM, Jeffrey Lyon wrote:
I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution.
I disagree with this proposition, too.
S/RTBH and/or flow-spec are great DDoS mitigation tools which don't require any further investment beyond the network infrastructure an operator has already purchased and deployed. These should be the first mitigation tools anyone deploys; in many cases, they're all that's needed.
Is it fair to say that most folks in this thread believe there is not 'one size fits all', and that there are quite a few tools available in the security/networking toolbox? Some of these are outlined in past nanog tutorials: <http://www.nanog.org/meetings/nanog23/presentations/greene.pdf> <http://www.nanog.org/meetings/nanog26/presentations/ispsecure.pdf> <http://www.nanog.org/meetings/nanog28/presentations/sink.pdf> <http://www.nanog.org/meetings/nanog36/presentations/greene.ppt> Sorry to pick on barry here, but he's got a few preso's up from past NANOG's that cover this topic pretty well. All of them talk about a set of tools a network operator should be familiar with, with escalating costs (dollars and packet-loss/collateral damage), and some cut/pasteable configlets even.
I don't expect an employee of the vendor themselves to attest to this though.
Wrong again.
eh, roland's always happy to bash on employers :) but, he's got some solid standing on this set of points. Again, if you know what you're doing then feel free to go off and do it, but at first blush there are LOTS of people putting 'servers' out on the 'public network' behind devices whoafully prepared to deal with 'real world traffic demands', so instead of making the same mistake, perhaps learning from some experience would be in order? The original poster seemed to be asking about appliance based solutions, so your pointed remarks about Roland aside he was actually answering the question. I wonder if Stefan Fouant would offer some of his experience with 'not arbor' vendor solutions to be used when other techniques come up short? (note I think Roland may have been party to some of the presenations I linked in this... I certainly was for one of them at least, in case that matters.) -Chris
;>
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Injustice is relatively easy to bear; what stings is justice.
-- H.L. Mencken
On Jan 5, 2010, at 11:41 AM, Christopher Morrow wrote:
(note I think Roland may have been party to some of the presenations I linked in this...
Yes, sir, and thanks for posting those links! You and Barry and Tim Battles and Sean Donelan and Danny McPherson and Don Smith and Steve Bellovin and Jared Mauch and John Kristoff and a lot of other folks too numerous to mention here have contributed greatly to advancing the state of the art and generating the body of real-world opsec material out there, which it behooves all network operators to study and take into consideration in their own contexts. I also highly recommend this book by Dave Smith and Gregg Schudel of Cisco - it's the best (and only!) book on real-world opsec out there, available in dead-tree, Kindle, and Adobe Reader formats: <http://www.amazon.com/Router-Security-Strategies-Securing-Network/dp/1587053365/ref=sr_1_1?ie=UTF8&s=books&qid=1262667257&sr=8-1> [Full disclosure; I'm cited in the book, but received and continue to receive no renumeration of any kind due to same.] Here's another preso on this same topic which may be of interest: <http://files.me.com/roland.dobbins/k54qkv> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Jan 5, 2010, at 11:57 AM, Dobbins, Roland wrote:
You and Barry and Tim Battles and Sean Donelan and Danny McPherson and Don Smith and Steve Bellovin and Jared Mauch and John Kristoff and a lot of other folks too numerous to mention
. . . including Paul Vixie, Darrel Lewis, Ryan McDowell, Paul Quinn, Michael Behringer, Craig Huegen, Craig Labvoitz, Dave Smith, & Gregg Schudel, and many others. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Not necessarily an appliance, per se. But a "solution". :) A solution preferably that integrates with NetFlow and RTBH. An in-line solution obviously requires an appliance, or at least special/additional hardware. A software-only solution that sucks in NetFlow data and can speak BGP to inject /32 routes is also good. This is essentially what I have right now. With white-listing as a safety-net, I can chose whether traffic should be blocked automatically or punted for human eyes/brains/fingers to be the intelligence. I'm interested in seeing products (including software) that already have the development (anomaly detection, trends/reports, etc.) work done so I can spend my cycles elsewhere. Additional usefulness (not mentioned earlier) would be some form of API or other hook into the system so non-NetFlow input (e.g. syslog) could generate the /32 routes as well. I'm looking at taking the first whack at immediate mitigation at the border/edge (upstream) via uRPF and RTBH. Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Thanks, Rick On Mon, Jan 4, 2010 at 8:41 PM, Christopher Morrow <morrowc.lists@gmail.com>wrote:
The original poster seemed to be asking about appliance based solutions, so your pointed remarks about Roland aside he was actually answering the question. I wonder if Stefan Fouant would offer some of his experience with 'not arbor' vendor solutions to be used when other techniques come up short?
On Jan 5, 2010, at 12:05 PM, Rick Ernst wrote:
A solution preferably that integrates with NetFlow and RTBH. An in-line solution obviously requires an appliance, or at least special/additional hardware.
The key is to not be inline all the time, but only inline *when needed*. This removes operational complexity, provides the ability to oversubscribe, and simplifies the routine troubleshooting matrix.
I'm looking at taking the first whack at immediate mitigation at the border/edge (upstream) via uRPF and RTBH.
Good plan.
Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams.
Automagic is generally bad, as it can be gamed. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Mon, Jan 4, 2010 at 9:08 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Jan 5, 2010, at 12:05 PM, Rick Ernst wrote:
A solution preferably that integrates with NetFlow and RTBH. An in-line
solution obviously requires an appliance, or at least special/additional hardware.
The key is to not be inline all the time, but only inline *when needed*. This removes operational complexity, provides the ability to oversubscribe, and simplifies the routine troubleshooting matrix.
I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. "What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc."
I'm looking at taking the first whack at immediate mitigation at the border/edge (upstream) via uRPF and RTBH.
Good plan.
Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams.
Automagic is generally bad, as it can be gamed.
I know you said "generally", but if I'm seeing 200Kpps from a.b.c.d, I don't care if a.b.c.d is spoofed. I want the traffic blocked from the guts of my network. Note that my original question was in the context of "a D/DoS composed of lots of itty-bitty packets". Other attack mechanisms do not necessarily lend themselves to "chop 'em off at the knees." Rick
On Jan 5, 2010, at 12:19 PM, Rick Ernst wrote:
I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. "What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc."
I strongly disagree with this, except for properties which are under sustained attack 24/7. If one has constructed one's detection/classification/traceback/mitigation system properly, one always knows at a glance the state of the system. Otherwise, whenever there's any issue whatsoever with the properties under protection, one must try and prove a negative - i.e., that the mitigation solution isn't causing the problem. Happens every time, heh.
I know you said "generally", but if I'm seeing 200Kpps from a.b.c.d, I don't care if a.b.c.d is spoofed. I want the traffic blocked from the guts of my network.
Not if it's legit, you don't, or if the attacker is spoofing, say, the IPs of the root nameservers, or the TLDs, or an e-commerce/supply-chain partner . . . or if the attack is originating behind a broadband mega-proxy, or a mobile CGN. ;> Also, if you've a variety of tools at your disposal, like S/RTBH and/or flow-spec, and then more sophisticated (and expensive) tools like IDMS, the freedom to choose the least intrusive/most situationally-appropriate tool to mitigate a given attack is essential for resource preservation and the ability to oversubscribe the more sophisticated tools.
Note that my original question was in the context of "a D/DoS composed of lots of itty-bitty packets". Other attack mechanisms do not necessarily lend themselves to "chop 'em off at the knees."
Absolutely, which is where the situationally-specific selection of tools/modes comes into play. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
-----Original Message----- From: Rick Ernst [mailto:nanog@shreddedmail.com] Sent: Tuesday, January 05, 2010 12:19 AM
I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. "What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc."
Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
On Tue, Jan 05, 2010, Stefan Fouant wrote:
Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp.
Has anyone deployed a DDoS distributed enough to inject ETOOMANY routes into the hardware forwarding tables of routers? I mean, I assume that there's checks and balances in place to limit then number of routes being injected into the network so one doesn't overload the tables, but what's the behaviour if/when this limit is reached? Does mitigation cease being as effective? Adrian
On Jan 5, 2010, at 12:39 PM, Adrian Chadd wrote:
I mean, I assume that there's checks and balances in place to limit then number of routes being injected into the network so one doesn't overload the tables, but what's the behaviour if/when this limit is reached? Does mitigation cease being as effective?
For IDMS 'scrubbing' solutions, one merely injects the route of the attack targets into one's iBGP, in order to draw all traffic towards that specific target into the scrubbing center. For S/RTBH and flow-spec, modern edge routers can scale to millions of routes; also note one isn't limited to /32s. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
* Adrian Chadd:
Has anyone deployed a DDoS distributed enough to inject ETOOMANY routes into the hardware forwarding tables of routers?
I think this is called "injecting the BGP table into your IGP", and quite a few folks have done it. Nobody claimed that it was an act of sabotage, though. -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
I think you, Roland, and I are all agreeing on the same argument. The (my) confusion entered with the statement of, "The key is to not be inline all the time, but only inline *when needed*. I inferred a topological or physical path change from that. Redirecting traffic (which is really just an extension of RTBH; a scrubber destination rather than Null0) is an understandable state. Rick On Mon, Jan 4, 2010 at 9:34 PM, Stefan Fouant <sfouant@shortestpathfirst.net
wrote:
-----Original Message----- From: Rick Ernst [mailto:nanog@shreddedmail.com] Sent: Tuesday, January 05, 2010 12:19 AM
I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. "What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc."
Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp.
Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
On Jan 5, 2010, at 12:39 PM, Rick Ernst wrote:
I think you, Roland, and I are all agreeing on the same argument.
GMTA. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Tue, 5 Jan 2010, Stefan Fouant wrote:
Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp.
That said, what are all those ISPs doing now that Cisco has stopped developing the Guard? -Hank
-----Original Message----- From: Hank Nussbacher [mailto:hank@efes.iucc.ac.il] Sent: Tuesday, January 05, 2010 1:02 AM
On Tue, 5 Jan 2010, Stefan Fouant wrote:
Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp.
That said, what are all those ISPs doing now that Cisco has stopped developing the Guard?
Well of course, moving to Arbor haha... eased in part by a little Cisco initiative called Clean Pipes 2.0 :) Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
(Resent, sorry for multiple copies -- I messed up from From: address) On 5 Jan 2010, at 06:16, Stefan Fouant wrote:
That said, what are all those ISPs doing now that Cisco has stopped developing the Guard?
Well of course, moving to Arbor haha... eased in part by a little Cisco initiative called Clean Pipes 2.0 :)
Is this really true? I've seen the white paper, I've been told that the this is the best way forward from the Guard, but I must say that I'm not yet totally convinced. The Guard product was something that can be separated from the Cisco Detection approach, i.e. one can activate the Guard via a means that did not necessarily involve the Detectors being the source of the activation, this doesn't seem to be true for the Arbor alternative (I believe that the TMS requires registering against the rest of the PeakFlow platform). The other thing that we noted relating to the platform is that there's nothing really "new" in the TMS (other than of course, much increased scrubbing rates!) compared to the Guard. There doesn't appear to be any direct comparison to the 'strong' scrubbing mode that the Cisco Guard implemented - whereby the device would proxy a bunch of traffic. If you're an SP who has some existing NetFlow solution, and don't really justify a spend for traffic intelligence within your network (or have something home-grown), is there an alternative scrubber that one might be able to use in a more standalone deployment that can approach the filtering levels of the Arbor kit? I should probably point out that we only really started our conversation with Arbor within the last month or so, so there are perhaps details relating to this that I've missed. I'd be happy to be corrected! Kind regards, Rob -- Rob Shakir <rjs@eng.gxn.net> Network Development Engineer GX Networks/Vialtus Solutions ddi: +44208 587 6077 mob: +44797 155 4098 pgp: 0xc07e6deb nic-hdl: RJS-RIPE This email is subject to: http://www.vialtus.com/disclaimer.html
My somewhat educated opinion on the matter is that appliance developers want to sit on the edge and see all your traffic merely to protect their own interests and market share. NS-5000s have been good to us for bulk filtering and we rely on appliances for more intelligent inspection. Dollar for dollar NS is substantially more robust in my experience. Best regards, Jeff On Jan 5, 2010 9:46 AM, "Rob Shakir" <rjs@eng.gxn.net> wrote: (Resent, sorry for multiple copies -- I messed up from From: address) On 5 Jan 2010, at 06:16, Stefan Fouant wrote: >> >> That said, what are all those ISPs doing now th... Is this really true? I've seen the white paper, I've been told that the this is the best way forward from the Guard, but I must say that I'm not yet totally convinced. The Guard product was something that can be separated from the Cisco Detection approach, i.e. one can activate the Guard via a means that did not necessarily involve the Detectors being the source of the activation, this doesn't seem to be true for the Arbor alternative (I believe that the TMS requires registering against the rest of the PeakFlow platform). The other thing that we noted relating to the platform is that there's nothing really "new" in the TMS (other than of course, much increased scrubbing rates!) compared to the Guard. There doesn't appear to be any direct comparison to the 'strong' scrubbing mode that the Cisco Guard implemented - whereby the device would proxy a bunch of traffic. If you're an SP who has some existing NetFlow solution, and don't really justify a spend for traffic intelligence within your network (or have something home-grown), is there an alternative scrubber that one might be able to use in a more standalone deployment that can approach the filtering levels of the Arbor kit? I should probably point out that we only really started our conversation with Arbor within the last month or so, so there are perhaps details relating to this that I've missed. I'd be happy to be corrected! Kind regards, Rob -- Rob Shakir <rjs@eng.gxn.net> Network Development Engineer GX Networks/Vialtus Solutions ddi: +44208 587 6077 mob: +44797 155 4098 pgp: 0xc07e6deb nic-hdl: RJS-RIPE This email is subject to: http://www.vialtus.com/disclaimer.html
On Jan 5, 2010, at 9:59 PM, Jeffrey Lyon wrote:
My somewhat educated opinion on the matter is that appliance developers want to sit on the edge and see all your traffic merely to protect their own interests and market share.
This isn't generally a smart approach; the value of providing maximum topological coverage and the ability to oversubscribe the system by moving inwards a bit from the edge provides clear advantages, in most circumstances. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Jan 5, 2010, at 9:44 PM, Rob Shakir wrote:
If you're an SP who has some existing NetFlow solution, and don't really justify a spend for traffic intelligence within your network (or have something home-grown), is there an alternative scrubber that one might be able to use in a more standalone deployment that can approach the filtering levels of the Arbor kit?
One thing folks can do is to implement S/RTBH and/or flow-spec at the edges, then take some Intel boxes, throw some high-performance network cards in them, along with Snort-inline, and put them in a scrubbing center, making use of BGP diversion/re-injection to get traffic into them on an as-needed basis. Don't make use of the useless bidirectional/stateful 'IPS' signatures, but do manual filtering on a case-by-case basis, unidirectionally. Below that, put a layer of WCCPv2-clustered Squid proxies to provide additional layer-7 filtering capabilities for Web-based traffic. Below that, re-inject the scrubbed traffic and send it on its way via one's redirection mechanism of choice to the destination servers. Does this 'poor man's IDMS' do everything and scale to the degree of the various commercial systems? No, of course not. But it's a way to get into layer-4-plus dynamic DDoS mitigation in a relatively economical way (at least from a capex perspective); for some folks, this type of solution may prove sufficient to needs, keeping in mind the limitations of this approach and the systems integration/support burden involved, of course. And even if/when it's clear more advanced capabilities are needed, starting out this way, even in a limited PoC, provides valuable operational experience with the diversion/re-injection model which will prove useful in evaluating more advanced commercial systems an an appropriate juncture.
I should probably point out that we only really started our conversation with Arbor within the last month or so, so there are perhaps details relating to this that I've missed. I'd be happy to be corrected!
There's actually quite a bit more, but short of answering specific questions in an operational context, this kind of vendor-specific discussion is probably well beyond what vendor employees like me (these strictures don't apply to anyone else, mind) can and should in all propriety participate in on nanog-l; please feel free to reach out 1:1 to the Arbor folks you've already been talking with to discuss further, and I'm happy to help out 1:1, as well. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Tue, Jan 5, 2010 at 10:38 AM, Dobbins, Roland <rdobbins@arbor.net> wrote:
Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams.
Automagic is generally bad, as it can be gamed.
... and manual wont scale in ddos -- Suresh Ramasubramanian (ops.lists@gmail.com)
On Jan 5, 2010, at 12:19 PM, Suresh Ramasubramanian wrote:
... and manual wont scale in ddos
Actually, it can and does. ;> I'm referring to the employment and selection of situationally-appropriate tools, mind. The tools themselves must of necessity perform their work in a largely automated fashion once they're employed, which is what I believe you actually meant. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Tue, Jan 5, 2010 at 10:52 AM, Dobbins, Roland <rdobbins@arbor.net> wrote:
I'm referring to the employment and selection of situationally-appropriate tools, mind. The tools themselves must of necessity perform their work in a largely automated fashion once they're employed, which is what I believe you actually meant.
fair enough. -- Suresh Ramasubramanian (ops.lists@gmail.com)
-----Original Message----- From: Suresh Ramasubramanian [mailto:ops.lists@gmail.com] Sent: Tuesday, January 05, 2010 12:19 AM
On Tue, Jan 5, 2010 at 10:38 AM, Dobbins, Roland <rdobbins@arbor.net> wrote:
Additional mitigation would be via manual or automatic RTBH or
security/abuse@ involvement with upstreams.
Automagic is generally bad, as it can be gamed.
... and manual wont scale in ddos
There are pros and cons to each approach. Certain types of things can be automated, in fact I've done this using the Auto-mitigate feature in Arbor coupled with pre-configured mitigation templates for certain types of traffic and it works very well. But generally, as Roland mentioned automagic stuff can be gamed and for the majority of the stuff you are going to want an operator to look at the alert before making the decision to offramp. The trick is to try to automate as much around the process as possible - I've worked in environments where just making little changes to incident handling response methods reduced the time to mitigate an attack from hours to minutes, all the while still requiring an operator to press the "big red button" to offramp and enable the mitigation. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
On Jan 5, 2010, at 12:39 PM, Stefan Fouant wrote:
The trick is to try to automate as much around the process as possible - I've worked in environments where just making little changes to incident handling response methods reduced the time to mitigate an attack from hours to minutes, all the while still requiring an operator to press the "big red button" to offramp and enable the mitigation.
Concur 100% - and when the end-customer is under attack and screaming, this reduction in time to detect/classify/traceback/mitigate makes all the difference. Your very salient comments highlight the paramount importance of preparation as the key enabling phase of the six-phase security incident-handling methodology: 1. Preparation. 2. Detection/identification. 3. Classification. 4. Traceback. 5. Reaction. 6. Post-mortem (feeding lessons learned back into the Preparation phase). ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
On Tue, Jan 5, 2010 at 10:35 AM, Rick Ernst <nanog@shreddedmail.com> wrote:
I'm interested in seeing products (including software) that already have the development (anomaly detection, trends/reports, etc.) work done so I can spend my cycles elsewhere.
This might fit the bill - http://www.zurich.ibm.com/aurora/ Now commercially available as http://www-01.ibm.com/software/tivoli/products/netcool-performance-flow/ Full disclosure - I work for big blue - but not in any division that works on Aurora / Tivoli Netcool. -- Suresh Ramasubramanian (ops.lists@gmail.com)
-----Original Message----- From: Christopher Morrow [mailto:morrowc.lists@gmail.com] Sent: Monday, January 04, 2010 11:41 PM
The original poster seemed to be asking about appliance based solutions, so your pointed remarks about Roland aside he was actually answering the question. I wonder if Stefan Fouant would offer some of his experience with 'not arbor' vendor solutions to be used when other techniques come up short?
Interesting thread! And I'm happy to chime in - thanks Chris! I too would have to strongly agree with Roland's comments about not front-ending your mitigation solution with firewalls or load-balancers - these are precisely the types of things which topple over first under a big attack, as such having your mitigation devices behind them makes little sense. If you must use firewalls and/or LBs, put them behind the mitigation device, where the traffic has already been scrubbed and your state tables won't be exhausted. Having said that, if you've got a router capable of doing generic packet filters upstream of your mitigation device, this is certainly a good place to apply stateless filters which can pitch any traffic you are sure you will never need to receive. Flowspec and/or automated blackhole routing works very well for this application when you want to get rid of certain types of cruft, before it hits your mitigation device. Now, on to the OPs original question regarding appliance based solutions, I would say I am actually a firm believer in having multiple vendors in place if you can afford it. Arbor definitely has a corner on the market here, with the most comprehensive solution which entails everything from detection to mitigation and pretty much everything in between. Arbor can also automate the FlowSpec process and/or generate router ACLs for propagation to upstream routers... They do a lot of other stuff as well such as identification of BGP hijacking, DNS analysis, can automate a lot of the RTBH or S/RTBH stuff. Where Arbor generally suffers is with sophisticated attack traffic which requires complex mitigations - these often require a lot of tweaking in order to properly scrub the traffic... knowing your environment and which attack vectors are likely to be exploited is your best bet here, where you can configure mitigation templates in advance for rapid deployment during an attack. I've also worked with the RioRey devices and I have to say that although they don't have all the bells and whistles that some of the other vendors offer, their approach to mitigation is entirely unique and can genuinely mitigate attacks in record-time. Without disclosing too much of their intellectual property, I will say that their algorithms essentially look at the randomness and probability of address space distribution within the attack traffic, and can generally offer a high degree of certainty of scrubbing the majority of the bad traffic - and they do this WITHOUT having to do DPI as many other vendors are currently doing. Bottom line - if you're looking for something with a lot of bells and whistles and the ability to monitor/detect/analyze/etc., you're probably better off going with an Arbor solution. If you have lower OpEx and just want something that you can "set it and forget it", you'd be hard pressed not to look at the RioRey. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
On Tue, 5 Jan 2010 04:20:51 +0000 "Dobbins, Roland" <rdobbins@arbor.net> wrote:
S/RTBH and/or flow-spec are great DDoS mitigation tools which don't require any further investment beyond the network infrastructure an operator has already purchased and deployed. These should be the first mitigation tools anyone deploys; in many cases, they're all that's needed.
I still wish we would have had something like Bellovin's Pushback implemented as a separate protocol rather than flow-spec over BGP, but having lost that battle we have been playing around with a (free) community, but vetted participant, flow-spec over BGP service if folks are interested in trying it out. Just shoot me note offline. You need an ASN, a flow-spec capable router and must be a verifiable admin/sec contact for said ASN (whatever that means :-). Basic idea is for folks who want to receive one or more sets of flow-spec feeds and/or inject things they want others to filter on (limited to your own routes) you can do so. No need for direct peering and like you say Roland, many networks already have all they need to start doing these sorts of things. Please note, we realize there are a variety of issues in implementing this sort of thing, but if we can find a way to make it trustworthy and workable, that is why we're here. Those not familiar with flow-spec can read up: <http://tools.ietf.org/html/rfc5575> In a nutshell, distributed router filters via BGP. John
participants (11)
-
Adrian Chadd -
Christopher Morrow -
Dobbins, Roland -
Florian Weimer -
Hank Nussbacher -
Jeffrey Lyon -
John Kristoff -
Rick Ernst -
Rob Shakir -
Stefan Fouant -
Suresh Ramasubramanian