John, Contrary to popular belief, I (not alone, of course) run, manage, defend, and continually architect very large networks. Very large. On none of them do we outsource the protection of them -- because, in cases where we have extended trust in the past, we have been screwed (PC translation: disappointed). So we protect ourselves. It's been a business decision for my customers' networks (ie. their network) not to outsource security, or rely on an upstreampipedream, for protection of any sort. Thus, I personally can't provide any insight here. Sorry. - ferg -- John Neiberger <jneiberger@gmail.com> wrote: In this case it's a business decision. I understand that we could simply weigh the costs of an attack with the costs of preemptively detecting and mitigating an attack, but in our case we won't lose hard dollars like an ecommerce site would. We have different reasons for wanting to have some protection in place before we need it. I look at it like it's an insurance policy, but I don't want to be ripped off. It's like I'm getting estimates on building a protective dike around my house. One contractor tells me that the floodwaters commonly reach six feet so I should pay him $12,000 to build a wall at least that high. Another contractor is telling me that he'll build a six-foot wall for $6,000. Another contractor is telling me that the floodwaters most likely won't go over two feet and he suggests that I pay him $1,000 for a three-foot-high wall. If it turns out that we really do need a six-foot-high wall then so be it. I'm not the one who pays the bills so it isn't really my decision. I just want to make sure I have a clearer picture of reality before I make any suggestions to my boss. Thanks again, John On 7/28/05, Fergie (Paul Ferguson) <fergdawg@netzero.net> wrote:
I should've asked the most important question first -- is this a technical decision, or a business decision? I mean, forgive me for pointing out the obvious, but you made an issue of cost in your original post...
- ferg
-- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg@netzero.net or fergdawg@sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/
On Fri, 29 Jul 2005, Fergie (Paul Ferguson) wrote:
John,
Contrary to popular belief, I (not alone, of course) run, manage, defend, and continually architect very large networks. Very large. On none of them do we outsource the protection of them -- because, in cases where we have extended trust in the past, we have been screwed (PC translation: disappointed).
So we protect ourselves.
It's been a business decision for my customers' networks (ie. their network) not to outsource security, or rely on an upstreampipedream, for protection of any sort.
Thus, I personally can't provide any insight here. Sorry.
- ferg
Ferg, Not everyone is in a position to have anetwork large enough to be "self-defending". I think he has clearly stated they are not in a position from a capacity standpoint to self-defend. If he has a few sites with some T1's or DS3's or whatever, his goal is to not stop the traffic at his router, but not ever allow the traffic onto his pipe. I too have been involved in large, very large, networks and we used to see it happen everyday. Customers with OC12's getting smoked off the planet because of some kiddie made someone else mad in IRC. If the upstream offers a "value add" service such as DoS protection, why balk at it? -j
Ferg, That's an understandable attitude given the nature of your networks. In our case, I'm just talking about two or three T1s that provide Internet connectivity to our website for our customers. I appreciate your input, though. I will accept all advice and input if it gets me closer to a better understanding of the realities of topic at hand and if it helps weed out some of the marketing fluff that's being heaped upon me by salespeople. :) Thanks! John On 7/28/05, Fergie (Paul Ferguson) <fergdawg@netzero.net> wrote:
John,
Contrary to popular belief, I (not alone, of course) run, manage, defend, and continually architect very large networks. Very large. On none of them do we outsource the protection of them -- because, in cases where we have extended trust in the past, we have been screwed (PC translation: disappointed).
So we protect ourselves.
It's been a business decision for my customers' networks (ie. their network) not to outsource security, or rely on an upstreampipedream, for protection of any sort.
Thus, I personally can't provide any insight here. Sorry.
- ferg
-- John Neiberger <jneiberger@gmail.com> wrote:
In this case it's a business decision. I understand that we could simply weigh the costs of an attack with the costs of preemptively detecting and mitigating an attack, but in our case we won't lose hard dollars like an ecommerce site would. We have different reasons for wanting to have some protection in place before we need it. I look at it like it's an insurance policy, but I don't want to be ripped off.
It's like I'm getting estimates on building a protective dike around my house. One contractor tells me that the floodwaters commonly reach six feet so I should pay him $12,000 to build a wall at least that high. Another contractor is telling me that he'll build a six-foot wall for $6,000. Another contractor is telling me that the floodwaters most likely won't go over two feet and he suggests that I pay him $1,000 for a three-foot-high wall.
If it turns out that we really do need a six-foot-high wall then so be it. I'm not the one who pays the bills so it isn't really my decision. I just want to make sure I have a clearer picture of reality before I make any suggestions to my boss.
Thanks again, John
On 7/28/05, Fergie (Paul Ferguson) <fergdawg@netzero.net> wrote:
I should've asked the most important question first -- is this a technical decision, or a business decision? I mean, forgive me for pointing out the obvious, but you made an issue of cost in your original post...
- ferg
-- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg@netzero.net or fergdawg@sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/
On 7/28/05, John Neiberger <jneiberger@gmail.com> wrote:
Ferg,
That's an understandable attitude given the nature of your networks. In our case, I'm just talking about two or three T1s that provide Internet connectivity to our website for our customers.
I appreciate your input, though. I will accept all advice and input if it gets me closer to a better understanding of the realities of topic at hand and if it helps weed out some of the marketing fluff that's being heaped upon me by salespeople. :)
Thanks! John
On 7/28/05, Fergie (Paul Ferguson) <fergdawg@netzero.net> wrote:
John,
Contrary to popular belief, I (not alone, of course) run, manage, defend, and continually architect very large networks. Very large. On none of them do we outsource the protection of them -- because, in cases where we have extended trust in the past, we have been screwed (PC translation: disappointed).
So we protect ourselves.
It's been a business decision for my customers' networks (ie. their network) not to outsource security, or rely on an upstreampipedream, for protection of any sort.
Thus, I personally can't provide any insight here. Sorry.
- ferg
-- John Neiberger <jneiberger@gmail.com> wrote:
In this case it's a business decision. I understand that we could simply weigh the costs of an attack with the costs of preemptively detecting and mitigating an attack, but in our case we won't lose hard dollars like an ecommerce site would. We have different reasons for wanting to have some protection in place before we need it. I look at it like it's an insurance policy, but I don't want to be ripped off.
It's like I'm getting estimates on building a protective dike around my house. One contractor tells me that the floodwaters commonly reach six feet so I should pay him $12,000 to build a wall at least that high. Another contractor is telling me that he'll build a six-foot wall for $6,000. Another contractor is telling me that the floodwaters most likely won't go over two feet and he suggests that I pay him $1,000 for a three-foot-high wall.
If it turns out that we really do need a six-foot-high wall then so be it. I'm not the one who pays the bills so it isn't really my decision. I just want to make sure I have a clearer picture of reality before I make any suggestions to my boss.
Thanks again, John
On 7/28/05, Fergie (Paul Ferguson) <fergdawg@netzero.net> wrote:
I should've asked the most important question first -- is this a technical decision, or a business decision? I mean, forgive me for pointing out the obvious, but you made an issue of cost in your original post...
- ferg
-- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg@netzero.net or fergdawg@sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/
Thinking about this a bit, I subscribe to the theory that it somewhat depends on how big of a target you are. People who have large networks usually offer more services to more people, thus they have more exposure. "Most of the time" when I see the large DoS attacks they are to customers with nice fat pipes. I've rarely seen large (2-3Gb) attacks to small customers with a few T1's. Not to say it doesn't ever happen. But if you're not hosting a ton of sites there's not that much reason for someone to DoS you. Botnets are a commodity and when they are used, inevitably, a portion of the bots are found and fixed. There's usually 2 reasons why people are DoS'd. You, or a customer, has pissed someone else off or it's being done to you for extortion reasons. If you're hosting a small site for your business as mostly an informational purpose then you're likely not going to be really pissing someone off. If you have a large ecommerce site then you are a target for extortion purposes. Following this chain of logic, if you have a large site, you have large pipes. I guess what I'm getting at is this may be one area where security through obscurity may actually pan out, speaking **only** to security of bandwidth/packet rate DoS attacks. As always, there are exceptions and anything can happen. I'm just speaking from my personal experience. I've worked in a largeish ISP NOC for about 3 years and this is mostly what I've seen. Some things you can do that are free include making sure that if/when you get attacked you have a plan in place of how to deal with it. This includes having up to date contact information for your service provider. Knowing what their capabilities are and how they deal with attacks. Having circuit information, hardware information, and hardware vendor contact information available to help all involved parties aid in mitigating the attack. This can save huge amounts of time when "bad things happen" and this applies no matter how large or small you are. --chip -- Just my $.02, your mileage may vary, batteries not included, etc....
hey, chip's a good egg as well, listen to him too :) On Thu, 28 Jul 2005, chip wrote:
Some things you can do that are free include making sure that if/when you get attacked you have a plan in place of how to deal with it. This includes having up to date contact information for your service provider. Knowing what their capabilities are and how they deal with attacks. Having circuit information, hardware information, and hardware vendor contact information available to help all involved parties aid in mitigating the attack. This can save huge amounts of time when "bad things happen" and this applies no matter how large or small you are.
This sort of thing is very often overlooked. As with any emergency plan: 1) have a plan 2) test the plan 3) validate the plan Also, chip didn't mention this, but... perhaps what is being attacked doesn't HAVE to work. Your provider might also have the possibility to let you blackhole things inside their network, so if something less important is attacked, just make it go away 'free' don't pay for mitigation if it's not required... -Chris chemical engineer... :)
On Thu, 28 Jul 2005, John Neiberger wrote:
Ferg,
That's an understandable attitude given the nature of your networks. In our case, I'm just talking about two or three T1s that provide Internet connectivity to our website for our customers.
I appreciate your input, though. I will accept all advice and input if it gets me closer to a better understanding of the realities of topic at hand and if it helps weed out some of the marketing fluff that's being heaped upon me by salespeople. :)
Ok, so why not jump in with 1 foot atleast :) A note first though: 1) UUNET/MCI does sell this product (I don't sell it personally, I don't sell anything actually) 2) UUNET/MCI's sales method for this product is 'confusing' (to me atleast, but recall I'm a chemical engineer...) 3) UUNET/MCI has been providing this service for free for 6+ years, now with special gear and a price for 'enhanced services' now, down to business. The core of your question is two parts: a) how much should you spend b) how much protection do you need For the 'a' part a few folks have said: "Pay what you are willing to part with". That means you have to decide how much protection you want and how much you'll need (see 'b'). For 'b' I can say, after 5+ years defending UUNET's customers globally (well, the team I work on does this globally it's not just me) and giving a talk here or there about this subject: "Attackers will do just enough to be effective" Keep in mind there is no way for them to know you have a 9600 baud modem or a oc-48. I've seen 400mbps attacks against modem users, and a modem's worth of 'attack' aimed at a oc-12 customer :( Normally the attackers aim a weapon at the victim, shoot and add more weapons if required. They will add more until they get their effect. This COULD mean that if you purchased 60 gbps of attack mitigation capacity you'd get screwed in the end... There is a trade off: "how much is realistic to expect", this has nothing to do with your end-site connectivity. I'd aim at an average (high average) attack size. I'd aim at 500mbps/1gbps, I'd also ask a few other questions: 1) how does this mitigation get started? (phone call, ticket, call back? or customer initiated bgp update? or prayers to the ddos-mitigation-god?) 2) how much capacity is available regardless of what is purchased? 3) how quickly can extra capacity be added if required? (days? hours? seconds? at all?) 4) how much latency will be incurred if I have a /32 under mitigation? what about a /24? a /16? does it matter? 5) how much granularity in the policy of said device(s) do I have? 6) how does reporting work for this service? (how do I know anything is happening?) 7) are there dedicated individuals prepared to answer my questions at 0dark:30 on a Saturday Christmas night? As I said, I do this for a living, I have a little bit of a bias :) but I'm sure if you listen to Mr. Feger he's a smart guy as well, who knows this problem as well as I do... Good luck! If you want other info about this service (the mci version of it) and don't want to jaw with a sales droid you can get me off-list. Same goes for other folks, I'd just note I'm away from email a bit over the next few days so I may be a little slow to respond :) -Chris
Hi, I'm very interested in technical solutions of ISP based (D)DOS solutions. Where can I find document/information on it? thanks. Joe Send instant messages to your online friends http://asia.messenger.yahoo.com
Take a look at this link: http://www.cisco.com/en/US/netsol/ns480/networking_solutions_sub_solution_ho... HTH, John On 7/28/05, Joe Shen <joe_hznm@yahoo.com.sg> wrote:
Hi,
I'm very interested in technical solutions of ISP based (D)DOS solutions. Where can I find document/information on it?
thanks.
Joe
Send instant messages to your online friends http://asia.messenger.yahoo.com
I'm very interested in technical solutions of ISP based (D)DOS solutions. Where can I find document/information on it?
design one yourself or buy mitigation capacity from someone who does it already... or I think I may give a talk at the next nanog on this topic, since it seems like the call for papers included a request to chat about this topic :)
On 29/07/05, Joe Shen <joe_hznm@yahoo.com.sg> wrote:
I'm very interested in technical solutions of ISP based (D)DOS solutions. Where can I find document/information on it?
A very quick google search for "ISP ddos mitigation nanog" gets me this link - http://www.honeypots.net/incidents/ddos-mitigation Has links to some interesting presentations, including - starting from the basics Barry Greene's templates for Egress and Ingress filtering on Cisco (oh and RFC3704), Several presentations on filtering bogons, Chris Morrow on blackhole route servers [VERY useful, that] Plus presentations on ddos mitigation boxes (Riverhead, Arbor etc) . Oh yes - dig through the nanog / apricot etc archives for past presentations on darknets and the team cymru darknet project -- Suresh Ramasubramanian (ops.lists@gmail.com)
participants (7)
-
chip -
Christopher L. Morrow -
Fergie (Paul Ferguson) -
James Feger -
Joe Shen -
John Neiberger -
Suresh Ramasubramanian