Howdy, Is there any place where people with experience dealing with DDoS attacks hang out? I'm getting very little assistance from my upstream beyond "call whomever is in charge of each IP attacking and make them stop", and "even though we null route the destination IP being attacked, this traffic will be billed". I've got a nice snippet of flows, so I can mostly see where everything is coming from, and it's obvious what the target is, but my flow-stat/flow-report skills are pretty weak. Oddly, in eight years of working for smallish ISPs I've never been hit very hard, believe it or not. Is the response from my upstream typical? I was expecting a bit more cooperation rather than them seeing as this as an opportunity to bill me for lots of traffic. Thanks, Charles -- Charles Sprickman spork@inch.com
Hmmm..... Maybe if NANOG had irc.nanog.org, maybe that might be something to consider - a real-time network of communication for network operators to deal with issues, etc. -- Jonathan On Sat, 19 Jun 2004 22:04:36 -0400 (EDT), Charles Sprickman <spork@inch.com> wrote:
Howdy,
Is there any place where people with experience dealing with DDoS attacks hang out? I'm getting very little assistance from my upstream beyond "call whomever is in charge of each IP attacking and make them stop", and "even though we null route the destination IP being attacked, this traffic will be billed".
I've got a nice snippet of flows, so I can mostly see where everything is coming from, and it's obvious what the target is, but my flow-stat/flow-report skills are pretty weak.
Oddly, in eight years of working for smallish ISPs I've never been hit very hard, believe it or not. Is the response from my upstream typical? I was expecting a bit more cooperation rather than them seeing as this as an opportunity to bill me for lots of traffic.
Thanks,
Charles
-- Charles Sprickman spork@inch.com
-- Jonathan M. Slivko - jslivko@gmail.com "Linux: The Choice for the GNU Generation" - http://www.linux.org/ - Don't fear the penguin. .^. /V\ /( )\ ^^-^^ He's here to help.
On Sat, 19 Jun 2004, Jonathan Slivko wrote:
Maybe if NANOG had irc.nanog.org, maybe that might be something to consider - a real-time network of communication for network operators to deal with issues, etc.
There's always http://puck.nether.net/mailman/listinfo/nsp-security -- srs (postmaster|suresh)@outblaze.com // gpg : EDEDEFB9 manager, outblaze.com security and antispam operations
On Sun, 20 Jun 2004, Suresh Ramasubramanian wrote:
On Sat, 19 Jun 2004, Jonathan Slivko wrote:
Maybe if NANOG had irc.nanog.org, maybe that might be something to consider - a real-time network of communication for network operators to deal with issues, etc.
There's always http://puck.nether.net/mailman/listinfo/nsp-security
I can tell you right off AS8059 doesn't meet the requirements. I'd gladly respond to any reports of attacks from them, but I don't think you'd ever see any. Basement multihomers unite. Charles
-- srs (postmaster|suresh)@outblaze.com // gpg : EDEDEFB9 manager, outblaze.com security and antispam operations
On Sat, 19 Jun 2004, Charles Sprickman wrote:
On Sun, 20 Jun 2004, Suresh Ramasubramanian wrote:
On Sat, 19 Jun 2004, Jonathan Slivko wrote:
Maybe if NANOG had irc.nanog.org, maybe that might be something to consider - a real-time network of communication for network operators to deal with issues, etc.
There's always http://puck.nether.net/mailman/listinfo/nsp-security
I can tell you right off AS8059 doesn't meet the requirements. I'd gladly respond to any reports of attacks from them, but I don't think you'd ever see any.
which of your 2 upstreams isn't helping out? I'm fairly certain both providers have security groups, and do mitigate attacks for customers on a regular basis. Perhaps you are not getting in touch with the correct customer service folks? We often have this issue ;(
Basement multihomers unite.
hurray!
On Sun, 20 Jun 2004, Christopher L. Morrow wrote:
which of your 2 upstreams isn't helping out? I'm fairly certain both providers have security groups, and do mitigate attacks for customers on a regular basis. Perhaps you are not getting in touch with the correct customer service folks? We often have this issue ;(
I don't want to go too much into it, but HE.net, once they supplied me with the proper channels immediately null-routed the IP, hurrah! I'm waiting on the answer as to whether we get billed or not for this traffic. The other upstream whom I won't name is through a reseller. That wasn't necessarily our first choice, but their own sales department told us to go with a reseller as they were not interested in two cabinets and a 100Mb handoff, so that's what we did. I'm hoping their reseller is just misunderstanding something here. For a long time he kept telling me "this is illegal, you need to contact the source networks and make them stop it", so I'm guessing DDoS is not a subject he's intimately familiar with (nor am I, but I understand the mechanics of it, and I don't think that I could contact each source in my lifetime). Thanks to everyone for your input. To answer some other questions, the box under attack is not a client box, but it is the main webserver for the ISP's own site and ~user sites. It's also has shell accounts, but since I've been here I've not seen one complaint about any of our users. Most seem to not know much beyond how to use "pine". I think most of our heavy-duty irc users are using windows clients at home, any irc tools on the server are horribly dated. Not saying it's not a possibility, but I do personally watch "abuse@" and I've not seen anyone complain about the box. Thanks again, Charles
Basement multihomers unite.
hurray!
Charles Sprickman wrote:
I don't want to go too much into it, but HE.net, once they supplied me with the proper channels immediately null-routed the IP, hurrah! I'm waiting on the answer as to whether we get billed or not for this traffic.
One other way to get a hold of clueful contacts, especially if you have your own AS, is the inoc-dba project - http://www.pch.net/inoc-dba/ srs
On Sat, 19 Jun 2004, Charles Sprickman wrote:
On Sun, 20 Jun 2004, Christopher L. Morrow wrote:
which of your 2 upstreams isn't helping out? I'm fairly certain both providers have security groups, and do mitigate attacks for customers on a regular basis. Perhaps you are not getting in touch with the correct customer service folks? We often have this issue ;(
I don't want to go too much into it, but HE.net, once they supplied me with the proper channels immediately null-routed the IP, hurrah! I'm waiting on the answer as to whether we get billed or not for this traffic.
obviously I can't speak for eiteher of your providers, but normally you'd only get billed for traffic that goes down your link to the provider, not for traffic which enters the provider and isn't delivered to you.
I'm hoping their reseller is just misunderstanding something here. For a long time he kept telling me "this is illegal, you need to contact the source networks and make them stop it", so I'm guessing DDoS is not a subject he's intimately familiar with (nor am I, but I understand the mechanics of it, and I don't think that I could contact each source in my lifetime).
Depending on the situation you might not have much other recourse :( To stop the pain though, each provider should provide you with some immediate actions. Perhaps asking them if you can do customer triggered blackholing?
I could host and/or setup the irc server if anyone is interested. On Sun, Jun 20, 2004 at 03:23:06AM +0000, Christopher L. Morrow wrote:
On Sat, 19 Jun 2004, Charles Sprickman wrote:
On Sun, 20 Jun 2004, Suresh Ramasubramanian wrote:
On Sat, 19 Jun 2004, Jonathan Slivko wrote:
Maybe if NANOG had irc.nanog.org, maybe that might be something to consider - a real-time network of communication for network operators to deal with issues, etc.
There's always http://puck.nether.net/mailman/listinfo/nsp-security
I can tell you right off AS8059 doesn't meet the requirements. I'd gladly respond to any reports of attacks from them, but I don't think you'd ever see any.
which of your 2 upstreams isn't helping out? I'm fairly certain both providers have security groups, and do mitigate attacks for customers on a regular basis. Perhaps you are not getting in touch with the correct customer service folks? We often have this issue ;(
Basement multihomers unite.
hurray!
-- Bubba Parker sysadmin@citynetwireless.net CityNet LLC http://www.citynetinfo.com/
Is there any place where people with experience dealing with DDoS attacks hang out? I'm getting very little assistance from my upstream beyond "call whomever is in charge of each IP attacking and make them stop", and "even though we null route the destination IP being attacked, this traffic will be billed".
It seems that you should look somewhere else for your next bandwidth contract...
I've got a nice snippet of flows, so I can mostly see where everything is coming from, and it's obvious what the target is, but my flow-stat/flow-report skills are pretty weak.
Fake or real source IPs ? TCP SYNs, ICMPs, UDPs ? Rubens
--On Saturday, June 19, 2004 22:04 -0400 Charles Sprickman <spork@inch.com> wrote:
Howdy,
Is there any place where people with experience dealing with DDoS attacks hang out? I'm getting very little assistance from my upstream beyond "call whomever is in charge of each IP attacking and make them stop", and "even though we null route the destination IP being attacked, this traffic will be billed".
That's outrageous but not unheard of....if it never makes it to you then you shouldn't be billed for it.
I've got a nice snippet of flows, so I can mostly see where everything is coming from, and it's obvious what the target is, but my flow-stat/flow-report skills are pretty weak.
Oddly, in eight years of working for smallish ISPs I've never been hit very hard, believe it or not. Is the response from my upstream typical? I was expecting a bit more cooperation rather than them seeing as this as an opportunity to bill me for lots of traffic.
The normal flow unless you're a big guy yourself is to talk to your upstreams who contact theirs and put null routes in place at both steps. Depending on the size of the DDoS. My current place of employment we got nailed down with 100mbit+ SYN attack here recently (I had an eng from one of the major upstreams, can't rememebr which, quote it at north of 200mbit, but by the time it made it to me we were only attempting to sink about 90-120mbit, but we couldn't hardly keep up with that). Most places will not charge for that. And I think it's absurd that anyone does, and that you should probably take your business elsewhere if your upstream is engaged in this sort of gouging.
Charles Sprickman wrote:
Is there any place where people with experience dealing with DDoS attacks hang out? I'm getting very little assistance from my upstream beyond "call whomever is in charge of each IP attacking and make them stop", and "even though we null route the destination IP being attacked, this traffic will be billed".
While I hate the "blame the victim" mentality in general, I'd guess that up to half of all the packet floods we've experienced were aimed at compromised client boxes that were hosting illegitimate services. If your victim has no idea why they're being attacked, I'd scrutinize the target host very carefully. Or if your victim is a shell host who's probably got some skript kiddie engaged in channel wars, it will likely save you a lot of time and grief to just dump that client. Is losing one worth sacrificing the rest? Unless you know exactly why you're being attacked and are willing to suffer these consequences indefinitely, you will do yourself a big favor by looking at the victim to see why the attack is occurring and removing the target from your network.
Just following up with a bit more info. While I have no way of knowing whether these IPs are the true source, and there's likely more that I didn't capture in the short windows where the router was up and exporting netflow data, this is what I have. If anyone here is in charge of the following blocks, perhaps you might want to have a look: 208.39.142 (comcast, business cable) 216.235.244 (e-xpedient) 218.244.162 (chinacom) 218.247.37 (china network connect) 61.48.80 (china network communications group) 62.231.65 (romania data systems) Actually, looking at those sources, I'm betting they're not spoofed. :) Thanks, Charles -- Charles Sprickman spork@inch.com On Sat, 19 Jun 2004, Charles Sprickman wrote:
Howdy,
Is there any place where people with experience dealing with DDoS attacks hang out? I'm getting very little assistance from my upstream beyond "call whomever is in charge of each IP attacking and make them stop", and "even though we null route the destination IP being attacked, this traffic will be billed".
With the lamentable exception of the IRC suggestions, there have been some very good comments on this. However, in the interest of beating a dead horse (and not aimed directly at Charles) - Think about stuff like this when picking your transit providers. There is some conventional wisdom that Internet transit is a commodity. While it is in some ways, there are a couple areas that are anything but: Security (and security response), including DDOS abatement. Does your provider do Remote Triggered Blackhole Filtering? Does your provider have backscatter servers? Does your provider have Arbor or some other solution? Support - Does your provider have a true 24x7 security contact? Is there escalation? Can you reach someone clueful when you really need to? Business Practices - Are your providers so shifty that they will drag their feet on fixing a DDOS problem in order to get your 95% billing pegged to the capacity of your link? When we select providers based purely on cost, as some web hosters/network access providers tend to do, then you have to put up with deficiencies in these areas. As engineers we must be able to communicate these qualitative differences to the folks who are looking at the bottom line. You get what you pay for, most of the time. You almost never get what you don't pay for. - Dan On 6/19/04 10:04 PM, "Charles Sprickman" <spork@inch.com> wrote:
Howdy,
Is there any place where people with experience dealing with DDoS attacks hang out? I'm getting very little assistance from my upstream beyond "call whomever is in charge of each IP attacking and make them stop", and "even though we null route the destination IP being attacked, this traffic will be billed".
I've got a nice snippet of flows, so I can mostly see where everything is coming from, and it's obvious what the target is, but my flow-stat/flow-report skills are pretty weak.
Oddly, in eight years of working for smallish ISPs I've never been hit very hard, believe it or not. Is the response from my upstream typical? I was expecting a bit more cooperation rather than them seeing as this as an opportunity to bill me for lots of traffic.
Thanks,
Charles
-- Charles Sprickman spork@inch.com
participants (10)
-
Bubba Parker -
Charles Sprickman -
Christopher L. Morrow -
Daniel Golding -
Jonathan Slivko -
Laurence F. Sheldon, Jr. -
Michael Loftis -
Mike Lewinski -
Rubens Kuhl Jr. -
Suresh Ramasubramanian