Hello, I provide broadband connectivity to mostly residential users. Over the past few years, instances of DDoS against the network - specfically targeting end users - has been on the rise, and today I can qualify many of these as simple acts of revenge where someone will engage a dos (possibly, services like 'booters' or similar) because they lost an online game or had some interactive in a forum they didn't like. I have good 'consumer broadband' filtering rules in place which make sense and protect against quite a lot of obviously ddos oriented traffic streams. The next step I want to engage, for those types of traffic which I can positively identify as not spoofed, is to send out abuse reports to owners of ip ranges used to launch these attacks. Ideally I'd like to be able to write up some form letter describing the attack, the source ip(s) of note, some disassembled sample packets, and then feed a list of IP source addresses and have it mail it out to the abuse contact at each source network. I am wondering if anyone has a pointer or reference to any tools which might help facillitate this? Thank you. Mike-
We need to come up with some sort of international Abuse Reduction and Reporting Engagement Suite of Tools as a Service. M. Original Message From: Mike Sent: Tuesday, November 18, 2014 19:59 To: nanog@nanog.org Subject: abuse reporting tools Hello, I provide broadband connectivity to mostly residential users. Over the past few years, instances of DDoS against the network - specfically targeting end users - has been on the rise, and today I can qualify many of these as simple acts of revenge where someone will engage a dos (possibly, services like 'booters' or similar) because they lost an online game or had some interactive in a forum they didn't like. I have good 'consumer broadband' filtering rules in place which make sense and protect against quite a lot of obviously ddos oriented traffic streams. The next step I want to engage, for those types of traffic which I can positively identify as not spoofed, is to send out abuse reports to owners of ip ranges used to launch these attacks. Ideally I'd like to be able to write up some form letter describing the attack, the source ip(s) of note, some disassembled sample packets, and then feed a list of IP source addresses and have it mail it out to the abuse contact at each source network. I am wondering if anyone has a pointer or reference to any tools which might help facillitate this? Thank you. Mike-
On 11/18/2014 8:11 PM, Michael Brown wrote:
We need to come up with some sort of international Abuse Reduction and Reporting Engagement Suite of Tools as a Service.
M.
I've been considering a post for a couple of weeks but decided most of my complaints were petty. I've been getting lots of "ssh attacks against my network" emails from various people on the internet. All of them have no standard for what logs they show or what format they show them in, or what format the whole email is in, so frequently I'm being told "Trust me, based on this one connection attempt to this non-qualified hostname that occured on this non-TZ timestamp, you need to stop your users abuse." Immediately thereafter they tell me the IP address has already been blocked in their firewall for an unspecified length of time and give no routes for amelioration. So I'm left with a very unsatisfactory feeling of either shutting down a possibly innocent customer based on a machines word, or attempting to start a dialog with random_script_user_99@hotmail.com. I suspect someone is going to pipe up in a second and say that there is a suite of tools, but the real problem is that nobody is using it. Robert
On Tue, Nov 18, 2014 at 7:41 PM, Robert Drake <rdrake@direcpath.com> wrote:
On 11/18/2014 8:11 PM, Michael Brown wrote: [snip] amelioration. So I'm left with a very unsatisfactory feeling of either shutting down a possibly innocent customer based on a machines word, or attempting to start a dialog with random_script_user_99@hotmail.com.
Under those circumstances, how do you know it's not a social-engineering based DoS being attempted? Preferably, take no action to shutdown services without decent confirmation; as malicious reports of a fraudulent, bogus, dramatized, or otherwise misleading nature are sometimes used by malicious actors to target a legitimate user. My suggestion would be table the report of a single SSH connection and really do nothing with it. If there is actually abuse being conducted, you should either be able to independently verify the actual abuse, e.g. by checking packet level data or netflow data, or you should begin to receive a pattern of complaints; more unique contacts, that you can investigate and verify are legit. contacts from unique networks. If neither occurs, then just keep a log as an unconfirmed abuse report, which if unconfirmed for a few days may be forwarded to the end user for their information/records. -- -JH
Robert
On Tue, Nov 18, 2014 at 7:41 PM, Robert Drake <rdrake@direcpath.com> wrote:
On 11/18/2014 8:11 PM, Michael Brown wrote: [snip] amelioration. So I'm left with a very unsatisfactory feeling of either shutting down a possibly innocent customer based on a machines word, or attempting to start a dialog with random_script_user_99@hotmail.com.
Under those circumstances, how do you know it's not a social-engineering based DoS being attempted? Preferably, take no action to shutdown services without decent confirmation; as malicious reports of a fraudulent, bogus, dramatized, or otherwise misleading nature are sometimes used by malicious actors to target a legitimate user.
My suggestion would be table the report of a single SSH connection and really do nothing with it. If there is actually abuse being conducted, you should either be able to independently verify the actual abuse, e.g. by checking packet level data or netflow data, or you should begin to receive a pattern of complaints; more unique contacts, that you can investigate and verify are legit. contacts >>from unique networks.
If you know the destination IP address that the traffic is supposedly going to you can also just ACL it, that way if it's a customer of a customer you don't shut down the customer's entire business over something one person downstream is doing and you 'fix' the issue at the same time. The right answer really depends on how responsive your customer is to the complaints in the first place. -Drew
Some folks might disagree with this, but if it's an important service that I have running on a network, I will block a series of garbage AS's (closer to /8 the better) at the firewall (not at the edge) and that reduces the headaches by 50%. This isn't practical at the edge, but for system administration is the only way I have found to minimize the problem. A lot of times the owners of these IPs don't really care and won't take action. For example, the amount of garbage that comes out of FDC Servers in Chicago at times and not much is done. On Tue, Nov 18, 2014 at 6:58 PM, Mike <mike-nanog@tiedyenetworks.com> wrote:
Hello,
I provide broadband connectivity to mostly residential users. Over the past few years, instances of DDoS against the network - specfically targeting end users - has been on the rise, and today I can qualify many of these as simple acts of revenge where someone will engage a dos (possibly, services like 'booters' or similar) because they lost an online game or had some interactive in a forum they didn't like. I have good 'consumer broadband' filtering rules in place which make sense and protect against quite a lot of obviously ddos oriented traffic streams. The next step I want to engage, for those types of traffic which I can positively identify as not spoofed, is to send out abuse reports to owners of ip ranges used to launch these attacks. Ideally I'd like to be able to write up some form letter describing the attack, the source ip(s) of note, some disassembled sample packets, and then feed a list of IP source addresses and have it mail it out to the abuse contact at each source network. I am wondering if anyone has a pointer or reference to any tools which might help facillitate this?
Thank you.
Mike-
Just wait for GigE-everywhere. I am almost sure that these new Gig-to-the-toaster residential installs have very little rate filtering (or abuse response); let's hope that oversubscription solves the issue handily as it has traditionally. /kc On Tue, Nov 18, 2014 at 08:19:01PM -0600, Rafael Possamai said:
Some folks might disagree with this, but if it's an important service that I have running on a network, I will block a series of garbage AS's (closer to /8 the better) at the firewall (not at the edge) and that reduces the headaches by 50%. This isn't practical at the edge, but for system administration is the only way I have found to minimize the problem. A lot of times the owners of these IPs don't really care and won't take action. For example, the amount of garbage that comes out of FDC Servers in Chicago at times and not much is done.
On Tue, Nov 18, 2014 at 6:58 PM, Mike <mike-nanog@tiedyenetworks.com> wrote:
Hello,
I provide broadband connectivity to mostly residential users. Over the past few years, instances of DDoS against the network - specfically targeting end users - has been on the rise, and today I can qualify many of these as simple acts of revenge where someone will engage a dos (possibly, services like 'booters' or similar) because they lost an online game or had some interactive in a forum they didn't like. I have good 'consumer broadband' filtering rules in place which make sense and protect against quite a lot of obviously ddos oriented traffic streams. The next step I want to engage, for those types of traffic which I can positively identify as not spoofed, is to send out abuse reports to owners of ip ranges used to launch these attacks. Ideally I'd like to be able to write up some form letter describing the attack, the source ip(s) of note, some disassembled sample packets, and then feed a list of IP source addresses and have it mail it out to the abuse contact at each source network. I am wondering if anyone has a pointer or reference to any tools which might help facillitate this?
Thank you.
Mike-
-- Ken Chase - math@sizone.org
On Tue, 18 Nov 2014 16:58:24 -0800 Mike <mike-nanog@tiedyenetworks.com> wrote:
I provide broadband connectivity to mostly residential users. Over the past few years, instances of DDoS against the network - specfically targeting end users - has been on the rise, and today I can qualify many of these as simple acts of revenge where someone will engage a dos (possibly, services like 'booters' or similar) because they lost an online game or had some interactive in a forum they didn't like.
Hi Mike, I certainly sympathize with you about dealing with this sort of activity. Since you seem to be willing to invest some effort into mitigating it, what would also be interesting is to compile a summary of this activity that you're seeing. Answering questions such as how often does it happen, the duration when it does, what games are most commonly associated with the attacks you're seeing, what are the attack characteristics and so on. Having good insight into these attacks in formulating responses or going off and performing their own research to get closer to the who, why and how so they can be mitigated in other ways too. If you ever attend a NANOG, a presentation about your experiences might be welcome, it would very likely be in the security track, which I sometimes help moderate if you want to consider it.
I have good 'consumer broadband' filtering rules in place which make sense and protect against quite a lot of obviously ddos oriented traffic streams.
Do you ever find that the attacks overwhelm your network or are they usually just big enough to disrupt your downstream customer?
I am wondering if anyone has a pointer or reference to any tools which might help facillitate this?
I can point you to some tools and references I'm aware of, but I can't talk about how effectively they are operationally or whether or not you should abide by or use them. AbuseHelper <http://abusehelper.be/> IETF RFC 5965 An Extensible Format for Email Feedback Reports <https://tools.ietf.org/html/rfc5965> IETF RFC 6650 Creation and Use of Email Feedback Reports <https://tools.ietf.org/html/rfc6650> Network Abuse Reporting 2.0 <http://www.x-arf.org/> Net::Abuse::Utils <http://search.cpan.org/~mikegrb/Net-Abuse-Utils/> John
On Wed, Nov 19, 2014 at 12:14 PM, John Kristoff <jtk@cymru.com> wrote:
On Tue, 18 Nov 2014 16:58:24 -0800 Mike <mike-nanog@tiedyenetworks.com> wrote:
I provide broadband connectivity to mostly residential users.
I can point you to some tools and references I'm aware of, but I can't talk about how effectively they are operationally or whether or not you should abide by or use them.
Don't forget IETF RFC 5970 "IODEF" format as well. It provides a much more comprehensive and flexible reporting format than either X-ARF or RFC 5965 (both of which are really geared primarily towards single badguy / single incident). With that power comes greater complexity, though. I'll have to look at Net::Abuse::Utils since that's the first I've ever heard of it and I don't know what it can do. If it can't make IODEF, I'm a capable Perl programmer, so I can take a look, but no promises. -- Paul W Bennett
On Nov 19, 2014, at 9:14 AM, John Kristoff <jtk@cymru.com> wrote:
On Tue, 18 Nov 2014 16:58:24 -0800 Mike <mike-nanog@tiedyenetworks.com> wrote:
I am wondering if anyone has a pointer or reference to any tools which might help facillitate this?
I can point you to some tools and references I'm aware of, but I can't talk about how effectively they are operationally or whether or not you should abide by or use them.
AbuseHelper <http://abusehelper.be/>
IETF RFC 5965 An Extensible Format for Email Feedback Reports <https://tools.ietf.org/html/rfc5965>
IETF RFC 6650 Creation and Use of Email Feedback Reports <https://tools.ietf.org/html/rfc6650>
Network Abuse Reporting 2.0 <http://www.x-arf.org/>
Net::Abuse::Utils <http://search.cpan.org/~mikegrb/Net-Abuse-Utils/>
You can also use this facility to find the abuse email address of an IP https://abusix.com/contactdb.html And I wrote this tool, tailored for DMARC failure reports, but it has some code to report any email. Yo could hack the code easily for your own purposes: https://github.com/linkedin/lafayette/
Inspired by this thread (and other recent similar ones about how hard it is to report abuse in the right format to the right people), I've decided I'm going to start work on the Perl module presumed by this gist ... https://gist.github.com/PWBENNETT/18970413677c5df79c6a Reporting network abuse should be *EASY*. Say it with me ... *EASY*. No promises, at this stage, but I thought some of you would like to know that this project is at least in the pre-planning stages. -- Paul W Bennett
On Thu, Nov 20, 2014 at 6:44 AM, Paul Bennett <paul.w.bennett@gmail.com> wrote:
Inspired by this thread (and other recent similar ones about how hard it is to report abuse in the right format to the right people), I've decided I'm going to start work on [a] Perl module
Well ... preliminary ground work has started. It's not much, yet, but it's out there _just_ enough to (hopefully) prove I'm serious about this https://github.com/PWBENNETT/Net-Abuse-Reporter Patches / collaboration more than welcome, if you think you can glean what's going on inside my head (related to this project, at least). -- Paul W Bennett
First please filter the source addr on all egress traffic, please. Please. Second, please don’t be the network admin whom emails: “… To: notOurOrgAbuseEmail@tocici.com From: cluelessAdmin@example.com Subject: An attempt of intrusion comes from your ip . …” Just in case you missed the obvious: message body was empty, $cluelessAdmin didn’t do a basic whois for our OrgAbuseEmail, and $cluelessAdmin ASSumed we knew which of our 2,048 IPs apparently started WWIII while providing absolutely zero collaborating evidence (attaching or linking to raw tcpdump is very nice, “-d” is Ok too). We often receive dozens of these totally useless/blank emails, in clusters of a few minutes. Tricks like that earn an instant 144-hour null route badge for whichever sending company’s entire presumed netblock (if we can’t find an obvious AS), repeat offenses earn longer and more colorful badges. All get a personal voicemail to the $cluelessAdmin company’s exec(s)/admin(s). I deliver these voicemails roughly three times a week now. Teh Stupid leaves burn marks on our NOC techs, and the poor geeks can only take so much! Other suggestions, such as watching and responding to s/NetFlow spikes, or tracking/linking multiple complaining networks before even attempting to look at origins…these sometimes warrant a followup depending upon volume and frequency (easily tracked with an SQLLite + PHP-based tool/api). We’ve found things are more-often just fat fingers, someone more bored than harmful, or someone that hasn’t figured out zmap options yet. As for a genuine DDoS, with a spoofed-source - can you really do much about this? For years we’ve just automatically null-routed (+RTBH) the ingress target (and, if obvious, any egress source) for a shortish random() period, and everyone typically gets bored shortly thereafter. Our current null-route based homegrown DDoS mitigation platform requires barely ~10 seconds from detection/onset to mitigation, so we tend to elimianate most fun and drama pretty quickly. For more business-focused clients, services like CloudFlare typically keeps DDoS attacks off ingress IPs. (BTW: in addition business sites, we host Minecraft, Teamspeak, and other "l33t hax0r” targeted services) Gregg Berkholtz
On Nov 18, 2014, at 4:58 PM, Mike <mike-nanog@tiedyenetworks.com> wrote:
Hello,
I provide broadband connectivity to mostly residential users. Over the past few years, instances of DDoS against the network - specfically targeting end users - has been on the rise, and today I can qualify many of these as simple acts of revenge where someone will engage a dos (possibly, services like 'booters' or similar) because they lost an online game or had some interactive in a forum they didn't like. I have good 'consumer broadband' filtering rules in place which make sense and protect against quite a lot of obviously ddos oriented traffic streams. The next step I want to engage, for those types of traffic which I can positively identify as not spoofed, is to send out abuse reports to owners of ip ranges used to launch these attacks. Ideally I'd like to be able to write up some form letter describing the attack, the source ip(s) of note, some disassembled sample packets, and then feed a list of IP source addresses and have it mail it out to the abuse contact at each source network. I am wondering if anyone has a pointer or reference to any tools which might help facillitate this?
Thank you.
Mike-
participants (11)
-
Drew Weaver -
Franck Martin -
Gregg Berkholtz -
Jimmy Hess -
John Kristoff -
Ken Chase -
Michael Brown -
Mike -
Paul Bennett -
Rafael Possamai -
Robert Drake