Reporting DDOS reflection attacks
Like most small providers, we occasionally get hit by DoS attacks. We got hammered by an SSDP reflection attack (udp port 1900) last week. We took a 27 second log and from there extracted about 160k unique IPs. It is really difficult to find abuse emails for 160k IPs. We know about abuse.net but abuse.net requires hostnames, not IPs for lookups and not all IP addresses have valid DNS entries. The only other way we know of to report problems is to grab the abuse email addresses is whois. However, whois is not structured and is not set up to deal with this number of requests - even caching whois data based on subnets will result in many thousands of lookups. Long term it seems like structured data and some kind of authentication would be ideal for reporting attacks. But right now how should we be doing it?
On 8 Nov 2014, at 1:56, srn.nanog@prgmr.com wrote:
But right now how should we be doing it?
<http://www.team-cymru.org/Services/ip-to-asn.html> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Sat, Nov 8, 2014 at 2:00 AM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 8 Nov 2014, at 1:56, srn.nanog@prgmr.com wrote:
But right now how should we be doing it?
Once you get the ASN or at least the domain name of the ISP providing service to the reflecting host, several major reputable ISPs (including my employer, who I can't name because I'm not an official spokesperson) will welcome RFC 5070 "IODEF" reports for general network abuse and RFC 5965 "MARF" format for email abuse, directed to abuse@ the main domain for that ISP. http://www.ietf.org/rfc/rfc5070.txt http://www.ietf.org/rfc/rfc5965.txt -- Paul W Bennett
Out of curiosity, have any of you had luck reporting the sources of attacks to the admins of the origin ASNs? Any failure or success stories you can share? Macca On Sat, Nov 8, 2014 at 6:20 PM, Paul Bennett <paul.w.bennett@gmail.com> wrote:
On Sat, Nov 8, 2014 at 2:00 AM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 8 Nov 2014, at 1:56, srn.nanog@prgmr.com wrote:
But right now how should we be doing it?
Once you get the ASN or at least the domain name of the ISP providing service to the reflecting host, several major reputable ISPs (including my employer, who I can't name because I'm not an official spokesperson) will welcome RFC 5070 "IODEF" reports for general network abuse and RFC 5965 "MARF" format for email abuse, directed to abuse@ the main domain for that ISP.
http://www.ietf.org/rfc/rfc5070.txt
http://www.ietf.org/rfc/rfc5965.txt
-- Paul W Bennett
On 8 Nov 2014, at 17:09, McDonald Richards wrote:
Any failure or success stories you can share?
In my experience, it's the generally broadband access operators who will sometimes respond, when contacted about reflection/amplification attacks leveraging misconfigured, abusable CPE. Generally speaking, networks running misconfigured, abusable devices by definition aren't very actively managed - and so those operators don't often respond, unless one has personal contacts within the relevant group(s). YMMV, of course. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
I can offer an indirect story, and not quite a reflection attack, but a DDoS one. We happen to have a host that had an IPMI board exposed to the net, that got compromised, and became a vector for a DDoS attack. The target reported the attack to at least some of the sources, including Windstream/Hosted Solutions, where this particular server is located. They contacted me, and I dealt with things with about a 1-hour turn-around from when a trouble ticket hit my inbox (well, still dealing with things - that IPMI card is offline until I get around to securing it, and it's the occasional reboot-by-phone-call until then). So at least one small success. Miles Fidelman McDonald Richards wrote:
Out of curiosity, have any of you had luck reporting the sources of attacks to the admins of the origin ASNs?
Any failure or success stories you can share?
Macca
On Sat, Nov 8, 2014 at 6:20 PM, Paul Bennett <paul.w.bennett@gmail.com> wrote:
On Sat, Nov 8, 2014 at 2:00 AM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 8 Nov 2014, at 1:56, srn.nanog@prgmr.com wrote:
But right now how should we be doing it? <http://www.team-cymru.org/Services/ip-to-asn.html> Once you get the ASN or at least the domain name of the ISP providing service to the reflecting host, several major reputable ISPs (including my employer, who I can't name because I'm not an official spokesperson) will welcome RFC 5070 "IODEF" reports for general network abuse and RFC 5965 "MARF" format for email abuse, directed to abuse@ the main domain for that ISP.
http://www.ietf.org/rfc/rfc5070.txt
http://www.ietf.org/rfc/rfc5965.txt
-- Paul W Bennett
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
On 11/07/2014 11:20 PM, Paul Bennett wrote:
On Sat, Nov 8, 2014 at 2:00 AM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 8 Nov 2014, at 1:56, srn.nanog@prgmr.com wrote:
But right now how should we be doing it?
Once you get the ASN or at least the domain name of the ISP providing service to the reflecting host, several major reputable ISPs (including my employer, who I can't name because I'm not an official spokesperson) will welcome RFC 5070 "IODEF" reports for general network abuse and RFC 5965 "MARF" format for email abuse, directed to abuse@ the main domain for that ISP.
Thanks, the IP->subnet/ASN lookup and rfc5070 look like exactly what we need to start with. I'm fairly certain it would have gotten us the same contact for all the IPs we reported last week. Since IODEF is so flexible, are there any exact guidelines or examples on how to use it to report a DDOS? For example, should there be a separate XML document for each prefix or one for the entire list? What I found was https://tools.ietf.org/html/draft-ietf-mile-iodef-guidance-03#page-21 but it could use some more explanation.
Hey, We've been hit on/off with large scale amplification attacks over the last few years. We found looking up src ASN of the attack and reporting is not super helpful, as many blocks come from sub allocations and you'll just get redirected to someone else. This will just cause more overhead and legwork for you in the long run. Whois data *seems* to be a little more reliable, and there's an abuseEmail script out there that helps automate the abuse contact lookup ( http://abuseemail.sourceforge.net/ ). We've added a bit of logic in front of this to aggregate the flows per destination abuse email, then send a report with all listed flows + timestamp. Feel free to ping me offlist if you want some more info on this. /Ruairi On 7 November 2014 18:56, <srn.nanog@prgmr.com> wrote:
Like most small providers, we occasionally get hit by DoS attacks. We got hammered by an SSDP reflection attack (udp port 1900) last week. We took a 27 second log and from there extracted about 160k unique IPs.
It is really difficult to find abuse emails for 160k IPs.
We know about abuse.net but abuse.net requires hostnames, not IPs for lookups and not all IP addresses have valid DNS entries.
The only other way we know of to report problems is to grab the abuse email addresses is whois. However, whois is not structured and is not set up to deal with this number of requests - even caching whois data based on subnets will result in many thousands of lookups.
Long term it seems like structured data and some kind of authentication would be ideal for reporting attacks. But right now how should we be doing it?
On 11/08/2014 03:30 AM, Ruairi Carroll wrote:
Whois data *seems* to be a little more reliable, and there's an abuseEmail script out there that helps automate the abuse contact lookup ( http://abuseemail.sourceforge.net/ ).
I believe this script is out of date and I would not use this script without doing a thorough review/update. For example, 100.43.102.0/24 is reported to be reserved but whois clearly shows that it is allocated to Xplornet Communications Inc. Then when I remove the reserved allocation from the script, the abuse email returned is arin.net rather than xplornet.com. Using dig +short 102.43.100.origin.asn.cymru.com TXT and then whois as22995 would have gotten me the same abuse email address as what I originally found.
I've used https://abusix.com/contactdb.html Be prepared for a lot of backscatter. You'll get autoresponders, automated ticketing systems sending frequent updates, bounce messages (from full abuse@ inboxes), and be surveyed for how well they're not performing. Also, be prepared for ISPs / hosting providers to ask for additional information, like logs proving the attack came from their customer. Oh, and be prepared to feel sorry for their customers whose VMs are deleted for "hacking", rather than being informed of their misconfiguration. On the bright side, some 10% will actually correct the problem, thereby costing the attacker a few minutes of work to re-scan for active amplifiers. :P Damian Professional Pessimist On Fri, Nov 7, 2014 at 10:56 AM, <srn.nanog@prgmr.com> wrote:
Like most small providers, we occasionally get hit by DoS attacks. We got hammered by an SSDP reflection attack (udp port 1900) last week. We took a 27 second log and from there extracted about 160k unique IPs.
It is really difficult to find abuse emails for 160k IPs.
We know about abuse.net but abuse.net requires hostnames, not IPs for lookups and not all IP addresses have valid DNS entries.
The only other way we know of to report problems is to grab the abuse email addresses is whois. However, whois is not structured and is not set up to deal with this number of requests - even caching whois data based on subnets will result in many thousands of lookups.
Long term it seems like structured data and some kind of authentication would be ideal for reporting attacks. But right now how should we be doing it?
Also, abusix is not completely accurate (and they've never responded to my emails reporting problems). For example, any IPs from apnic and nic.ad.jp return the registry's abuse address, which doesn't do anything. Don't forget about all the providers with incorrect abuse contacts, or providers that require you to fill out some form, or providers that auto-respond with messages saying it's not their IP space (I'm looking at you charter... 71.90.222.x is definitely your space, despite what your abuse system thinks). Some tips: 1) Verify the servers are still vulnerable. This is pretty straightforward, and saves everyone involved some time 2) Your abuse emails should include tcpdump-like output (or you'll get tons of replies asking for logs) 3) Sticking to one abusive IP per email seems to get the best response rate (or you confuse all the automated systems for parsing these) 4) We provide instructions for fixing the issue for some common software... this seems to help some of the people who have no idea what they are doing. 5) Make sure you don't send this from your email address. It should definitely be it's own mailbox due to volume of bounces and autoreplies you'll see. Don't expect that sending abuse emails is going to have any noticeable effect on the size of the attacks you see. The openresolverproject stats show the scope of the issue: http://openresolverproject.org/breakdown.cgi On 11/8/2014 5:48 PM, Damian Menscher wrote:
I've used https://abusix.com/contactdb.html
Be prepared for a lot of backscatter. You'll get autoresponders, automated ticketing systems sending frequent updates, bounce messages (from full abuse@ inboxes), and be surveyed for how well they're not performing.
Also, be prepared for ISPs / hosting providers to ask for additional information, like logs proving the attack came from their customer.
Oh, and be prepared to feel sorry for their customers whose VMs are deleted for "hacking", rather than being informed of their misconfiguration.
On the bright side, some 10% will actually correct the problem, thereby costing the attacker a few minutes of work to re-scan for active amplifiers. :P
Damian Professional Pessimist
On Fri, Nov 7, 2014 at 10:56 AM, <srn.nanog@prgmr.com> wrote:
Like most small providers, we occasionally get hit by DoS attacks. We got hammered by an SSDP reflection attack (udp port 1900) last week. We took a 27 second log and from there extracted about 160k unique IPs.
It is really difficult to find abuse emails for 160k IPs.
We know about abuse.net but abuse.net requires hostnames, not IPs for lookups and not all IP addresses have valid DNS entries.
The only other way we know of to report problems is to grab the abuse email addresses is whois. However, whois is not structured and is not set up to deal with this number of requests - even caching whois data based on subnets will result in many thousands of lookups.
Long term it seems like structured data and some kind of authentication would be ideal for reporting attacks. But right now how should we be doing it?
On 11/09/2014 09:31 AM, Brian Rak wrote:
Some tips: 1) Verify the servers are still vulnerable. This is pretty straightforward, and saves everyone involved some time For a DDOS, I'd be concerned that the provider would now think my activity was malicious.
2) Your abuse emails should include tcpdump-like output (or you'll get tons of replies asking for logs) Is the output from nfdump close enough?
3) Sticking to one abusive IP per email seems to get the best response rate (or you confuse all the automated systems for parsing these) The smallest email abuse report I sent last week contained over 15,000 IPs. Is it really better to send that many emails?
Do you know if third-parties such as SANS ISC or ShadowServer take lists of IPs? Frank -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of srn.nanog@prgmr.com Sent: Friday, November 07, 2014 12:57 PM To: nanog@nanog.org Subject: Reporting DDOS reflection attacks Like most small providers, we occasionally get hit by DoS attacks. We got hammered by an SSDP reflection attack (udp port 1900) last week. We took a 27 second log and from there extracted about 160k unique IPs. It is really difficult to find abuse emails for 160k IPs. We know about abuse.net but abuse.net requires hostnames, not IPs for lookups and not all IP addresses have valid DNS entries. The only other way we know of to report problems is to grab the abuse email addresses is whois. However, whois is not structured and is not set up to deal with this number of requests - even caching whois data based on subnets will result in many thousands of lookups. Long term it seems like structured data and some kind of authentication would be ideal for reporting attacks. But right now how should we be doing it?
Another DDoS/DoS email thread in progress, ah?... these seem to occur often lately... So....Perfect timing to remind all in the list that there is a NANOG BCOP in the works on this topic. Some of us have been working on documenting our collective knowledge about real practices that can help our community deal with this annoying networking decease...in a vendor agnostic manner... Our DDoS/DoS attack Best Common Ops Practices doc seeks to provide community-wide guidelines on what to do before, during and after a DDoS/DoS attack. If any of you want to contribute and join us to help the community on what we have documented so far, please check out the document below and/or drop me a note... http://bcop.nanog.org/index.php/BCOP_Drafts Yardiel Fuentes yardiel@gmail.com twitter: #techguane On Nov 8, 2014, at 6:19 PM, Frank Bulk wrote:
Do you know if third-parties such as SANS ISC or ShadowServer take lists of IPs?
Frank
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of srn.nanog@prgmr.com Sent: Friday, November 07, 2014 12:57 PM To: nanog@nanog.org Subject: Reporting DDOS reflection attacks
Like most small providers, we occasionally get hit by DoS attacks. We got hammered by an SSDP reflection attack (udp port 1900) last week. We took a 27 second log and from there extracted about 160k unique IPs.
It is really difficult to find abuse emails for 160k IPs.
We know about abuse.net but abuse.net requires hostnames, not IPs for lookups and not all IP addresses have valid DNS entries.
The only other way we know of to report problems is to grab the abuse email addresses is whois. However, whois is not structured and is not set up to deal with this number of requests - even caching whois data based on subnets will result in many thousands of lookups.
Long term it seems like structured data and some kind of authentication would be ideal for reporting attacks. But right now how should we be doing it?
On 9 Nov 2014, at 6:46, Yardiel D. Fuentes wrote:
There are some good general recommendations in this document (Word format? Really?), but this is incorrect and harmful, and should be removed: iii. Consider dropping any DNS reply packets which are larger than 512 Bytes – these are commonly found in DNS DoS Amplification attacks. This *breaks the Internet*. Don't do it. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On 11/8/14 6:33 PM, Roland Dobbins wrote:
this is incorrect and harmful, and should be removed:
iii. Consider dropping any DNS reply packets which are larger than 512 Bytes – these are commonly found in DNS DoS Amplification attacks.
This *breaks the Internet*. Don't do it.
+1
On 9November2014Sunday, at 11:40, Doug Barton <dougb@dougbarton.us> wrote:
On 11/8/14 6:33 PM, Roland Dobbins wrote:
this is incorrect and harmful, and should be removed:
iii. Consider dropping any DNS reply packets which are larger than 512 Bytes – these are commonly found in DNS DoS Amplification attacks.
This *breaks the Internet*. Don't do it.
+1
actually, if you think this will help you, by all means drop any DNS packets which are gt. 512bytes, not UDP, and not IPv4. /bill
participants (12)
-
Brian Rak -
Damian Menscher -
Doug Barton -
Frank Bulk -
manning bill -
McDonald Richards -
Miles Fidelman -
Paul Bennett -
Roland Dobbins -
Ruairi Carroll -
srn.nanog＠prgmr.com -
Yardiel D. Fuentes