Handling of Abuse Complaints
NANOG Community, I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack. My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go. I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future. Thanks, Jason
On Mon 2016-Aug-29 10:55:27 -0500, Jason Lee <jason.m.lee@gmail.com> wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
If you've informed them of the issue, given them time to resolve, and they've failed to take action, at some point you need to escalate and cauterize the wound to prevent abuse traffic spewing forth from the cusotmer's (and subsequently your) network. How you implement that specifically is your call, but I would at least start giving specific timelines to the customer and outline the steps that will be taken if they fail to remediate by those times in order to give them fair warning. I've been fairly specific previously about crafting filters to drop just the offending traffic, which should be doable here given the vector, but in other cases where it was obvious the offending hosts were simply compromised to hell and spewing myriad garbage traffic, I have cut users off completely to chop C&C access etc. This was during time at a regional commercial ISP on business circuits. -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
Thanks,
Jason
Dear Customer, Cyber criminals are using your network (and ours) to unlawfully attack other computers on the Internet. The specific security problem with your DNS server at 127.0.0.1 was first reported to you on Date1 (original message attached). Please be advised that we will interrupt network access to that server on Date2. This will likely disrupt your service. To avoid disruption, please contact me at Email with a mitigation plan no later than close of business Date3. I stand ready to assist any way that I can. Regards, Your Name On Mon, Aug 29, 2016 at 11:55 AM, Jason Lee <jason.m.lee@gmail.com> wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
Thanks,
Jason
-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
"unlawfully" is probably redundant, unless these are otherwise law-abiding cyber criminals. /pedant -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of William Herrin Sent: Monday, August 29, 2016 9:28 AM To: Jason Lee <jason.m.lee@gmail.com> Cc: nanog@nanog.org Subject: Re: Handling of Abuse Complaints Dear Customer, Cyber criminals are using your network (and ours) to unlawfully attack other computers on the Internet. The specific security problem with your DNS server at 127.0.0.1 was first reported to you on Date1 (original message attached). Please be advised that we will interrupt network access to that server on Date2. This will likely disrupt your service. To avoid disruption, please contact me at Email with a mitigation plan no later than close of business Date3. I stand ready to assist any way that I can. Regards, Your Name On Mon, Aug 29, 2016 at 11:55 AM, Jason Lee <jason.m.lee@gmail.com> wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
Thanks,
Jason
-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> This electronic mail transmission contains information from Warner Pacific Insurance Services that may be confidential or privileged. Such information is solely for the intended recipient, and use by any other party is not authorized. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this message, its contents or any attachments is prohibited. Any wrongful interception of this message is punishable as a Federal Crime. If you have received this message in error, please notify the sender immediately by telephone (800) 801-2300 or by electronic mail at postmaster@warnerpacific.com.
I would suggest that violation of the ISP’s ToS should also be consideration, since what may be illegal in one jurisdiction may not be illegal in some other jurisdictions. Repeated abuse and violations of an ISP’s ToS should also be a consideration to terminate a customer relationship, and ISPs are fully within their rights to take this type of action. - ferg
On Aug 29, 2016, at 9:31 AM, Gareth Tupper <Gareth.Tupper@warnerpacific.com> wrote:
"unlawfully" is probably redundant, unless these are otherwise law-abiding cyber criminals.
/pedant
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of William Herrin Sent: Monday, August 29, 2016 9:28 AM To: Jason Lee <jason.m.lee@gmail.com> Cc: nanog@nanog.org Subject: Re: Handling of Abuse Complaints
Dear Customer,
Cyber criminals are using your network (and ours) to unlawfully attack other computers on the Internet.
The specific security problem with your DNS server at 127.0.0.1 was first reported to you on Date1 (original message attached). Please be advised that we will interrupt network access to that server on Date2. This will likely disrupt your service.
To avoid disruption, please contact me at Email with a mitigation plan no later than close of business Date3.
I stand ready to assist any way that I can.
Regards, Your Name
On Mon, Aug 29, 2016 at 11:55 AM, Jason Lee <jason.m.lee@gmail.com> wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
Thanks,
Jason
-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
This electronic mail transmission contains information from Warner Pacific Insurance Services that may be confidential or privileged. Such information is solely for the intended recipient, and use by any other party is not authorized. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this message, its contents or any attachments is prohibited. Any wrongful interception of this message is punishable as a Federal Crime. If you have received this message in error, please notify the sender immediately by telephone (800) 801-2300 or by electronic mail at postmaster@warnerpacific.com.
— Paul Ferguson ICEBRG.io Seattle, Washington, USA
On Aug 29, 2016, at 9:37 AM, Paul Ferguson <fergdawgster@mykolab.com> wrote:
I would suggest that violation of the ISP’s ToS should also be consideration, since what may be illegal in one jurisdiction may not be illegal in some other jurisdictions.
Unless your abuse / security desk is staffed by lawyers it's probably better to avoid words like "criminal" and "unlawfully" altogether and stick to "in violation of our ToS".
Repeated abuse and violations of an ISP’s ToS should also be a consideration to terminate a customer relationship, and ISPs are fully within their rights to take this type of action.
And don't need to lean on "it's probably illegal" to do so, nor imply that if it were legal they wouldn't necessarily enforce their ToS. (All assuming that being abused as part of a dDoS reflector actually is against your ToS. If it's not things get more complex.) Cheers, Steve
- ferg
On Aug 29, 2016, at 9:31 AM, Gareth Tupper <Gareth.Tupper@warnerpacific.com> wrote:
"unlawfully" is probably redundant, unless these are otherwise law-abiding cyber criminals.
/pedant
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of William Herrin Sent: Monday, August 29, 2016 9:28 AM To: Jason Lee <jason.m.lee@gmail.com> Cc: nanog@nanog.org Subject: Re: Handling of Abuse Complaints
Dear Customer,
Cyber criminals are using your network (and ours) to unlawfully attack other computers on the Internet.
The specific security problem with your DNS server at 127.0.0.1 was first reported to you on Date1 (original message attached). Please be advised that we will interrupt network access to that server on Date2. This will likely disrupt your service.
To avoid disruption, please contact me at Email with a mitigation plan no later than close of business Date3.
I stand ready to assist any way that I can.
Regards, Your Name
On Mon, Aug 29, 2016 at 11:55 AM, Jason Lee <jason.m.lee@gmail.com> wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
Thanks,
Jason
-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
This electronic mail transmission contains information from Warner Pacific Insurance Services that may be confidential or privileged. Such information is solely for the intended recipient, and use by any other party is not authorized. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this message, its contents or any attachments is prohibited. Any wrongful interception of this message is punishable as a Federal Crime. If you have received this message in error, please notify the sender immediately by telephone (800) 801-2300 or by electronic mail at postmaster@warnerpacific.com.
— Paul Ferguson ICEBRG.io Seattle, Washington, USA
On Mon, Aug 29, 2016 at 12:47 PM, Steve Atkins <steve@blighty.com> wrote:
Unless your abuse / security desk is staffed by lawyers it's probably better to avoid words like "criminal" and "unlawfully" altogether
Not really an ambiguous situation IMHO, but whatever floats your boat. Bear in mind, though, that if you reasonably suspect your company is caught up in a specific violation of the law and you fail to validate and/or end the violation, your inaction brings liability on the company. Even though you're not a lawyer. That's true from the highest executive to the lowest janitor.
and stick to "in violation of our ToS".
This I would avoid. A ToS is a contract. Contracts are open to negotiation. The law is not. If you don't want to say "unlawfully attack," then stop at "attack." On Mon, Aug 29, 2016 at 1:04 PM, Laszlo Hanyecz <laszlo@heliacal.net> wrote:
I know this is against the popular religion here but how is this abuse on the part of your customer? Google, Level3 and many others also run open resolvers, because they're useful services. This is why we can't have nice things.
Google mitigates the attack vector with rate limiting through custom software. I would venture a guess that Jason's customer is not that sophisticated. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On 8/29/2016 11:47, Steve Atkins wrote:
Unless your abuse / security desk is staffed by lawyers it's probably better to avoid words like "criminal" and "unlawfully" altogether and stick to "in violation of our ToS".
Or "in violation of your contract (which includes, by reference, our TOS) with us." -- "Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid." --Albert Einstein From Larry's Cox account.
I know this is against the popular religion here but how is this abuse on the part of your customer? Google, Level3 and many others also run open resolvers, because they're useful services. This is why we can't have nice things. On 2016-08-29 15:55, Jason Lee wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
Thanks,
Jason
It's quite possible to operate an open resolver while still making it very difficult to use in an amplification attack - maybe coach your user into using rate limiting if you are particularly keen not to 'shape' their traffic at this stage. PowerDNS has a very powerful load balancer that can be used effectively although it's name escapes me now. PowerDNS 3x and 4x also has an effective anti spoofing mechanism. *Kind Regards,Lee Fuller* *PGP Fingerprint <https://leefuller.io/pgp/>: * 4ACAEBA4B9EE1B3A075034302D5C3D050E6ED55A On 29 August 2016 at 18:04, Laszlo Hanyecz <laszlo@heliacal.net> wrote:
I know this is against the popular religion here but how is this abuse on the part of your customer? Google, Level3 and many others also run open resolvers, because they're useful services. This is why we can't have nice things.
On 2016-08-29 15:55, Jason Lee wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
Thanks,
Jason
Google, Level 3 and the like's open DNS resolvers are strictly rate-limited. They can't be used as DDOS amplifiers. On the other hand, there are tons of open resolvers on the internet without any sort of limiting. These are very effective amplifiers. Regards, Filip On 29.8.2016 19:04, Laszlo Hanyecz wrote:
I know this is against the popular religion here but how is this abuse on the part of your customer? Google, Level3 and many others also run open resolvers, because they're useful services. This is why we can't have nice things.
On 2016-08-29 15:55, Jason Lee wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
Thanks,
Jason
There is a distance to travel between cant and cant effectively. Perhaps they can share how they ever so effectively have solved this conundrum. After all, they are apparently not getting any abuse reports ever. As an operator of several open resolvers (with rate limiting and automatic mitigation in effect) to support my customer base until the network landscape supports alternative approaches, I would like to know how they accomplished that little trick. Filip Hruska wrote:
Google, Level 3 and the like's open DNS resolvers are strictly rate-limited. They can't be used as DDOS amplifiers.
On the other hand, there are tons of open resolvers on the internet without any sort of limiting. These are very effective amplifiers.
Regards, Filip
On 29.8.2016 19:04, Laszlo Hanyecz wrote:
I know this is against the popular religion here but how is this abuse on the part of your customer? Google, Level3 and many others also run open resolvers, because they're useful services. This is why we can't have nice things.
On 2016-08-29 15:55, Jason Lee wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
Thanks,
Jason
On 08/29/2016 08:55 AM, Jason Lee wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
It depends on the nature of the complaint. If it's an amplification attack of some kind, figure out how the perp is doing it, and block it as appropriate. For example, do you filter incoming packets with source address of subnet network and broadcast (shorter than /30) and allnet (255.255.255.255) broadcast, and filter packets outbound with destinations of allnet broadcast? DNS and NTP can be tricked into generating packet storms. In particular, you may want to block excessive large DNS requests inbound using deep packet inspection at your edge. Not all abuse problems are the fault of the customer. You have to do your part as well.
In message <3dc3fd61-5123-0070-dd4e-435ce6785577@satchell.net>, Stephen Satchell writes:
On 08/29/2016 08:55 AM, Jason Lee wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
It depends on the nature of the complaint. If it's an amplification attack of some kind, figure out how the perp is doing it, and block it as appropriate. For example, do you filter incoming packets with source address of subnet network and broadcast (shorter than /30) and allnet (255.255.255.255) broadcast, and filter packets outbound with destinations of allnet broadcast?
DNS and NTP can be tricked into generating packet storms. In particular, you may want to block excessive large DNS requests inbound using deep packet inspection at your edge.
Not all abuse problems are the fault of the customer. You have to do your part as well.
I presume everyone of you is planning to install DNS servers that support RFC 7873 - DNS COOKIES? Yes, servers exist that support this and some TLD's are already using such servers (0.47%), Alexa .Gov and .AU servers (0.09%), Alexa Top 1000 (0.22%) and Alexa Bottom 1000 (.19%). Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Hi, On 29 August 2016 at 16:55, Jason Lee <jason.m.lee@gmail.com> wrote:
NANOG Community,
I was curious how various players in this industry handle abuse complaints. I'm drafting a policy for the service provider I'm working for about handing of complaints registered against customer IP space. In this example I have a customer who is running an open resolver and have received a few complaints now regarding it being used as part of a DDoS attack.
My initial response was to inform the customer and ask them to fix it. Now that its still ongoing over a month later, I'd like to take action to remediate the issue myself with ACLs but our customer facing team is pushing back and without an idea of what the industry best practice is, management isn't sure which way to go.
I'm hoping to get an idea of how others handle these cases so I can develop our formal policy on this and have management sign off and be able to take quicker action in the future.
As you are developing a policy and procedure, you might want to have a look at the resources provided (free) by the Messaging, Malware and Mobile Anti-Abuse Working Group (M3AAWG). Whilst not answering your question directly, it can be useful to have some general abuse best practice documents around when developing your own policies. Lots of resources are available at https://www.m3aawg.org/for-the-industry, including: - Best Common Practices for Hosting and Cloud Service Providers - Best Practices to Address Online, Mobile, and Telephony Threats - Feedback Reporting Recommendation - Overview of DNS Security - Port 53 Protection - Abuse Desk Common Practices - The Anti-Bot Code of Conduct for Internet Service Providers There's a lot of stuff about email and email spam (including a whole page on FBLs at https://www.m3aawg.org/fbl-resources), but there is some stuff there on abuse in other domains as well. It's well worth a gander. HTH, Alex
participants (14)
-
Alex Brooks
-
Filip Hruska
-
Gareth Tupper
-
Hugo Slabbert
-
Jason Lee
-
Joe Maimon
-
Larry Sheldon
-
Laszlo Hanyecz
-
Lee Fuller
-
Mark Andrews
-
Paul Ferguson
-
Stephen Satchell
-
Steve Atkins
-
William Herrin