Mitigating DNS amplification attacks
Hi! I was wondering if anyone had any experience with dealing with open resolvers as a web hoster? We currently have some 40,000 ip's that respond to DNS in our AS, the majority of which are not "open" but do reply with a referral to the root zones. We've been sending emails to our clients but as the servers are not managed by us, there's not much we can do at that level. Recently we've seen a large increase in the number and volume of DNS amplification DDOS's that are being reflected off of our AS. Just today we've seen at least 6 different attacks with between 4 and 10gbps leaving our AS each time. It's not really causing us issues at the moment because we have the capacity, but I'd hate to be on the receiving side. (and indeed, have been on the receiving side in the past, so I know how much it can suck) Has anyone ever tried mitigating/rate-limiting/etc these attacks in the network before? (vs at the server/application level) We have an Arbor peakflow device, but it's not really geared for this scenario I find. It will detect the outgoing attack via the flows, but all we can really do is null-route the victims ip in our AS. Ideally we would need a way to rate-limit DNS packets based on source ip. Maybe a linux box that handles dropping packets from the same source-ip over 1000/sec with some policy-based routing sending the DNS traffic to it? Does such a box exist already? If anyone has any ideas or suggestions, then by all means! There must be a better way to do this, and I'd really like to avoid re-inventing the wheel if it's been invented already. :) Thanks! Thomas
On May 1, 2013, at 6:43 AM, Thomas St-Pierre wrote:
We've been sending emails to our clients but as the servers are not managed by us, there's not much we can do at that level.
Sure, there is - shut them down if they don't comply. Most ISPs have AUP verbiage which would apply to a situation of this type.
Has anyone ever tried mitigating/rate-limiting/etc these attacks in the network before? (vs at the server/application level)
QoS doesn't work, as the programmatically-generated attack traffic 'crowds out' legitimate requests.
We have an Arbor peakflow device, but it's not really geared for this scenario I find.
Peakflow SP is a NetFlow-based anomaly-detection system which performs attack detection/classification/traceback. Please feel free to ping me offlist about additional system elements which perform attack mitigation. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Hi! On 13-04-30 7:57 PM, "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On May 1, 2013, at 6:43 AM, Thomas St-Pierre wrote:
We've been sending emails to our clients but as the servers are not managed by us, there's not much we can do at that level.
Sure, there is - shut them down if they don't comply. Most ISPs have AUP verbiage which would apply to a situation of this type.
Unfortunately I somehow doubt management is going to look favourably on a request to shut down so many clients. :( The large majority of the servers being used in the attacks are not open resolvers. Just DNS servers that are authoritative for a few domains, and the default config of the dns application does referrals to root for anything else. Yes there are ways of protecting against this on the server itself, but I don't see it happening here given the complexity of many of the solutions. I hate to say it, but if it's not "next -> next -> next -> finish", or integrated as an option in one of the common web hosting panels (cPanel, Plesk, etc) people won't do it. We still struggle just getting people to close actual open resolvers, and that is easy to configure.
Has anyone ever tried mitigating/rate-limiting/etc these attacks in the network before? (vs at the server/application level)
QoS doesn't work, as the programmatically-generated attack traffic 'crowds out' legitimate requests.
We have an Arbor peakflow device, but it's not really geared for this scenario I find.
Peakflow SP is a NetFlow-based anomaly-detection system which performs attack detection/classification/traceback. Please feel free to ping me offlist about additional system elements which perform attack mitigation.
Pinged off-list! Thanks! Thomas
On Tue, Apr 30, 2013 at 5:28 PM, Thomas St-Pierre <tstpierre@iweb.com>wrote:
On 13-04-30 7:57 PM, "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On May 1, 2013, at 6:43 AM, Thomas St-Pierre wrote:
We've been sending emails to our clients but as the servers are not managed by us, there's not much we can do at that level.
Sure, there is - shut them down if they don't comply. Most ISPs have AUP verbiage which would apply to a situation of this type.
Unfortunately I somehow doubt management is going to look favourably on a request to shut down so many clients. :( The large majority of the servers being used in the attacks are not open resolvers. Just DNS servers that are authoritative for a few domains, and the default config of the dns application does referrals to root for anything else.
Offering a DNS service to your customers may allow you to provide a good alternative to push those customers onto. You can then manage it properly. But I think DNS isn't the real issue here, it's the fact you're receiving spoofed traffic. I'd start by tracking the attacks backwards through your upstreams, as obviously someone in the path isn't enforcing BCP 38. Stop the spoof capability and the attacks will stop. It requires less effort overall (vs your counterparts at every hosting provider needing to solve the problem for their networks) and provides the best benefit to the victims. Damian
Hi Damian! We offer a DNS hosted solution, most people still use their own servers though. (especially those with control panels such as cPanel or plesk, where it's built-in). As for BCP38, I would love to stop the spoofed packets, however with them coming from our upstreams, (Level3, Cogent, Tata, etc) I don't see how we can. Thanks!, Thomas From: Damian Menscher <damian@google.com<mailto:damian@google.com>> Date: Tuesday, 30 April, 2013 8:32 PM To: "Thomas St.Pierre" <tstpierre@iweb.com<mailto:tstpierre@iweb.com>> Cc: "Dobbins, Roland" <rdobbins@arbor.net<mailto:rdobbins@arbor.net>>, NANOG list <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Mitigating DNS amplification attacks On Tue, Apr 30, 2013 at 5:28 PM, Thomas St-Pierre <tstpierre@iweb.com<mailto:tstpierre@iweb.com>> wrote: On 13-04-30 7:57 PM, "Dobbins, Roland" <rdobbins@arbor.net<mailto:rdobbins@arbor.net>> wrote:
On May 1, 2013, at 6:43 AM, Thomas St-Pierre wrote:
We've been sending emails to our clients but as the servers are not managed by us, there's not much we can do at that level.
Sure, there is - shut them down if they don't comply. Most ISPs have AUP verbiage which would apply to a situation of this type.
Unfortunately I somehow doubt management is going to look favourably on a request to shut down so many clients. :( The large majority of the servers being used in the attacks are not open resolvers. Just DNS servers that are authoritative for a few domains, and the default config of the dns application does referrals to root for anything else. Offering a DNS service to your customers may allow you to provide a good alternative to push those customers onto. You can then manage it properly. But I think DNS isn't the real issue here, it's the fact you're receiving spoofed traffic. I'd start by tracking the attacks backwards through your upstreams, as obviously someone in the path isn't enforcing BCP 38. Stop the spoof capability and the attacks will stop. It requires less effort overall (vs your counterparts at every hosting provider needing to solve the problem for their networks) and provides the best benefit to the victims. Damian
On May 1, 2013, at 7:42 AM, Thomas St-Pierre wrote:
As for BCP38, I would love to stop the spoofed packets, however with them coming from our upstreams, (Level3, Cogent, Tata, etc) I don't see how we can.
Contact them on a case-by-case basis to report the spoofed traffic used to stimulate the servers into responding, including the layer-4 classification criteria, traffic rates, and timestamps available via flow telemetry. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On 04/30/2013 05:28 PM, Thomas St-Pierre wrote:
The large majority of the servers being used in the attacks are not open resolvers. Just DNS servers that are authoritative for a few domains, and the default config of the dns application does referrals to root for anything else.
It sounds like you're already aware that this is the default behavior for an authoritative-only server, and while the referral to the roots is a largeish response and has been used for amplification attacks, it's also rather difficult to mitigate against. A BIND server can be configured to not do that, but contacting each of your customers about it might not have a good ROI. See https://www.dns-oarc.net/oarc/articles/upward-referrals-considered-harmful for more information. Meanwhile, thank you very much for being proactive in this regard. Would that more SPs were as net.responsible as you. :) Doug
Please look at something like rate limiting. Please look at preventing these spoofed packets from entering your network and report the issue. Please provide advice and insights as well as directing customers to the openresolverproject.org website. We want to close these down, if you need an accurate list of IPs in your ASN, please email me and I can give you very accurate data. Thanks! On Apr 30, 2013, at 7:43 PM, Thomas St-Pierre <tstpierre@iweb.com> wrote:
Hi!
I was wondering if anyone had any experience with dealing with open resolvers as a web hoster? We currently have some 40,000 ip's that respond to DNS in our AS, the majority of which are not "open" but do reply with a referral to the root zones. We've been sending emails to our clients but as the servers are not managed by us, there's not much we can do at that level.
Recently we've seen a large increase in the number and volume of DNS amplification DDOS's that are being reflected off of our AS. Just today we've seen at least 6 different attacks with between 4 and 10gbps leaving our AS each time. It's not really causing us issues at the moment because we have the capacity, but I'd hate to be on the receiving side. (and indeed, have been on the receiving side in the past, so I know how much it can suck)
Has anyone ever tried mitigating/rate-limiting/etc these attacks in the network before? (vs at the server/application level)
We have an Arbor peakflow device, but it's not really geared for this scenario I find. It will detect the outgoing attack via the flows, but all we can really do is null-route the victims ip in our AS. Ideally we would need a way to rate-limit DNS packets based on source ip. Maybe a linux box that handles dropping packets from the same source-ip over 1000/sec with some policy-based routing sending the DNS traffic to it? Does such a box exist already?
If anyone has any ideas or suggestions, then by all means! There must be a better way to do this, and I'd really like to avoid re-inventing the wheel if it's been invented already. :)
Thanks! Thomas
On Tue, Apr 30, 2013 at 8:35 PM, Jared Mauch <jared@puck.nether.net> wrote:
Please provide advice and insights as well as directing customers to the openresolverproject.org website. We want to close these down, if you need an accurate list of IPs in your ASN, please email me and I can give you very accurate data.
I think that a public list of open-resolvers is probably overdue, and the only way to get them fixed. It is trivial to scan the entire IPv4 address space for DNS servers that do no throttling even without the resources of a malicious botnet. Smurf was only "fixed" because, as there were fewer networks not running `no ip directed-broadcast,` the remaining amplification sources were flooded with huge amounts of malicious traffic. The public list of smurf amplifiers turned out to be the only way to really deal with it. I predict the same will be true with DNS. -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator / Innovative Network Concepts
On May 1, 2013, at 5:42 PM, Jeff Wheeler wrote:
The public list of smurf amplifiers turned out to be the only way to really deal with it.
It certainly helped; but the real solution was to get Cisco, et. al. to disable directed broadcasts by default. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Well, I was going more for a public list of ISP that refuse to BCP38 their networks. But that's just me =D On point: (If your corporation is massive enough) Basically: . Mirror DST Port 53; . Write some software to stats who's spamming the same DST IP with the same query; . Dynamic ACL them; then . Give a talk to your customers =D ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 05/01/13 06:42, Jeff Wheeler wrote:
Please provide advice and insights as well as directing customers to the openresolverproject.org website. We want to close these down, if you need an accurate list of IPs in your ASN, please email me and I can give you very accurate data. I think that a public list of open-resolvers is probably overdue, and
On Tue, Apr 30, 2013 at 8:35 PM, Jared Mauch <jared@puck.nether.net> wrote: the only way to get them fixed.
It is trivial to scan the entire IPv4 address space for DNS servers that do no throttling even without the resources of a malicious botnet.
Smurf was only "fixed" because, as there were fewer networks not running `no ip directed-broadcast,` the remaining amplification sources were flooded with huge amounts of malicious traffic. The public list of smurf amplifiers turned out to be the only way to really deal with it. I predict the same will be true with DNS.
participants (7)
-
Alain Hebert -
Damian Menscher -
Dobbins, Roland -
Doug Barton -
Jared Mauch -
Jeff Wheeler -
Thomas St-Pierre