All, Open resolvers pose a security threat. I wanted to let everyone know about a search tool that can help you find the ones within your organization. Treat it like a big "BETA" stamp is across it, but please try it out and see if you can close down any hosts within your network. This threat is larger than the SMURF amplification attacks in the past and can result in some quite large attacks. I've seen this spilling out into other mailing lists (e.g.: juniper-nap and others). Please send feedback about links that should be included or documentation and spelling errors to me. openresolverproject.org Some basic stats: 27 million resolvers existed as of this dataset collection only 2.1 million of them were "closed". We have a lot to do to close the hosts, please do what you can to help. Thanks, - Jared
On Mon, 25 Mar 2013, Jared Mauch wrote:
We have a lot to do to close the hosts, please do what you can to help.
I would like to be able to request an IP list of open resolvers in my ASN, perhaps sent to the contact details in RIPE whois database to make sure I'm not falsely representing that ASN. There doesn't seem to be a way to get this done using that webpage? -- Mikael Abrahamsson email: swmike@swm.pp.se
On 25/03/2013 14:33, Mikael Abrahamsson wrote:
I would like to be able to request an IP list of open resolvers in my ASN, perhaps sent to the contact details in RIPE whois database to make sure I'm not falsely representing that ASN.
Why would that matter? This is publicly available information. A per-asn query would be very useful. Nick
On Mon, 25 Mar 2013 15:38:01 -0000, Nick Hilliard said:
On 25/03/2013 14:33, Mikael Abrahamsson wrote:
I would like to be able to request an IP list of open resolvers in my ASN, perhaps sent to the contact details in RIPE whois database to make sure I'm not falsely representing that ASN.
Why would that matter? This is publicly available information.
Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly publicly available info).
On Mar 25, 2013, at 11:44 AM, Valdis.Kletnieks@vt.edu wrote:
On Mon, 25 Mar 2013 15:38:01 -0000, Nick Hilliard said:
On 25/03/2013 14:33, Mikael Abrahamsson wrote:
I would like to be able to request an IP list of open resolvers in my ASN, perhaps sent to the contact details in RIPE whois database to make sure I'm not falsely representing that ASN.
Why would that matter? This is publicly available information.
Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly publicly available info).
Scoping your responses based on query-source should work just fine in this case. There's documentation on how to do that online here: http://www.zytrax.com/books/dns/ch9/close.html I highly recommend doing this with your name server. If you have examples of how to do this you want to share and have me post, as I mentioned, please send me your edits and additions. I want to make this valuable. - Jared
On Mon, Mar 25, 2013 at 11:44 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Mon, 25 Mar 2013 15:38:01 -0000, Nick Hilliard said:
On 25/03/2013 14:33, Mikael Abrahamsson wrote:
I would like to be able to request an IP list of open resolvers in my ASN, perhaps sent to the contact details in RIPE whois database to make sure I'm not falsely representing that ASN.
Why would that matter? This is publicly available information.
Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly publicly available info).
'virginia tech dns configuration' into the webcrawler and: https://computing.vt.edu/content/dns-addresses also: ; <<>> DiG 9.7.0-P1 <<>> @198.82.247.34 www.google.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29982 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 4, ADDITIONAL: 4 ;; QUESTION SECTION: ;www.google.com. IN A ;; ANSWER SECTION: www.google.com. 31 IN A 74.125.228.51 ...snip... -chris
On Mon, 25 Mar 2013 23:19:31 -0400, Christopher Morrow said:
Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly publicly available info).
'virginia tech dns configuration' into the webcrawler and: https://computing.vt.edu/content/dns-addresses
Just proving my point - you didn't find the webpage that also lists their IPv6 addresses. :) Now explain how you find a recursive nameserver that isn't listed in an NS entry and *hasn't* been publicized someplace that Google can find it.
also ; <<>> DiG 9.7.0-P1 <<>> @198.82.247.34 www.google.com
That might, must *might* mind you, be somewhat tangentially related to why I asked Jared what the BCP is for dealing with mobile devices with hardcoded DNS lists. :) (Otherwise read as "we'll be glad to fix it if somebody has a brilliant idea on how to do so without generating more calls to the help desk than the near-zero rate we currently get about DNS amplification issues"....)
On 26/03/2013 07:51, Valdis.Kletnieks@vt.edu wrote:
Now explain how you find a recursive nameserver that isn't listed in an NS entry and *hasn't* been publicized someplace that Google can find it.
Um, you run one of e.g.: http://nmap.org/nsedoc/scripts/dns-recursion.html http://monkey.org/~provos/dnsscan/ Then wait for a while while it churns through the ~224*2^24 packets it needs to scan the entire ipv4 internet. Of course, you could write your own code, but that would take at least 1/2 an hour. Then you have every open resolver on the internet. Now, can you tell me how this is beyond the computing skill of someone who controls a bigass botnet?
(Otherwise read as "we'll be glad to fix it if somebody has a brilliant idea on how to do so without generating more calls to the help desk than the near-zero rate we currently get about DNS amplification issues"....)
The whole point of this thread is that dns amplification hurts other people, not the resolver which is being abused. Just like in the old days, abusing open mail relays hurt other people more than the relay operator. Nick
On Mar 26, 2013, at 3:13 PM, Nick Hilliard wrote:
The whole point of this thread is that dns amplification hurts other people, not the resolver which is being abused.
Actually, it often hurts the resolver(s) being abused, as well, leading to availability problems for those who legitimately need the recursive service in question. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Composed on a virtual keyboard, please forgive typos. On Mar 26, 2013, at 18:27, "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On Mar 26, 2013, at 3:13 PM, Nick Hilliard wrote:
The whole point of this thread is that dns amplification hurts other people, not the resolver which is being abused.
Actually, it often hurts the resolver(s) being abused, as well, leading to availability problems for those who legitimately need the recursive service in question.
On more than one occasion, the operator of an open resolver being used to amplify an attack at our network has called / emailed asking us to stop abusing them. It seems the query rate "we" were sending them was crippling their servers. Sometimes they threaten to filter us. How thoughtful of them! Reminds me of: "Yer h4x0ring me on port 80!!1!1!!1" -- TTFN, patrick
On Tue, 26 Mar 2013 08:13:49 -0000, Nick Hilliard said:
Then wait for a while while it churns through the ~224*2^24 packets it needs to scan the entire ipv4 internet. Of course, you could write your own code, but that would take at least 1/2 an hour.
Then you have every open resolver on the internet.
No you don't. You have every open resolver on a very small legacy portion of the Internet address space that's likely to solve itself in less time than the open mail relay problem took to solve. I know for a fact there's open resolvers on the IPv6 side of the fence. Good luck scanning for those. :) And yes, the miscreants *will* get a list of the IPv6 ones, because unlike whitehat researchers, they won't have a problem with finding a botnet or two, and asking every bot on the nets "What ASN are you in, and what DNS server address did DHCP hand you?"
(Otherwise read as "we'll be glad to fix it if somebody has a brilliant idea on how to do so without generating more calls to the help desk than the near-zero rate we currently get about DNS amplification issues"....)
The whole point of this thread is that dns amplification hurts other people, not the resolver which is being abused. Just like in the old days, abusing open mail relays hurt other people more than the relay operator.
Yes, and in those days, a lot of sites said "sure, we'll fix it if somebody has a good cheat sheet of how to close our relay *without breaking our own users*". (And yes, I was around in "the old days", and I remember just how hard it was for some sites to fix the problem). The problem is that for the open mail relay problem, you could usually scan your mailserver logs or watch the network traffic for the tell-tale 'MAIL FROM:<fred.q.random@mydomain.com>' and then you'd have a pretty good idea that you needed to get in touch with Fred and have him fix his machine to whatever new regime you decided to use to close your open relay, whether it was SMTP AUTH or 'mail after POP3 AUTH' or whatever. Figuring out which user sent a DNS packet from off-campus is a bit more difficult, as the DNS transaction usually doesn't contain any userid info And if you get a recursive lookup for www.ebay.com from a hotel network, you don't have a lot of other traffic you can try to correlate. Up until a few months ago, we *could* have cross-correlated the DNS traffic to POP3 checks from the same IP address - but that doesn't work as well since we outsourced almost all our mail to Google. ;)
On 26/03/2013 14:21, Valdis.Kletnieks@vt.edu wrote:
And if you get a recursive lookup for www.ebay.com from a hotel network,
I'm struggling to understand why it's necessary to hard-code dns servers into the ip networking configuration of a portable device. By definition, these devices will already have dhcp enabled. Nick
And if you get a recursive lookup for www.ebay.com from a hotel network, I'm struggling to understand why it's necessary to hard-code dns servers into the ip networking configuration of a portable device. By definition,
On 26/03/2013 14:21, Valdis.Kletnieks@vt.edu wrote: these devices will already have dhcp enabled.
Nick I can only talk as a very small outfit, too often our roaming/office customers have issues with their local DNS server (in embedded devices
Well, On 03/27/13 07:20, Nick Hilliard wrote: like their $60 linksys) or the local telco (ACME Cable Company)... Having them using our DNS servers save a lot of time and a massive amount of money and headache... same idea as offering them our outgoing mail server (with authentication of course) instead of letting them deal with their Telco's. Mind you none of my 2 servers are a problem because they where always rate-limited... But a few of my clients where for an amount around ~80Mbps of amps. And got fixed within the hour. Now about the struggling about BCP38... ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
On Wed, Mar 27, 2013 at 11:20:54AM +0000, Nick Hilliard wrote:
I'm struggling to understand why it's necessary to hard-code dns servers into the ip networking configuration of a portable device. By definition, these devices will already have dhcp enabled.
It's necessary because many operations are screwing with DNS results in order to advance/suppress political agendas, impose their moral code via censorship, profit via redirection to search portals, etc. If we could actually trust that J. Random Hotel would not do so, then yes, whatever DNS servers are assigned via DHCP would suffice. (Let me caveat this by saying that I don't have a problem with screwing with DNS results for operational reasons, e.g., I think refusing to send DNS queries into DROP-listed space is a good security practice.) ---rsk
On 27/03/2013 12:40, Rich Kulawiec wrote:
It's necessary because many operations are screwing with DNS results in order to advance/suppress political agendas, impose their moral code via censorship, profit via redirection to search portals, etc. If we could actually trust that J. Random Hotel would not do so, then yes, whatever DNS servers are assigned via DHCP would suffice.
then use a vpn and/or provide that service to your users. Sure, hotels and public access wifi does all sorts of stupid and obnoxious stuff, but the way to work around this is not by hardwiring your dns to some open resolver. Nick
Little bit of fun with http://bindguard.activezone.de/ This little example with an open resolver with only 200 queries a minute... The following list show the # of queries made followed by the query in question. False positive: 69.x.x.x 2 a1.mzstatic.com IN A + 2 a1001.phobos.apple.com IN A + 1153 a.root-servers.net IN A + ^- 1153 root queries under 10s... from a small office... Old uncontrolled botnet: 5.x.x.141 1020 isc.org IN ANY +ED 64.x.x.22 1440 isc.org IN ANY +ED 64.x.x.82 1075 isc.org IN ANY +ED 64.x.x.50 1011 isc.org IN ANY +ED 64.x.x.242 1103 isc.org IN ANY +ED This result come from my cheap scripts(tm) and bindguard. If anyone wish to try it I can provide you with some support files and examples. Just contact me offlist. PS: It will be later today... Enjoy today's drama. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
On Mar 27, 2013, at 8:47 AM, Nick Hilliard <nick@foobar.org> wrote:
then use a vpn and/or provide that service to your users. Sure, hotels and public access wifi does all sorts of stupid and obnoxious stuff, but the way to work around this is not by hardwiring your dns to some open resolver.
I've been in many a hotel where 4.2.2.1 is reachable with ttl=1 You must use a VPN or something else to get around places like that. The hotel I'm typing from right now is even more broken.. Jareds-MacBook-Air:~ jared$ ping 4.2.2.1 PING 4.2.2.1 (4.2.2.1): 56 data bytes 64 bytes from 4.2.2.1: icmp_seq=0 ttl=53 time=17.159 ms 64 bytes from 4.2.2.1: icmp_seq=0 ttl=53 time=17.181 ms (DUP!) 64 bytes from 4.2.2.1: icmp_seq=1 ttl=53 time=16.787 ms 64 bytes from 4.2.2.1: icmp_seq=1 ttl=53 time=17.156 ms (DUP!) 64 bytes from 4.2.2.1: icmp_seq=2 ttl=53 time=22.056 ms 64 bytes from 4.2.2.1: icmp_seq=2 ttl=53 time=22.081 ms (DUP!) ^C - Jared
Once upon a time, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> said:
Now explain how you find a recursive nameserver that isn't listed in an NS entry and *hasn't* been publicized someplace that Google can find it.
The same way you find open mail relays, SSH hosts with weak user/password combos, bad WordPress installs, etc. - scan for them. If it is open to the Internet, it will be found (or probably already has been). -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On Mar 26, 2013, at 5:59 AM, Chris Adams <cmadams@hiwaay.net> wrote:
Once upon a time, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> said:
Now explain how you find a recursive nameserver that isn't listed in an NS entry and *hasn't* been publicized someplace that Google can find it.
The same way you find open mail relays, SSH hosts with weak user/password combos, bad WordPress installs, etc. - scan for them. If it is open to the Internet, it will be found (or probably already has been).
Let me rephrase the question… How do you find an open IPv6 recursive name server that isn't listed in an NS entry and hasn't been publicized someplace that Google can find it? Owen
On 03/26/2013 09:28 AM, Owen DeLong wrote:
On Mar 26, 2013, at 5:59 AM, Chris Adams <cmadams@hiwaay.net> wrote:
Once upon a time, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> said:
Now explain how you find a recursive nameserver that isn't listed in an NS entry and *hasn't* been publicized someplace that Google can find it.
The same way you find open mail relays, SSH hosts with weak user/password combos, bad WordPress installs, etc. - scan for them. If it is open to the Internet, it will be found (or probably already has been).
Let me rephrase the question… How do you find an open IPv6 recursive name server that isn't listed in an NS entry and hasn't been publicized someplace that Google can find it?
That question was already answered ... ask the bots what their resolving name servers are, then check to see if they are open. As IPv6 deployment increases, the answers will increasingly include IPv6 open resolvers. Doug
On Mar 26, 2013, at 9:39 AM, Doug Barton <dougb@dougbarton.us> wrote:
On 03/26/2013 09:28 AM, Owen DeLong wrote:
On Mar 26, 2013, at 5:59 AM, Chris Adams <cmadams@hiwaay.net> wrote:
Once upon a time, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> said:
Now explain how you find a recursive nameserver that isn't listed in an NS entry and *hasn't* been publicized someplace that Google can find it.
The same way you find open mail relays, SSH hosts with weak user/password combos, bad WordPress installs, etc. - scan for them. If it is open to the Internet, it will be found (or probably already has been).
Let me rephrase the question… How do you find an open IPv6 recursive name server that isn't listed in an NS entry and hasn't been publicized someplace that Google can find it?
That question was already answered ... ask the bots what their resolving name servers are, then check to see if they are open. As IPv6 deployment increases, the answers will increasingly include IPv6 open resolvers.
Doug
Let me again rephrase… As a white-hat attempting to find problems to address through legitimate means, how do you … Owen
On Mar 26, 2013, at 9:39 AM, Doug Barton <dougb@dougbarton.us> wrote:
On 03/26/2013 09:28 AM, Owen DeLong wrote:
On Mar 26, 2013, at 5:59 AM, Chris Adams <cmadams@hiwaay.net> wrote:
Once upon a time, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> said:
Now explain how you find a recursive nameserver that isn't listed in an NS entry and *hasn't* been publicized someplace that Google can find it. The same way you find open mail relays, SSH hosts with weak user/password combos, bad WordPress installs, etc. - scan for them. If it is open to the Internet, it will be found (or probably already has been).
Let me rephrase the question… How do you find an open IPv6 recursive name server that isn't listed in an NS entry and hasn't been publicized someplace that Google can find it? That question was already answered ... ask the bots what their resolving name servers are, then check to see if they are open. As IPv6 deployment increases, the answers will increasingly include IPv6 open resolvers.
Doug
Let me again rephrase…
As a white-hat attempting to find problems to address through legitimate means, how do you …
On 3/26/13 10:10 AM, Owen DeLong wrote: passive DNS collection , e.g. many people lave large lists of resolvers that have connected to their authoritative nameservers.
Owen
On Tue, Mar 26, 2013 at 6:06 PM, John Levine <johnl@iecc.com> wrote:
As a white-hat attempting to find problems to address through legitimate means, how do you …
You make friends with people with busy authoritative servers and see who's querying them.
I'm confused. Don't most authoritative servers have to answer to just about anyone in order to be useful? Matt
On Tue, Mar 26, 2013 at 7:04 PM, Matthew Petach <mpetach@netflight.com>wrote:
On Tue, Mar 26, 2013 at 6:06 PM, John Levine <johnl@iecc.com> wrote:
As a white-hat attempting to find problems to address through legitimate means, how do you …
You make friends with people with busy authoritative servers and see who's querying them.
I'm confused. Don't most authoritative servers have to answer to just about anyone in order to be useful?
Matt
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
On Tue, Mar 26, 2013 at 7:07 PM, Tom Paseka <tom@cloudflare.com> wrote:
On Tue, Mar 26, 2013 at 7:04 PM, Matthew Petach <mpetach@netflight.com> wrote:
On Tue, Mar 26, 2013 at 6:06 PM, John Levine <johnl@iecc.com> wrote:
As a white-hat attempting to find problems to address through legitimate means, how do you …
You make friends with people with busy authoritative servers and see who's querying them.
I'm confused. Don't most authoritative servers have to answer to just about anyone in order to be useful?
Matt
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
OK, but we started this discussion about open recursive resolvers, right? Securing your recursive resolvers is a very different problem space from trying to come up with rate limits for your authoritative nameservers. In terms of impacts people are feeling today, is most of the pain coming from open recursive servers being abused by miscreants, or from miscreants doing spoofed queries against authoritative nameservers? The concern Valdis raised about securing recursives while still being able to issue static nameserver IPs to mobile devices is an orthogonal problem to Owen putting rate limiters on the authoritative servers for he.net. If we're all lighting up pitchforks and raising torches, I'd kinda like to know at which castle we're going to go throw pitchforks. Matt
On Tue, Mar 26, 2013 at 7:14 PM, Matthew Petach <mpetach@netflight.com> wrote:
The concern Valdis raised about securing recursives while still being able to issue static nameserver IPs to mobile devices is an orthogonal problem to Owen putting rate limiters on the authoritative servers for he.net. If we're all lighting up pitchforks and raising torches, I'd kinda like to know at which castle we're going to go throw pitchforks.
Open Recursive DNS Resolvers. The need to start seriously addressing this issue is kind of dire. The problem with DNS Amplification attacks is getting worse, not better. Oh yeah... and BCP38, too. :-) They both kind of go hand-in-hand. - ferg -- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
On Tue, 26 Mar 2013, Matthew Petach wrote:
The concern Valdis raised about securing recursives while still being able to issue static nameserver IPs to mobile devices is an orthogonal problem to Owen putting rate limiters on the authoritative servers for he.net. If we're all lighting up pitchforks and raising torches, I'd kinda like to know at which castle we're going to go throw pitchforks.
BCP38. As you can see from the wandering conversation, there are many attack vectors that hinge on the ability to spoof the source address, and thereby misdirect responses to your DDoS target. BCP38 filtering stops them all. Or, we can ignore BCP38 for several more years, go on a couple years crusade against open recursive resolvers, then against non-rate-limited authoratative servers, default public RO SNMP communities, etc. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Tue, Mar 26, 2013 at 7:25 PM, Jon Lewis <jlewis@lewis.org> wrote:
On Tue, 26 Mar 2013, Matthew Petach wrote:
The concern Valdis raised about securing recursives while still being able to issue static nameserver IPs to mobile devices is an orthogonal problem to Owen putting rate limiters on the authoritative servers for he.net. If we're all lighting up pitchforks and raising torches, I'd kinda like to know at which castle we're going to go throw pitchforks.
BCP38. As you can see from the wandering conversation, there are many attack vectors that hinge on the ability to spoof the source address, and thereby misdirect responses to your DDoS target. BCP38 filtering stops them all. Or, we can ignore BCP38 for several more years, go on a couple years crusade against open recursive resolvers, then against non-rate-limited authoratative servers, default public RO SNMP communities, etc.
And I don't plan on being around doing this sort of work in another 10+ years, so let's stop farting around. :-p - ferg -- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
Same ol' same ol' (at least since I started this around '93 =D) On 03/26/13 22:25, Jon Lewis wrote:
On Tue, 26 Mar 2013, Matthew Petach wrote:
The concern Valdis raised about securing recursives while still being able to issue static nameserver IPs to mobile devices is an orthogonal problem to Owen putting rate limiters on the authoritative servers for he.net. If we're all lighting up pitchforks and raising torches, I'd kinda like to know at which castle we're going to go throw pitchforks.
BCP38. As you can see from the wandering conversation, there are many attack vectors that hinge on the ability to spoof the source address, and thereby misdirect responses to your DDoS target. BCP38 filtering stops them all. Or, we can ignore BCP38 for several more years, go on a couple years crusade against open recursive resolvers, then against non-rate-limited authoratative servers, default public RO SNMP communities, etc.
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
IP Spoofing still exists because of lazy Peers... Same as the ability to hijack a subnet with BGP... ( *wave* DoD from 2-3 weeks ago ) But, as always, its our responsibility to kill our infrastructure, was IRC Servers in the past, now DNS Servers... Just for those lazy Peers to not HAVE to fix their broken setup. Same ol', same ol'. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
Both. And BCP-38 as well. Lots of effort is needed in this space. Jared Mauch On Mar 26, 2013, at 7:14 PM, Matthew Petach <mpetach@netflight.com> wrote:
In terms of impacts people are feeling today, is most of the pain coming from open recursive servers being abused by miscreants, or from miscreants doing spoofed queries against authoritative nameservers?
In message <CAEmG1=qgJmvCXg9qvk8RVtURyAWmuQLz7yWraQ4TPUkccPxoLw@mail.gmail.com> , Matthew Petach writes:
In terms of impacts people are feeling today, is most of the pain coming from open recursive servers being abused by miscreants, or from miscreants doing spoofed queries against authoritative nameservers?
Both are currently being abused. Rate limiting itself causes operational problems for legitimate users of authoritative servers and will only have a limited effect for a limited time. It is a stop gap measure. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
In message <CAL89Sg+XDKc=_6UWosAZ=wyPJb9tm2GaN0-vDk8Kyiji+vEUUQ@mail.gmail.com> , Tom Paseka writes:
On Tue, Mar 26, 2013 at 7:04 PM, Matthew Petach <mpetach@netflight.com>wrot= e:
On Tue, Mar 26, 2013 at 6:06 PM, John Levine <johnl@iecc.com> wrote:
As a white-hat attempting to find problems to address through legitimat= e means, how do you =85
You make friends with people with busy authoritative servers and see who's querying them.
I'm confused. Don't most authoritative servers have to answer to just about anyone in order to be useful?
Matt
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
You are assuming that there is a recursive server making the queries and that there are not multiple recursive server behind a NAT. Neither of these assumptions in true in practice and with the deployment of CGNs these will become less true. I have two recursive server at home behind a NAT today. Both do DNSSEC. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka <tom@cloudflare.com> wrote:
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
Right now that's a complaint for the mainstream software authors, not for the system operators. When the version of Bind in Debian Stable implements this feature, I'll surely turn it on. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On 2013-03-27, at 09:47, William Herrin <bill@herrin.us> wrote:
On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka <tom@cloudflare.com> wrote:
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
Right now that's a complaint for the mainstream software authors, not for the system operators. When the version of Bind in Debian Stable implements this feature, I'll surely turn it on.
RRL is a moving target, although a promising one. There are currently three implementations of RRL which all behave slightly differently. There is active discussion between the vendors who have implemented RRL, and between early adopters and the vendors. The specification is not yet stable, and changes in the functionality and the rate-limiting behaviour continue to be made. My assessment is that the implementations I have seen are ready for production use, but I think it's understandable given the moving goalpoasts that some vendors have not yet promoted the code to be included in stable releases. As an operator, I understand the benefits of using packaged, stable releases of code. However, we also have a responsibility to deal with operational problems in a timely way. I think it's worth considering that it may well be worth deviating from internal policies about code deployment in this instance; the benefits of doing so can be substantial, and the costs of doing so (especially if we expect them to be time-limited) are not that high. Joe
Joe Abley <jabley@hopcount.ca> wrote:
My assessment is that the implementations I have seen are ready for production use, but I think it's understandable given the moving goalpoasts that some vendors have not yet promoted the code to be included in stable releases.
It is in the current stable release of NSD 3.2.15 though it is a build-time option. It is in the current release candidate of knot DNS 1.2.0-rc4. It will be in BIND-9.10 which has not yet reached public beta. Our servers have been abused as reflectors, and we're using the BIND RRL patch with versions 9.8 and 9.9 to stop the attack traffic. There are other interim options such as using firewall rate limiting which is worse than RRL because it is much more likely to hurt legitimate queries. For example, http://www.bortzmeyer.org/rate-limiting-dns-open-resolver.html Or you can use a configuration add-on such as bindguard. http://bindguard.activezone.de Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first. Rough, becoming slight or moderate. Showers, rain at first. Moderate or good, occasionally poor at first.
On 3/27/2013 8:47 AM, William Herrin wrote:
On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka <tom@cloudflare.com> wrote:
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL). Right now that's a complaint for the mainstream software authors, not for the system operators. When the version of Bind in Debian Stable implements this feature, I'll surely turn it on.
Tracking the clients would be a huge dataset and be especially complicated in clusters. They'd be better off at detecting actual attack vectors rather than rate limiting. However, there are enough nodes out there to easily spread a trickle to avoid individual detections. You don't want to DOS your amplifier, after all. It also wouldn't be hard to rotate through different requests to defeat the "rate limits". Jack
On Wed, Mar 27, 2013 at 10:00 AM, Jack Bates <jbates@brightok.net> wrote:
On 3/27/2013 8:47 AM, William Herrin wrote:
Right now that's a complaint for the mainstream software authors, not for the system operators. When the version of Bind in Debian Stable implements this feature, I'll surely turn it on.
Tracking the clients would be a huge dataset and be especially complicated in clusters. They'd be better off at detecting actual attack vectors rather than rate limiting.
I count this among the several reasons I intend to wait until a solution has been accepted into the bind mainline. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On 3/27/2013 9:34 AM, William Herrin wrote:
On Wed, Mar 27, 2013 at 10:00 AM, Jack Bates <jbates@brightok.net> wrote:
Tracking the clients would be a huge dataset and be especially complicated in clusters. They'd be better off at detecting actual attack vectors rather than rate limiting.
I count this among the several reasons I intend to wait until a solution has been accepted into the bind mainline.
You'll also find that it serves little purpose. The only 2 methods for stopping DNS amplification to my knowledge are: 1) tcp 2) require all requests to pad out to maximum response 3) BCP38 (in spirit) The first has latency, load, and connection limitations. It is just too expensive. The second would stop amplification, however, it will not stop botnets using them in attempts to hide the bot nodes in a very effective manner. It's also unlikely that we'd ever see it implemented. The only effective fix is still BCP38 (in spirit). Jack
In message <51530632.3020402@brightok.net>, Jack Bates writes:
On 3/27/2013 9:34 AM, William Herrin wrote:
On Wed, Mar 27, 2013 at 10:00 AM, Jack Bates <jbates@brightok.net> wrote:
Tracking the clients would be a huge dataset and be especially complicated in clusters. They'd be better off at detecting actual attack vectors rather than rate limiting.
I count this among the several reasons I intend to wait until a solution has been accepted into the bind mainline.
You'll also find that it serves little purpose. The only 2 methods for stopping DNS amplification to my knowledge are:
1) tcp
2) require all requests to pad out to maximum response
3) BCP38 (in spirit)
4) a variant of dns cookies see the draft by Donald Eastlake III if you can accept a couple of extra bytes in the reply to a non cookie query * minimal amplification to spoofed queries. * remove the need to randomise source ports. * state is stored in the stubs / recursives serves about the upstream. * works with recursive servers and authoritative servers hash (server secret + client differentiator + time) -> crypto hash query [code = 0 (8 bits), extended id (64 bits), client differentiator (64 bits), server time (32 bits), crypto hash ] response [code (8 bit), extended id (64 bits), client differentiator' (64 bits), server time' (32 bits), crypto hash' ] A query is only accepted if the presented client differentiator and server time along with the server secret give the presented hash and not too much time has elasped otherwise code is set to 1 and the completed option is returned. clients record everything after the extended id to send with future queries. client differentiator, server time and crypto hash may be missing on the initial query.
The first has latency, load, and connection limitations. It is just too expensive.
The second would stop amplification, however, it will not stop botnets using them in attempts to hide the bot nodes in a very effective manner. It's also unlikely that we'd ever see it implemented.
The only effective fix is still BCP38 (in spirit).
Jack
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Jack Bates <jbates@brightok.net> wrote:
You'll also find that [DNS RRL] serves little purpose.
In my experience it works extremely well. Yes it is possible to work around it, but you still need to stop the attacks that are happening now. It is good to make the attacker's job harder.
1) tcp
RRL pushes legitimate clients to TCP if they get muddled up with attack traffic.
2) require all requests to pad out to maximum response
I expect that is as easy to deploy as BCP38, IPv6, and DNSSEC.
3) BCP38 (in spirit)
That should be deployed as well as RRL. Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first. Rough, becoming slight or moderate. Showers, rain at first. Moderate or good, occasionally poor at first.
On 3/27/2013 4:49 PM, Tony Finch wrote:
Jack Bates <jbates@brightok.net> wrote:
3) BCP38 (in spirit) That should be deployed as well as RRL.
Tony.
If BCP38 was properly deployed, what would be the purpose of RRL outside of misbehaving clients or direct attacks against that one server? We already know the fix for spoofing. Trying to tweak every service that spoofing effectively takes advantage of will not be a winning game. Sending legitimate clients to TCP is also a losing game. DNS is UDP for a reason. The infrastructure to switch it to TCP is prohibitive and completely destroys the anycast mechanisms. Jack
Jack Bates <jbates@brightok.net> wrote:
If BCP38 was properly deployed, what would be the purpose of RRL outside of misbehaving clients or direct attacks against that one server?
If fictional scenario, irrelevant answer. Given the current situation, efforts to deploy both RRL and BCP38 in parallel will reduce the mess we are in. Let's race to see who gets to full deployment first.
The infrastructure to switch it to TCP is prohibitive and completely destroys the anycast mechanisms.
Yeah, anycast for HTTP doesn't work at all. Just ask CloudFlare. Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first. Rough, becoming slight or moderate. Showers, rain at first. Moderate or good, occasionally poor at first.
On Wed, 27 Mar 2013 16:59:16 -0500, Jack Bates said:
On 3/27/2013 4:49 PM, Tony Finch wrote:
Jack Bates <jbates@brightok.net> wrote:
3) BCP38 (in spirit) That should be deployed as well as RRL.
Tony.
If BCP38 was properly deployed, what would be the purpose of RRL outside of misbehaving clients or direct attacks against that one server?
It would be quite sufficient just for the misbehaving clients aspect.
Jack Bates <jbates@brightok.net> wrote:
Tracking the clients would be a huge dataset and be especially complicated in clusters.
The memory usage is guite manageable: for the BIND patch it is at most 40-80 bytes (for 32 or 64 bit machines) per request per second. You're doing well if you need a megabyte. There's no need to get complicated with clusters: it's enough of an improvement just to track rates per server. Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first. Rough, becoming slight or moderate. Showers, rain at first. Moderate or good, occasionally poor at first.
It's been available in linux for a long time, just not in BIND… Here is a working ip6tales example: -A RH-Firewall-1-INPUT -s 2620:0:930::/48 -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -s 2620:0:930::/48 -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 53 -m limit --limit 30/minute --limit-burst 90 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 53 -m limit --limit 30/minute --limit-burst 90 -j ACCEPT YMMV and you may wish to provide tighter limits (less than 30 QPM or a burst of <90). Owen On Mar 27, 2013, at 6:47 AM, William Herrin <bill@herrin.us> wrote:
On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka <tom@cloudflare.com> wrote:
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
Right now that's a complaint for the mainstream software authors, not for the system operators. When the version of Bind in Debian Stable implements this feature, I'll surely turn it on.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Op 27-03-13 16:54, Owen DeLong schreef:
It's been available in linux for a long time, just not in BIND…
Not entirely true: http://www.redbarn.org/dns/ratelimits
Here is a working ip6tales example:
Tricky... There is also the 'hashlimit' module (at least for v4, not sure about v6), that may be a better approach, because it works on a 'per ip address'-basis. See https://lists.isc.org/pipermail/bind-users/2012-July/088223.html for some inspiration of how it may be of value. -- Marco On Mar 27, 2013, at 6:47 AM, William Herrin <bill@herrin.us> wrote:
On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka <tom@cloudflare.com> wrote:
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL). Right now that's a complaint for the mainstream software authors, not for the system operators. When the version of Bind in Debian Stable implements this feature, I'll surely turn it on.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
-- Marco Davids
On Mar 27, 2013, at 11:54 AM, Owen DeLong <owen@delong.com> wrote:
It's been available in linux for a long time, just not in BIND…
Here is a working ip6tales example:
-A RH-Firewall-1-INPUT -s 2620:0:930::/48 -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -s 2620:0:930::/48 -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 53 -m limit --limit 30/minute --limit-burst 90 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 53 -m limit --limit 30/minute --limit-burst 90 -j ACCEPT
YMMV and you may wish to provide tighter limits (less than 30 QPM or a burst of <90).
I am very concerned about examples such as this possibly being implemented by a well intentioned sysadmin or neteng type without understanding their query load and patterns. bind with the rrl patch does log when things are happening. While the data is possible to extract from iptables, IMHO it's not quite as easy to audit as a syslog. - Jared
On 2013-03-27, at 14:52, Jared Mauch <jared@puck.nether.net> wrote:
I am very concerned about examples such as this possibly being implemented by a well intentioned sysadmin or neteng type without understanding their query load and patterns. bind with the rrl patch does log when things are happening. While the data is possible to extract from iptables, IMHO it's not quite as easy to audit as a syslog.
For an authoritative-only server, people can expect coarse rate-limits such as those quoted earlier with iptables to give false positives and to reject legitimate queries. RRL is far safer. For a recursive server, I agree you need a much better understanding of your traffic patterns before you try something like the iptables example. Dropping queries from your own clients' stub resolvers has an immediate support cost. You *really* don't want false positives, there. Joe
AsI think as we all know the deficiency is the design of the DNS system overall. No disrespect to anybody, but lots of companies make money off of the design deficiencies and try to position themselves as offering 'value add services' or something similar. Basically they make money because the inherent design of the DNS system is the antithesis of being able to deliver information on a best effort basis. Entire 'value add' economic ecosystems are created with these kinds of things and once it is done, it is extremely difficult to un-do. If the endpoint is not available or is unreliable, this is all well understood and 100% captured in the modern implementations of the Internet with it be OSI or TCP/IP and even with numerous extensions from there. The fundamental cause and source of failure for these kinds of attacks comes from the the way the DNS (and lets not even get into 'valid' SSL certs) is designed. It is fundamentally flawed. I am sure there were plenty of political reasons for it to have ended up this way instead of being done in a more robust fashion? For all the gripes and complaints - all I see is complaints of the symptoms and nobody calling out the original cause of the disease? On Mar 27, 2013, at 6:47 AM, William Herrin <bill@herrin.us> wrote:
On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka <tom@cloudflare.com> wrote:
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
Right now that's a complaint for the mainstream software authors, not for the system operators. When the version of Bind in Debian Stable implements this feature, I'll surely turn it on.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Mar 27, 2013, at 10:11 PM, Michael DeMan <nanog@deman.com> wrote:
AsI think as we all know the deficiency is the design of the DNS system overall.
One of the largest DDoS attacks I've witnessed was SNMP-based, walking entire OID sub-trees (with spoofed source addresses) across thousands of CPEs that defaulted to allowing SNMP queries over the WAN interface. "Oops". Topped out around 70 Gbps if I remember correctly. No DNS involved.
The fundamental cause and source of failure for these kinds of attacks comes from the the way the DNS (and lets not even get into 'valid' SSL certs) is designed.
Not really. You're at least one layer too high. (not even going to question what "'valid' SSL certs" have to do with the DNS)
It is fundamentally flawed. I am sure there were plenty of political reasons for it to have ended up this way instead of being done in a more robust fashion?
I suspect if you look at the number of queries per second the best TCP stacks could handle circa mid-1980s and compare that number to an average UDP stack, you might see an actual reason instead of conspiracy theories.
For all the gripes and complaints - all I see is complaints of the symptoms and nobody calling out the original cause of the disease?
You mean connectionless datagram transmission without validation of packet source? Regards, -drc
On (2013-03-27 22:27 -1000), David Conrad wrote:
One of the largest DDoS attacks I've witnessed was SNMP-based, walking entire OID sub-trees (with spoofed source addresses) across thousands of CPEs that defaulted to allowing SNMP queries over the WAN interface. "Oops". Topped out around 70 Gbps if I remember correctly. No DNS involved.
Wonderful data point. Services are not the problem. Open recursors are not the problem, there are millions of them, and even if we close all of them, attack vector remains almost identically the same, as due to DNSSEC it's easy to find large RR in authorative servers. I think most everyone is missing the key notion that BCP38 does not need to be deployed my millions. Most people are NOT doing ACL filtering towards their transit customers, Tier1<->Tier2 cannot do it (strict IRR is not practical). Tier2<->Tier3 can do it, and should do it. We have about 6000 tier2 networks that we need to fix to make spooffing attack vectors impractical. It's entirely doable if we can agree that ACL towards your transit customer is BCP and start approaching/educating/helping (github scripts to do it automatically for your JunOS, IOS, TimOS, IOS-XR...) these 6000 networks. -- ++ytti
On Tue, Mar 26, 2013 at 07:07:16PM -0700, Tom Paseka wrote:
On Tue, Mar 26, 2013 at 7:04 PM, Matthew Petach <mpetach@netflight.com>wrote:
On Tue, Mar 26, 2013 at 6:06 PM, John Levine <johnl@iecc.com> wrote:
As a white-hat attempting to find problems to address through legitimate means, how do you …
You make friends with people with busy authoritative servers and see who's querying them.
I'm confused. Don't most authoritative servers have to answer to just about anyone in order to be useful?
Matt
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
unbound with it's dns-prefetching queries a dns servers again in I think the last 10% of ttl when returning hit to client to refresh ttl and keep it current. To me this doesn't seem excessive, and will improve performance for regularly accessed sites with short ttls which are quite common now (google, facebook, etc) It'd break if doing that extreme rate limiting. But so would things like rebooting a dns server, I think if rate limiting is done it has to be on the leniant side. Also how do you know that the dns resolver got a successful reply? Just because you've received a packet from a client doesn't mean that you can reach the client. So if there's one way traffic or excessive dual way packet loss the chances of prematurely blocking clients and creating longer outages is too great. That said, a lot of these amplifications attacks use ANY requests, which normal clients don't. And those could be rate limited down without effecting normal traffic I'm sure. Ben.
On 3/28/13, Ben Aitchison <ben@meh.net.nz> wrote:
On Tue, Mar 26, 2013 at 07:07:16PM -0700, Tom Paseka wrote:
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
The RFC doesn't say that is a should; a client MAY only query you once for a record within its TTL; the TTL is the duration after which the entry /must/ be expunged from the cache, it is an allowed maximum, not a minimum lifetime. A client may query plenty of times within its TTL. Sufficiently low rate limits on the authoritative would open the possibility of new kinds of attacks. If the authoritative DNS server decides to limit its rate of response, this might be used to conduct a DoS against the recursive nameserver's ability to lookup queries against the authoritative NS applying the limit. This could be leveraged remotely through a malicious website, remote loading bad image URLs from a significant number of non-existent subdomains, causing the rate limit to be attained. This may also be used to facilitate cache poisoning against legitimate recursors, targeting the domain whose authoritative servers apply a strict limit, by intentionally causing the recursor to make the maximum number of queries allowed, before sending spoofed responses. Especially a client that answers many different queries for a large number of clients and has limited cache sizes may query many times within a TTL. The average record cache lifetime might be 15 to 40 seconds (with as low as 1 second in cases), even if the record TTL is 86400. Or the cache may be manually flushed by the operator, in order to have a local DNS record change take effect more immediately (since most resolvers do not provide an admin command to flush only one zone from their cache). No guarantee is made about the size of the client's cache, number of records, or the client's cache aging policy. The response may be discarded or aged out, well before its TTL has elapsed. There may be other 'more popular' records on the same DNS resolver that are retained in the cache until TTL. Additional queries may be issued as a cache-poisoning avoidance mechanism. The same DNS servers might get queried multiple times successively for different records within the same zone.
unbound with it's dns-prefetching queries a dns servers again in I think the last 10% of ttl when returning hit to client to refresh ttl and keep it current.
-- -JH
In message <20130329034419.GA26823@meh.net.nz>, Ben Aitchison writes:
That said, a lot of these amplifications attacks use ANY requests, which normal clients don't. And those could be rate limited down without effecting normal traffic I'm sure.
Ben.
And you need to learn that normal clients *do* issue type any queries. Blocking any queries would be easy if normal clients didn't issue any queries. You would have need controls added to nameserver to block them if there wern't normal clients issuing any queries. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
In message <20130329034419.GA26823@meh.net.nz>, Ben Aitchison writes:
That said, a lot of these amplifications attacks use ANY requests, which normal clients don't. And those could be rate limited down without effecting normal traffic I'm sure.
And you need to learn that normal clients *do* issue type any queries. Blocking any queries would be easy if normal clients didn't issue any queries. You would have need controls added to nameserver to block them if there wern't normal clients issuing any queries.
So you fsckin' rate limit them to a reasonable level. Really, I've spent a disappointing amount of time listening to the "but but but you can't DOOOOOOOOO that" from the ISC camp over the years, and while I understand Vixie's concerns about breaking things in unexpected ways, the reality of it all is that a DDoS attack is trivially identifiable from other traffic for any number of reasons, such as "like duh we don't usually see a megabit of queries from off site" or "like duh we don't usually see repeated queries for the same question from off site" or "like duh we don't usually see ANY queries from off site". So now go back and read what Ben wrote again, because
And those could be rate limited down without effecting normal traffic I'm sure.
THIS BIT IS THE EFFIN' POINT, WHICH YOU GUYS KEEP EFFIN' IGNORING. Look, this is a bad situation. Many networks don't BCP38. Many networks have unlimited open recursers. Many networks don't monitor for trouble. And then someone finds out how to take advantage. Well, all those things are bad, I'm sure we agree. However, some of us have decades of precedent and lots of deployment that make running an open recurser a necessity. That CAN be done, at least in our case, through some exemptions, and then running everything else through a drinking straw, because we KNOW that normal usage patterns of remote clients are ${x}. Now sadly I can't easily do a better job than just rate limiting inbound and outbound traffic because ISC won't entertain the idea. But what agenda does that bullheadedness serve? If you think you're "saving DNS" by not allowing administrators to twiddle with intelligent response rates, well, many of us will just take a bigger wrench and fix it with the brute force method. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Mar 29, 2013, at 6:58 PM, Joe Greco wrote:
Really, I've spent a disappointing amount of time listening to the "but but but you can't DOOOOOOOOO that"
What they're really worried about is folks arbitrarily deciding to permanently mask out ANY queries altogether as a matter of policy, rather than either rate-limiting them or selectively filtering them during an actual attack, and only within the scope of the servers/records being abused for that particular attack. Many measures which are not only permissible but are often vitally necessary in order to achieve partial service recovery during an attack can cause prohibitive levels of brokenness when implemented as matters of permanently-enforced policy. Given the history of such overt stupidity as blocking TCP/53, disallowing UDP DNS packets larger than 512 bytes, blocking ICMP necessary for PMTU-D, et. al., their concerns are not unreasonable. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Mar 29, 2013, at 6:58 PM, Joe Greco wrote:
Really, I've spent a disappointing amount of time listening to the "but b= ut but you can't DOOOOOOOOO that"=20
What they're really worried about is folks arbitrarily deciding to permanen= tly mask out ANY queries altogether as a matter of policy, rather than eith= er rate-limiting them or selectively filtering them during an actual attack= , and only within the scope of the servers/records being abused for that pa= rticular attack.
Many measures which are not only permissible but are often vitally necessar= y in order to achieve partial service recovery during an attack can cause p= rohibitive levels of brokenness when implemented as matters of permanently-= enforced policy. Given the history of such overt stupidity as blocking TCP= /53, disallowing UDP DNS packets larger than 512 bytes, blocking ICMP neces= sary for PMTU-D, et. al., their concerns are not unreasonable.
There's a difference between "concerns" and bullheadedness. In the meantime, refusing to give admins tools to cope with an attack in a surgical-strike manner is basically just helping the attackers. As an administrator, I can cause brokenness in any number of clever, dumb, or accidental ways. However, it is also up to me to cause the network to work in the manner we need it to, and if I had a better understanding of our traffic than Vixie has, /which I do/, then I am in a better place to make intelligent decisions about what should and shouldn't be allowed and at what rates. In the meantime, all the "but but but THAT'LL BREAK THE INTARWEB" stuff, okay, great, so they don't want to supply tools that might break things. News flash, 300Gbps DNS attack underway. Not like THAT will break anything. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On 03/29/2013 04:58 AM, Joe Greco wrote:
Really, I've spent a disappointing amount of time listening to the "but but but you can't DOOOOOOOOO that" from the ISC camp over the years
Joe, Perhaps you haven't been keeping up with the Response Rate Limiting work? http://www.redbarn.org/dns/ratelimits http://ss.vix.com/~vixie/isc-tn-2012-1.txt
Ben Aitchison wrote:
Authoritative DNS servers need to implement rate limiting. (a client shouldn't query you twice for the same thing within its TTL).
unbound with it's dns-prefetching queries a dns servers again in I think the last 10% of ttl when returning hit to client to refresh ttl and keep it current.
They are the worst things to do against DDOS, as queries must be repeated if query or reply packets are dropped, often because of DDOS. Rate limiting with token bucket of 5 or 7 packet deep could be useful, though it enables 5 or 7 times of amplification.
That said, a lot of these amplifications attacks use ANY requests, which normal clients don't. And those could be rate limited down without effecting normal traffic I'm sure.
We should rather obsolete DNSSEC, which amplifies a lot even though it is not really deployed. Masataka Ohta
On Mar 26, 2013, at 7:04 PM, Matthew Petach <mpetach@netflight.com> wrote:
On Tue, Mar 26, 2013 at 6:06 PM, John Levine <johnl@iecc.com> wrote:
As a white-hat attempting to find problems to address through legitimate means, how do you …
You make friends with people with busy authoritative servers and see who's querying them.
I'm confused. Don't most authoritative servers have to answer to just about anyone in order to be useful?
If you give the same answer 15x to the same person in a few seconds one can possibly infer they aren't a caching resolver or are broken. Either way you can think about ignoring them for a few with dampening or similar.
On Tue, 26 Mar 2013 19:13:43 -0700, Jared Mauch said:
If you give the same answer 15x to the same person in a few seconds one can possibly infer they aren't a caching resolver or are broken. Either way you can think about ignoring them for a few with dampening or similar.
So what you're saying is that a decade after Duane Wessels presented at NANOG saying that 98% of the traffic hitting the root nameservers was total bull, we're not doing much better?
On 3/26/13 7:04 PM, Matthew Petach wrote:
On Tue, Mar 26, 2013 at 6:06 PM, John Levine <johnl@iecc.com> wrote:
As a white-hat attempting to find problems to address through legitimate means, how do you … You make friends with people with busy authoritative servers and see who's querying them. I'm confused. Don't most authoritative servers have to answer to just about anyone in order to be useful? yes, and that's why they're useful for tracking who has a resolver.
Matt
----- Original Message -----
From: "Matthew Petach" <mpetach@netflight.com>
You make friends with people with busy authoritative servers and see who's querying them.
I'm confused. Don't most authoritative servers have to answer to just about anyone in order to be useful?
Yes. And that list will contain lots of recursive servers. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
On (2013-03-26 09:28 -0700), Owen DeLong wrote:
Let me rephrase the question… How do you find an open IPv6 recursive name server that isn't listed in an NS entry and hasn't been publicized someplace that Google can find it?
Pwn authorative server catering moderately popular domain and share query sources as torrent file. -- ++ytti
In a message written on Tue, Mar 26, 2013 at 06:40:38PM +0200, Saku Ytti wrote:
Pwn authorative server catering moderately popular domain and share query sources as torrent file.
Run authortative server hosting command and control for a botnet, log query sources and use for your own purposes. IPv6 Temporary / Privacy addresses and making your DNS server query from them outbound might be a useful trick. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On 03/25/2013 08:44 AM, Valdis.Kletnieks@vt.edu wrote:
On 25/03/2013 14:33, Mikael Abrahamsson wrote:
I would like to be able to request an IP list of open resolvers in my ASN, perhaps sent to the contact details in RIPE whois database to make sure I'm not falsely representing that ASN. Why would that matter? This is publicly available information. Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly
On Mon, 25 Mar 2013 15:38:01 -0000, Nick Hilliard said: publicly available info). Sounds like your making the faulty assumption that an attacker would use normal means to find your servers.
-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments
On 3/29/13, Scott Noel-Hemming <frogstarr78@gmail.com> wrote:
Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly publicly available info). Sounds like your making the faulty assumption that an attacker would use normal means to find your servers.
A distributed scan of the entire IPv4 space for all internet IPs running open DNS servers is fairly doable; actually a long term scan taking 100 to 200 days of continuous DNS scanning is completely trivial. The fact a recursive DNS server exists at a certain IP address can also be exposed to the operators of authoritative (or root) DNS servers, through the queries that the recursive servers make. For example: an internet advertiser can place syndicated ads on certain websites, containing images referring to a server on their domain (that requires resolving their domain), and then mine data from the IP addresses that are contacting their authoritative DNS servers in order to make queries. For some domains, the authoritative DNS servers might even want to ping the recursor, and use the result to decide what set of answers to send for future queries, in order to reply with choices that are anticipated to minimize latency. -- -JH
On Mar 31, 2013, at 5:09 PM, Jimmy Hess <mysidia@gmail.com> wrote:
On 3/29/13, Scott Noel-Hemming <frogstarr78@gmail.com> wrote:
Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly publicly available info). Sounds like your making the faulty assumption that an attacker would use normal means to find your servers.
A distributed scan of the entire IPv4 space for all internet IPs running open DNS servers is fairly doable; actually a long term scan taking 100 to 200 days of continuous DNS scanning is completely trivial.
I updated the openresolverproject.org data in less than 8 hours. The system would scan 1.0.0.0 , 1.0.0.1 … in sequence. Next time it runs, it's going to use a slightly different method which may expose a few more servers. The 2013-Mar-31 data showed: 2,471,484 servers returned refused. (369k change downward) 20,675,738 with correct answer in packet. If I extrapolate 369k/week closing, everything will be closed in about a year. (Compared to 2.1 mil refused the week before; compared to 21.4 Million with correct answer in packet the week before). I know many people are working on their respective hosts and/or network to close things down. Many thanks to everyone that is treating this as a critical issue to close these hosts. - jared
On Mar 31, 2013, at 8:46 PM, Jared Mauch wrote:
Many thanks to everyone that is treating this as a critical issue to close these hosts.
Just back to the office, and started checking my networks. Found one of the resolvers is a Netgear SOHO NAT box. EoL'd, no new firmware available. Anyone have any feeling for what percentage are these types of boxes? --Chris
On Mon, Apr 1, 2013 at 7:35 AM, Chris Boyd <cboyd@gizmopartners.com> wrote:
On Mar 31, 2013, at 8:46 PM, Jared Mauch wrote:
Many thanks to everyone that is treating this as a critical issue to close these hosts.
Just back to the office, and started checking my networks. Found one of the resolvers is a Netgear SOHO NAT box. EoL'd, no new firmware available. Anyone have any feeling for what percentage are these types of boxes?
A lot? :-/ - ferg -- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
On Mon, 1 Apr 2013, Chris Boyd wrote:
Just back to the office, and started checking my networks. Found one of the resolvers is a Netgear SOHO NAT box. EoL'd, no new firmware available. Anyone have any feeling for what percentage are these types of boxes?
If you buy "type of box" mean "small SOHO NAT router which does DNS resolving on the WAN interface" then I'd say "a lot". Someone does a rollout of new software and configuration and happens to mess up the config file (or the vendor just happens to enable global dns resolving in the new software) and this slips through testing, then you're there. I believe this happens all the time. That's why the publication of these lists are important, in a lot of cases there are a lot of people who are simply not aware of these devices doing this, and they need to be poked to notice. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Sun, 31 Mar 2013 16:09:35 -0500, Jimmy Hess said:
On 3/29/13, Scott Noel-Hemming <frogstarr78@gmail.com> wrote:
Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly publicly available info). Sounds like your making the faulty assumption that an attacker would use normal means to find your servers.
A distributed scan of the entire IPv4 <SNIP>
Stop right there. Anybody who is looking at this as an IPv4 issue is woefully misinformed about the nature of the problem.
On Mar 31, 2013, at 11:16 PM, Valdis.Kletnieks@vt.edu wrote:
On Sun, 31 Mar 2013 16:09:35 -0500, Jimmy Hess said:
On 3/29/13, Scott Noel-Hemming <frogstarr78@gmail.com> wrote:
Some of us have both publicly-facing authoritative DNS, and inward facing recursive servers that may be open resolvers but can't be found via NS entries (so the IP addresses of those aren't exactly publicly available info). Sounds like your making the faulty assumption that an attacker would use normal means to find your servers.
A distributed scan of the entire IPv4 <SNIP>
Stop right there.
Anybody who is looking at this as an IPv4 issue is woefully misinformed about the nature of the problem.
:) IPv4 it's easy to collect an inventory (the math works). IPv6, not nearly as easy. - Jared
On 1 Apr 2013, at 14:44, Jared Mauch <jared@puck.nether.net> wrote:
On Mar 31, 2013, at 11:16 PM, Valdis.Kletnieks@vt.edu wrote:
Anybody who is looking at this as an IPv4 issue is woefully misinformed about the nature of the problem.
:)
IPv4 it's easy to collect an inventory (the math works). IPv6, not nearly as easy.
You should be able to get a reasonable sample of IPv6 resolvers from the query logs of a popular authoritative server. Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/
On Mon, 01 Apr 2013 19:40:03 +0100, Tony Finch said:
You should be able to get a reasonable sample of IPv6 resolvers from the query logs of a popular authoritative server.
Hopefully, said logs are not easily accessible to the miscreants. (I still expect the most feasible method for the miscreants is to start a botnet and see what boxes get handed an IPv6 DNS via dhcp6).
On 4/1/13 11:59 AM, Valdis.Kletnieks@vt.edu wrote:
On Mon, 01 Apr 2013 19:40:03 +0100, Tony Finch said:
You should be able to get a reasonable sample of IPv6 resolvers from the query logs of a popular authoritative server. Hopefully, said logs are not easily accessible to the miscreants. Miscreants with popular zones clearly can do that.
Reverse-lookups for spam originating machines might for example be a sufficient source of queries if you control the reverse zone. The DNS makes it's own gravy.
(I still expect the most feasible method for the miscreants is to start a botnet and see what boxes get handed an IPv6 DNS via dhcp6).
On Mon, 1 Apr 2013 19:40:03 +0100 Tony Finch <dot@dotat.at> wrote:
You should be able to get a reasonable sample of IPv6 resolvers from the query logs of a popular authoritative server.
When I tried this in the past for IPv4, I missed the majority of potential open resolvers / open forwarders on the net compared to just searching the entire address space. And I was examining this from the perspective of what a very large TLD was seeing. I think it is likely that there are going to be a significant number of IPv6-based resolvers that are aren't as easily knowable. This of course is potentially good too, since if they are really that hard to find, then it makes them less likely to be as easily abused. So, in addition to BCP 38 (and don't forget to mention BCP 84 in the same breath), RRL for auth servers and hardening/closing resolvers... we should be advocating the migration to DNS over IPv6-only? :-) John
On 2013-03-25 16:38, Nick Hilliard wrote:
Why would that matter? This is publicly available information.
A list of 27 million open resolvers would be a pretty convenient input for miscreants who want to abuse them, I believe? I assume Jared & co doesn't want their collected work to be abused like that. I quickly scripted a RIPE-based ASN lookup tool for myself just so I could look up my own and customers AS-based IP ranges atleast. Thank you so much for making the site available, Jared. Very helpful! -- /ahnberg.
On Mar 25, 2013, at 11:54 AM, Mattias Ahnberg <mattias@ahnberg.pp.se> wrote:
On 2013-03-25 16:38, Nick Hilliard wrote:
Why would that matter? This is publicly available information.
A list of 27 million open resolvers would be a pretty convenient input for miscreants who want to abuse them, I believe? I assume Jared & co doesn't want their collected work to be abused like that.
I quickly scripted a RIPE-based ASN lookup tool for myself just so I could look up my own and customers AS-based IP ranges atleast. Thank you so much for making the site available, Jared. Very helpful!
There are other tools, such as the one linked to at the measurement factory that will email the contacts for the objects (e.g.: RIPE/ARIN otherwise). Here's the direct link for those that are interested. As I mentioned, this is alpha/beta quality code, but i feel very confident with the data. - Jared
On 25/03/2013 15:54, Mattias Ahnberg wrote:
A list of 27 million open resolvers would be a pretty convenient input for miscreants who want to abuse them, I believe? I assume Jared & co doesn't want their collected work to be abused like that.
http://nmap.org/nsedoc/scripts/dns-recursion.html http://monkey.org/~provos/dnsscan/ There are 224*2^24 possible unicast hosts, and a whole pile less which are routed on the DFZ. I don't think that we can pretend that it's going to help if we hide this information under a stone and hope that people who are inclined to launch DNS DDoS attacks are dumb enough not to be able to figure out how to use these tools. Highlighting the situation and getting operators to do something will help fix the problem. Nick
Well, Why would you only go after them? Easier target to mitigate the problem? That might be just me, but I find those peers allowing their customers to spoof source IP addresses more at fault. PS: Some form of adaptive rate limitation works for it btw =D ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 03/25/13 12:14, Nick Hilliard wrote:
On 25/03/2013 15:54, Mattias Ahnberg wrote:
A list of 27 million open resolvers would be a pretty convenient input for miscreants who want to abuse them, I believe? I assume Jared & co doesn't want their collected work to be abused like that. http://nmap.org/nsedoc/scripts/dns-recursion.html http://monkey.org/~provos/dnsscan/
There are 224*2^24 possible unicast hosts, and a whole pile less which are routed on the DFZ.
I don't think that we can pretend that it's going to help if we hide this information under a stone and hope that people who are inclined to launch DNS DDoS attacks are dumb enough not to be able to figure out how to use these tools.
Highlighting the situation and getting operators to do something will help fix the problem.
Nick
On 2013-03-25, at 12:35, Alain Hebert <ahebert@pubnix.net> wrote:
Well,
Why would you only go after them?
Easier target to mitigate the problem?
That might be just me, but I find those peers allowing their customers to spoof source IP addresses more at fault.
PS: Some form of adaptive rate limitation works for it btw =D
DNS servers (recursive and authoritative-only) are the low-hanging fruit du jour. I agree that there are many other effective amplifiers, and that even maximum DNS hygiene will not make the wider problem go away. A quick note on your final comment, though: whilst adaptive response rate limiting (so-called RRL) is fast developing into an effective mitigation for reflection attacks against authority-only servers, there is far less experience with traffic patterns or the effects of rate-limiting (using RRL or anything else) on recursive servers. The best advice for operation of recursive servers remains "restrict access to legitimate clients", not "apply rate-limiting". Joe
Subject: Re: Open Resolver Problems Date: Mon, Mar 25, 2013 at 12:45:40PM -0400 Quoting Joe Abley (jabley@hopcount.ca):
DNS servers (recursive and authoritative-only) are the low-hanging fruit du jour. I agree that there are many other effective amplifiers, and that even maximum DNS hygiene will not make the wider problem go away.
A quick note on your final comment, though: whilst adaptive response rate limiting (so-called RRL) is fast developing into an effective mitigation for reflection attacks against authority-only servers, there is far less experience with traffic patterns or the effects of rate-limiting (using RRL or anything else) on recursive servers.
The best advice for operation of recursive servers remains "restrict access to legitimate clients", not "apply rate-limiting".
Twice agree. I try to have ::1 as resolver on my server machines that are in a position to be used, and only accept queries on ::1. Takes care of access control nicely. For auth servers, those serving DNSSEC records are especially attractive as amplifiers. At the moment, I'd have a hard time defending unrestricted query rates on auth servers if they serve DNSSEC. I've successfully applied the Redbarn patches to my BIND, and I expect the NSD rate-control to be of similar quality, or better. -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 BELA LUGOSI is my co-pilot ...
On 2013-03-25, at 16:51, Måns Nilsson <mansaxel@besserwisser.org> wrote:
I've successfully applied the Redbarn patches to my BIND, and I expect the NSD rate-control to be of similar quality, or better.
We've formed the opinion at ICANN that the observed reaction to reflection attacks by BIND9 + Schryver/Vixie RRL is definitely different from NSD + NSD-RRL, but we don't yet know whether either one is better. Dave Knight is busy building a test lab at DNS-OARC so he can replay identical attack traffic against BIND9, NSD and knot with equivalent RRL configurations to observe their behaviour. The source data he's using initially is from a reflection attack against L-Root that landed in Hamburg; if others here have full pcaps of similar events and are interested in comparing the reactions to it from those three nameservers, let me know and I can put you in touch. Dave plans to talk about his methodology and findings at the DNS-OARC workshop in Dublin in May (assuming his presentation proposal is accepted). (The DNS-OARC workshop is cojoined with the RIPE meeting, for those who are DNS-curious and haven't already considered a couple of extra days of DNS fun alongside the RIPE meeting they were already planning to attend.) Joe
On Mon, 25 Mar 2013, Alain Hebert wrote:
That might be just me, but I find those peers allowing their customers to spoof source IP addresses more at fault.
Yes, I wish every "broadband speed test" could include spoofed IP. Unfortunately I would imagine most OSes won't allow normal users to spoof sender IP so I guess it's hard to recognise. We need a hall of shame for that. -- Mikael Abrahamsson email: swmike@swm.pp.se
On 25/03/2013 16:35, Alain Hebert wrote:
That might be just me, but I find those peers allowing their customers to spoof source IP addresses more at fault.
that is equally stupid and bad.
PS: Some form of adaptive rate limitation works for it btw =D
no, it doesn't. In order to ensure that your resolver clients are serviced properly, you need to keep the DNS query rate high enough that if someone has a large bcp38-enabled botnet, they can trash the hell out of whoever they want. The best solution is to disable open recursion completely, and police your clients regularly. Nick
Hi, Well... On 03/25/13 12:51, Nick Hilliard wrote:
On 25/03/2013 16:35, Alain Hebert wrote:
That might be just me, but I find those peers allowing their customers to spoof source IP addresses more at fault.
that is equally stupid and bad.
In my eyes, those peers are the source of it. One can justify Open Relay and the lag into moving into not being an attack vector... while the case for allowing IP spoofing is a tad harder to justify.
PS: Some form of adaptive rate limitation works for it btw =D
no, it doesn't. In order to ensure that your resolver clients are serviced properly, you need to keep the DNS query rate high enough that if someone has a large bcp38-enabled botnet, they can trash the hell out of whoever they want.
We all need to be more flexible and actually work toward fixing both end of the issue.
The best solution is to disable open recursion completely, and police your clients regularly.
Nick
I just intervene on one of today's DNS Amp... which is going to many targets mind you... on a client with a NT4.0 Server and another with FreeBSD 5.1 =D ( You can say bye bye to that NT4.0 client revenue :( ) Now about some of "those" peers start enforcing some form of source IP rules... PS: The Open Relay situation is easy to fix for a subscriber type corp (like say a Cable provider)... and less for smaller outfit providing all sort of IT services. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
On Mon, Mar 25, 2013 at 12:51 PM, Nick Hilliard <nick@foobar.org> wrote:
On 25/03/2013 16:35, Alain Hebert wrote:
That might be just me, but I find those peers allowing their customers to spoof source IP addresses more at fault.
that is equally stupid and bad.
Nothing equal about it. Open resolvers (and other forms of amplification attacks like the basic smurf) are a problem if and only if a target's source IP address can be spoofed. Service providers intentionally or negligently permitting their users to spoof source addresses outside that ISP's domain are the *root cause* of the problem. Even if you close all the open resolvers, most authoritative responses are larger than the queries. At best you've shrunk the amplification factor. What will you do next? Insist that everybody host their DNS somewhere sophisticated rather than running their own server? Hassling the folks who run open resolvers further victimizes the innocent. If you want to solve the problem, start by cleaning up your border so that only locally valid sources can exit. Next, identify peers who fail to demonstrate adequate control over their sources. Finally, set filters on those peers so that sources inconsistent with the received routes are dropped. They won't like it. They'll find it inconvenient, even disruptive to their traffic engineering efforts. But at some point, TE has to take a back seat to closing network abuse issues. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On 25/03/2013 17:51, William Herrin wrote:
Hassling the folks who run open resolvers further victimizes the innocent.
running open resolvers will continue to be a major problem as a DDoS platform on the Internet until everyone implements BCP38. When everyone has implemented ingress filtering, we can have a beer and agree that running open resolvers is less harmful. Until then, though, they're a menace. Nick
On Mon, Mar 25, 2013 at 2:09 PM, Nick Hilliard <nick@foobar.org> wrote:
On 25/03/2013 17:51, William Herrin wrote:
Hassling the folks who run open resolvers further victimizes the innocent.
running open resolvers will continue to be a major problem as a DDoS platform on the Internet until everyone implements BCP38. When everyone has implemented ingress filtering, we can have a beer and agree that running open resolvers is less harmful. Until then, though, they're a menace.
Nick, Running [unauthenticated UDP-based service du jour] will continue to be a major problem as a DDoS platform on the Internet until everyone implements BCP38. That [unauthenticated UDP-based service du jour] should thus be disallowed is an untenable position. We depend on [unauthenticated UDP-based service du jour] for the correct operation of the Internet, including such examples as authoritative DNS servers. We've been down this path before where we try to tighten the belt on everything we don't absolutely critically need for the sake of allowing the root problem to keep eking by. It ain't pretty and ultimately it isn't successful either: we merely create an arms race where the bad actors converge on the services we -can't- shut down. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
----- Original Message -----
From: "William Herrin" <bill@herrin.us>
We've been down this path before where we try to tighten the belt on everything we don't absolutely critically need for the sake of allowing the root problem to keep eking by. It ain't pretty and ultimately it isn't successful either: we merely create an arms race where the bad actors converge on the services we -can't- shut down.
Correct. So when the excuse-me-but-fuck are we going to get serious, and apply the IDP to any edge carrier who doesn't implement 38? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
On Mon, 25 Mar 2013, Nick Hilliard wrote:
The best solution is to disable open recursion completely, and police your clients regularly.
Is there an officially sounding document with "open resolver considered harmful" or alike? It's not trivial to deal with a corporate client with an open resolver. You can't really shut them off, can't filter them etc. Googling for <open resolver considered harmful> yields nothing I can point customers to. <http://www.ietf.org/rfc/rfc5358.txt> is the closest I can find? -- Mikael Abrahamsson email: swmike@swm.pp.se
On Mar 25, 2013, at 12:35 PM, Alain Hebert <ahebert@pubnix.net> wrote:
Well,
Why would you only go after them?
Easier target to mitigate the problem?
That might be just me, but I find those peers allowing their customers to spoof source IP addresses more at fault.
PS: Some form of adaptive rate limitation works for it btw =D
Folks should be deploying unicast-rpf facing their statically routed infrastructure. This includes server lans, PPPoE Pools, etc. Place the filtering at the edge where feasible. This would also include things like your firewall and other devices that shouldn't leak/emit spoofed packets. If you don't know how to do this, or check on it, please ask around, either here or on cisco-nsp or juniper-nap for your platforms. - Jared
On 25/03/2013 14:33, Mikael Abrahamsson wrote:
I would like to be able to request an IP list of open resolvers in my ASN, perhaps sent to the contact details in RIPE whois database to make sure I'm not falsely representing that ASN.
Or you could just get an off-site system (cloud VM), get the software from http://monkey.org/~provos/dnsscan/, and find all your own open recursive DNS servers. There are different levels of openness for recursive DNS servers though. It looks like Jared's project lists any DNS server that responds with anything other than refused as open. A DNS server could have open recursion "disabled", but still respond with referrals to the root-servers. Older versions of bind seem to do this when configured with allow-recursion for a limited range of IPs. While not really "open" such servers are still useful for DNS amplification. The example config at http://www.team-cymru.org/Services/Resolvers/instructions.html for a bind 9.x caching server can be adapted for older bind versions doing caching+authoratative such that it'll provide recursion to those who should have it, and authority for zones for which it needs to do so. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Mar 26, 2013, at 10:15 AM, Jon Lewis <jlewis@lewis.org> wrote:
On 25/03/2013 14:33, Mikael Abrahamsson wrote:
I would like to be able to request an IP list of open resolvers in my ASN, perhaps sent to the contact details in RIPE whois database to make sure I'm not falsely representing that ASN.
Or you could just get an off-site system (cloud VM), get the software from http://monkey.org/~provos/dnsscan/, and find all your own open recursive DNS servers.
There are different levels of openness for recursive DNS servers though. It looks like Jared's project lists any DNS server that responds with anything other than refused as open. A DNS server could have open recursion "disabled", but still respond with referrals to the root-servers. Older versions of bind seem to do this when configured with allow-recursion for a limited range of IPs. While not really "open" such servers are still useful for DNS amplification. The example config at
http://www.team-cymru.org/Services/Resolvers/instructions.html
for a bind 9.x caching server can be adapted for older bind versions doing caching+authoratative such that it'll provide recursion to those who should have it, and authority for zones for which it needs to do so.
I was throwing up the 'quick & dirty' data that I had for everyone to get access to quickly. There are a large number of attacks using these servers in the past week. I hope everyone takes a minute and gets with their unix/systems/DNS team and determines what they can do to minimize this. One other important item: Stop your customers from being able to spoof! If you punch in 8.8.8.8 (for example) into the system, you will see a number of devices where if a packet is directed at it that respond with 8.8.8.8, either by spoofing, or by forwarding that request to google and spoofing the origin IP. Same for the 73.73.73.73 IP as well. Those CPE devices should be locked down. - Jared
What are those who provide open resolvers, such as google, doing to combat the problem? It would be nice to be able to provide open resolvers as a service and combat the various threats associated with them. Cheers, Harry On 03/25/2013 10:22 AM, Jared Mauch wrote:
All,
Open resolvers pose a security threat. I wanted to let everyone know about a search tool that can help you find the ones within your organization. Treat it like a big "BETA" stamp is across it, but please try it out and see if you can close down any hosts within your network.
This threat is larger than the SMURF amplification attacks in the past and can result in some quite large attacks. I've seen this spilling out into other mailing lists (e.g.: juniper-nap and others).
Please send feedback about links that should be included or documentation and spelling errors to me.
openresolverproject.org
Some basic stats:
27 million resolvers existed as of this dataset collection
only 2.1 million of them were "closed".
We have a lot to do to close the hosts, please do what you can to help.
Thanks,
- Jared
I think if we get to that small number from tens of millions then we are in much better shape. Closing them and setting up rate limiting on your authorities will go a long way. Jared Mauch On Mar 25, 2013, at 9:45 AM, Harry Hoffman <hhoffman@ip-solutions.net> wrote:
What are those who provide open resolvers, such as google, doing to combat the problem?
It would be nice to be able to provide open resolvers as a service and combat the various threats associated with them.
Cheers, Harry
On 03/25/2013 10:22 AM, Jared Mauch wrote:
All,
Open resolvers pose a security threat. I wanted to let everyone know about a search tool that can help you find the ones within your organization. Treat it like a big "BETA" stamp is across it, but please try it out and see if you can close down any hosts within your network.
This threat is larger than the SMURF amplification attacks in the past and can result in some quite large attacks. I've seen this spilling out into other mailing lists (e.g.: juniper-nap and others).
Please send feedback about links that should be included or documentation and spelling errors to me.
openresolverproject.org
Some basic stats:
27 million resolvers existed as of this dataset collection
only 2.1 million of them were "closed".
We have a lot to do to close the hosts, please do what you can to help.
Thanks,
- Jared
All,
Open resolvers pose a security threat. I wanted to let everyone know about a search tool that can help you find the ones within your organization. Treat it like a big "BETA" stamp is across it, but please
There are a number of open resolvers that are that way by design (i.e. Google), but most of them are there by misconfiguration, having a small number (say < 100) of well-known open resolvers in the world is not a problem, having > 1 million probably is Mike -----Original Message----- From: Harry Hoffman [mailto:hhoffman@ip-solutions.net] Sent: 25 March 2013 14:46 To: nanog@nanog.org Subject: Re: Open Resolver Problems What are those who provide open resolvers, such as google, doing to combat the problem? It would be nice to be able to provide open resolvers as a service and combat the various threats associated with them. Cheers, Harry On 03/25/2013 10:22 AM, Jared Mauch wrote: try it out and see if you can close down any hosts within your network.
This threat is larger than the SMURF amplification attacks in the past
and can result in some quite large attacks. I've seen this spilling out into other mailing lists (e.g.: juniper-nap and others).
Please send feedback about links that should be included or
documentation and spelling errors to me.
openresolverproject.org
Some basic stats:
27 million resolvers existed as of this dataset collection
only 2.1 million of them were "closed".
We have a lot to do to close the hosts, please do what you can to help.
Thanks,
- Jared
On Mon, Mar 25, 2013 at 7:45 AM, Harry Hoffman <hhoffman@ip-solutions.net>wrote:
What are those who provide open resolvers, such as google, doing to combat the problem?
https://developers.google.com/speed/public-dns/docs/security#rate_limit Damian
On Mon, 25 Mar 2013 10:22:08 -0400, Jared Mauch said:
Some basic stats:
27 million resolvers existed as of this dataset collection
only 2.1 million of them were "closed".
We have a lot to do to close the hosts, please do what you can to help.
What's the current BCP on how to deal with mobile devices that hard-code your resolvers in their equivalent of /etc/resolv.conf (often because the owner of the device trusts their emnployers/whatever resolver more than they trust the DNS server that the hotel DHCP pointed them at)? (And yes, I *know* that "point at your employers DNS" works against a threat model of "local provider is an idiot" and fails against "local provider is willing to spoof replies from other DNS servers")
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net>
Open resolvers pose a security threat.
Could you clarify, here, Jared? Do "open DNS customer-resolver/recursive servers" *per se* cause a problem? Or is it merely "customer zone servers which are misconfigured to recurse", as has always been problematic? That is: is this just a reminder we never closed the old hole, or notification of some new and much nastier hole? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
On Mar 25, 2013, at 2:04 PM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net>
Open resolvers pose a security threat.
Could you clarify, here, Jared?
Do "open DNS customer-resolver/recursive servers" *per se* cause a problem?
Or is it merely "customer zone servers which are misconfigured to recurse", as has always been problematic?
That is: is this just a reminder we never closed the old hole, or notification of some new and much nastier hole?
There have been some moderate size attacks recently that I won't go into detail here about. The IPs that are on the website are certainly being used/abused. A recent attack saw a 90% match rate against the "master list" here. This means your open resolver is likely being used. Anything to raise the bar here will minimize the impact to those networks under attack. Turn on RPF facing your colocation and high-speed server lans. We all know hosts become compromised. Help minimize the impact of these attacks by a) doing BCP-38 b) locking down your recursive servers to networks you control c) locking down your authority servers to not provide the same answer 15x in a second to the same querying IP. If it's asking that same question 15x, then it's not you that's broken, it's that client. (Or it's being abused). - Jared
Well, On 03/25/13 16:45, Jared Mauch wrote:
On Mar 25, 2013, at 2:04 PM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net> Open resolvers pose a security threat. Could you clarify, here, Jared?
Do "open DNS customer-resolver/recursive servers" *per se* cause a problem?
Or is it merely "customer zone servers which are misconfigured to recurse", as has always been problematic?
That is: is this just a reminder we never closed the old hole, or notification of some new and much nastier hole? There have been some moderate size attacks recently that I won't go into detail here about. The IPs that are on the website are certainly being used/abused. A recent attack saw a 90% match rate against the "master list" here. This means your open resolver is likely being used.
Anything to raise the bar here will minimize the impact to those networks under attack. Turn on RPF facing your colocation and high-speed server lans. We all know hosts become compromised. Help minimize the impact of these attacks by
a) doing BCP-38 b) locking down your recursive servers to networks you control c) locking down your authority servers to not provide the same answer 15x in a second to the same querying IP. If it's asking that same question 15x, then it's not you that's broken, it's that client. (Or it's being abused).
- Jared
I think most of the audience here knows and are sensitive about it. The problems come from from those who don't give a *shit*... And they've been not giving a *shit* it for years. The magic is in "how" to make them care. Do the industry need to go "a la PCI-DSS" for Peers? PS: My pico ISP is soooo on your list Jared =D Not for long hopefully. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
In message <5150BE64.2020907@pubnix.net>, Alain Hebert writes:
Well,
On Mar 25, 2013, at 2:04 PM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net> Open resolvers pose a security threat. Could you clarify, here, Jared?
Do "open DNS customer-resolver/recursive servers" *per se* cause a problem?
Or is it merely "customer zone servers which are misconfigured to recurse", as has always been problematic?
That is: is this just a reminder we never closed the old hole, or notification of some new and much nastier hole? There have been some moderate size attacks recently that I won't go into detail here about. The IPs that a re on the website are certainly being used/abused. A recent attack saw a 90% match rate against the "master
On 03/25/13 16:45, Jared Mauch wrote: list" here. This means your open resolver is likely being used.
Anything to raise the bar here will minimize the impact to those networks under attack. Turn on RPF facing
your colocation and high-speed server lans. We all know hosts become compromised. Help minimize the impact of these attacks by
a) doing BCP-38 b) locking down your recursive servers to networks you control c) locking down your authority servers to not provide the same answer 15x in a second to the same querying
IP. If it's asking that same question 15x, then it's not you that's broken, it's that client. (Or it's bein g abused).
- Jared
I think most of the audience here knows and are sensitive about it.
The problems come from from those who don't give a *shit*... And they've been not giving a *shit* it for years.
The magic is in "how" to make them care.
There is only one way sure way to make them care which is to cut them off for a period and repeat the punishment if they fail to clean up their act. You give them notice. You publicise that you are going to do it unless they address their issue by date X. On date X you stop accepting routes through them or to them unless they have cleaned up their act. At the end of the period you start accepting traffic again. You leave the open recursive servers open. They are your canaries. BCP 38 was published in May 2000. There is no excuse for any ISP to not have the requisite equipement to do this.
Do the industry need to go "a la PCI-DSS" for Peers?
PS: My pico ISP is soooo on your list Jared =D Not for long hopefully.
----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Mar 25, 2013, at 5:15 PM, Alain Hebert <ahebert@pubnix.net> wrote:
Well,
On 03/25/13 16:45, Jared Mauch wrote:
On Mar 25, 2013, at 2:04 PM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net> Open resolvers pose a security threat. Could you clarify, here, Jared?
Do "open DNS customer-resolver/recursive servers" *per se* cause a problem?
Or is it merely "customer zone servers which are misconfigured to recurse", as has always been problematic?
That is: is this just a reminder we never closed the old hole, or notification of some new and much nastier hole? There have been some moderate size attacks recently that I won't go into detail here about. The IPs that are on the website are certainly being used/abused. A recent attack saw a 90% match rate against the "master list" here. This means your open resolver is likely being used.
Anything to raise the bar here will minimize the impact to those networks under attack. Turn on RPF facing your colocation and high-speed server lans. We all know hosts become compromised. Help minimize the impact of these attacks by
a) doing BCP-38 b) locking down your recursive servers to networks you control c) locking down your authority servers to not provide the same answer 15x in a second to the same querying IP. If it's asking that same question 15x, then it's not you that's broken, it's that client. (Or it's being abused).
- Jared
I think most of the audience here knows and are sensitive about it.
The problems come from from those who don't give a *shit*... And they've been not giving a *shit* it for years.
The magic is in "how" to make them care
If this started to move into an AUP violation direction (e.g.: ala spammers, etc) would that motivate people?
Do the industry need to go "a la PCI-DSS" for Peers?
I think that any effort we can take here to help educate people to the right standards is helpful. I'd like to see people fix hosts, routers and a number of other things.
PS: My pico ISP is soooo on your list Jared =D Not for long hopefully.
Appreciated. And many thanks for others that have emailed me saying their hosts have been fixed as well, and those that have emailed me updated text for the webpage. - jared
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net>
Open resolvers pose a security threat.
Could you clarify, here, Jared?
Do "open DNS customer-resolver/recursive servers" *per se* cause a
From: Jared Mauch [mailto:jared@puck.nether.net] On Mar 25, 2013, at 2:04 PM, Jay Ashworth <jra@baylink.com> wrote: problem?
Or is it merely "customer zone servers which are misconfigured to recurse", as has always been problematic?
That is: is this just a reminder we never closed the old hole, or notification of some new and much nastier hole?
There have been some moderate size attacks recently that I won't go into detail here about. The IPs that are on the website are certainly being used/abused. A recent attack saw a 90% match rate against the "master list" here. This means your open resolver is likely being used.
I'm just going to jump in here and ask what is probably a silly question, but let's suppose I just happen to have, or have access to, a botnet comprised of (tens of) millions of random hosts all over the internet, and I feel like destroying your DNS servers via DDoS; what's stopping me from just directly querying your servers continuously from said botnet until you melt? Those machines send you traffic indirectly through open resolvers, or hit you directly, but either way, it's the same number of machines issuing the same number of queries, and you're no less inundated. If your own servers rate limit to protect themselves, you're losing valid traffic, and if they don't, once you melt down, you're losing valid traffic... Jamie
On Mar 26, 2013, at 6:50 PM, Jamie Bowden wrote:
let's suppose I just happen to have, or have access to, a botnet comprised of (tens of) millions of random hosts all over the internet, and I feel like destroying your DNS servers via DDoS;
DNS reflection/amplification attacks aren't intended as attacks against the DNS, per se; they're intended to crush any/all targeted servers and/or fill transit pipes. Same for SNMP and ntp reflection attacks. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Mar 26, 2013, at 08:01 , "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On Mar 26, 2013, at 6:50 PM, Jamie Bowden wrote:
let's suppose I just happen to have, or have access to, a botnet comprised of (tens of) millions of random hosts all over the internet, and I feel like destroying your DNS servers via DDoS;
DNS reflection/amplification attacks aren't intended as attacks against the DNS, per se; they're intended to crush any/all targeted servers and/or fill transit pipes.
To be more clear, the point of DNS reflection attacks is to amplify the amount of bandwidth the botnet can muster (and perhaps hide the true source). If you have 10s of millions of bots, you don't need to amplify. You can crush any single IP address on the 'Net.
Same for SNMP and ntp reflection attacks.
And far too many other things. :( -- TTFN, patrick
On Tue, Mar 26, 2013 at 08:07:22AM -0400, Patrick W. Gilmore wrote:
On Mar 26, 2013, at 08:01 , "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On Mar 26, 2013, at 6:50 PM, Jamie Bowden wrote:
let's suppose I just happen to have, or have access to, a botnet comprised of (tens of) millions of random hosts all over the internet, and I feel like destroying your DNS servers via DDoS;
DNS reflection/amplification attacks aren't intended as attacks against the DNS, per se; they're intended to crush any/all targeted servers and/or fill transit pipes.
To be more clear, the point of DNS reflection attacks is to amplify the amount of bandwidth the botnet can muster (and perhaps hide the true source).
If you have 10s of millions of bots, you don't need to amplify. You can crush any single IP address on the 'Net.
TTFN, patrick
"You are the Brut Squad!"
On Mar 26, 2013, at 7:07 PM, Patrick W. Gilmore wrote:
To be more clear, the point of DNS reflection attacks is to amplify the amount of bandwidth the botnet can muster (and perhaps hide the true source).
Yes, hence the 'amplification' part. ;> More than hiding the actual sources, I think it's more about making it difficult (at first blush) for folks to seine out and filter the attack traffic from the normal 'background radiation' of legitimate traffic.
And far too many other things. :(
Good point - game servers, etc. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net>
b) locking down your recursive servers to networks you control
Sure. But OpenDNS, Google, and the other providers of recursive servers for edge cases can't do that anymore? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274
On Mar 26, 2013, at 10:38 , Jay Ashworth <jra@baylink.com> wrote:
From: "Jared Mauch" <jared@puck.nether.net>
b) locking down your recursive servers to networks you control
Sure. But OpenDNS, Google, and the other providers of recursive servers for edge cases can't do that anymore?
I wish people would stop bring that up. I guarantee I see at least as many reflection attack as anyone out there. I have not _once_ called/emailed Open, Google, Dyn, Ultra, or any other professional DNS provider asking them to stop amplifying attacks to us. If you can run a server as competently as they can, then no one will complain. For the other 99.99999999% of you, LOCK THAT SHIT DOWN. -- TTFN, patrick
Well, And why not targeting all that animosity to the peers allowing source IP spoofing? DNS Servers don't attack you, people letting their customers spoof source IP do. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 03/26/13 10:46, Nick Hilliard wrote:
On 26/03/2013 14:43, Patrick W. Gilmore wrote:
For the other 99.99999999% of you, LOCK THAT SHIT DOWN. amen, brother!
On Mar 26, 2013, at 11:06 AM, Alain Hebert <ahebert@pubnix.net> wrote:
Well,
And why not targeting all that animosity to the peers allowing source IP spoofing?
DNS Servers don't attack you, people letting their customers spoof source IP do.
I don't view this as an either-or situation. One needs both. - Jared
On 26/03/2013 15:06, Alain Hebert wrote:
And why not targeting all that animosity to the peers allowing source IP spoofing?
I do - and I gave a bunch of talks in europistan over the last 12 months which included explicit encouragement, practice and configuration for implementing BCP38 as part of real-time black hole system deployment.
DNS Servers don't attack you, people letting their customers spoof source IP do.
DNS amp packets attack me. Please stop them from leaving your network, and I will both implement BCP38 and encourage others to do so. Thank you. Nick
Well, On 03/26/13 11:38, Nick Hilliard wrote:
On 26/03/2013 15:06, Alain Hebert wrote:
And why not targeting all that animosity to the peers allowing source IP spoofing?
I do - and I gave a bunch of talks in europistan over the last 12 months which included explicit encouragement, practice and configuration for implementing BCP38 as part of real-time black hole system deployment.
DNS Servers don't attack you, people letting their customers spoof source IP do.
DNS amp packets attack me. Please stop them from leaving your network, and I will both implement BCP38 and encourage others to do so. Thank you.
Nick We're on it here...
Been using the work of http://bindguard.activezone.de/ to watch it =D There is a lot of targets... kinda hard to figure out the goal... ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
Am 27.03.2013 00:04, schrieb Alain Hebert:
We're on it here...
Been using the work of http://bindguard.activezone.de/ to watch it =D
There is a lot of targets... kinda hard to figure out the goal...
----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
A new export of my blocking list is now available. Over 1500 bad hosts are now in my bogon configuration and every day it be more and more ... See here: http://bindguard.activezone.de/bogon-list.txt In the last 10 weeks are 749 new blocked hosts [ result of a small environment ] :-( Mar 28 07:40:56 BindGuard: Stats (PID 34874): 761 entries, 525851 updates, 885070 ignored, 749 blocked, 1486207 events parsed. Mar 28 07:40:56 BindGuard: Used Memory: 0.24 MByte (Index: 131072 Bytes, Free: 15624 / Entries: 125670 Bytes) Mar 28 07:40:56 BindGuard: Version 0.69, Uptime: 10 weeks, 6 days, 23 hours, 12 minutes, 38 seconds Regards, Markus
I wanted to share PER-ASN data for those that are interested in this generally. If you are a contact for these ASNs, you can e-mail me from your corporate address to get access to the list. Thank you for many of you that have secured hosts!!!! COUNT ASN# 1357979 4134 1144551 8151 1089464 9121 896102 3462 623486 45899 618161 4837 554655 17974 513878 13285 486146 22927 486081 3352 483686 209 457952 3269 423432 9318 377811 1267 356685 5617 334125 19262 318274 24560 291114 4766 266952 9198 248563 9050 245165 18403 233639 18881 230318 9737 229251 7418 228180 8167 223017 36947 221978 7303 221487 43234 219076 3215 216288 8452 207389 3816 197305 5610 194241 45758 194227 45629 191824 19429 184158 4812 183616 6697 177759 9829 177058 17813 175616 36351 166778 17552 163671 6147 140349 8400 138785 21844 138369 9299 128307 8866 120188 13489 117991 5390 115057 4788 111726 7552 108487 9394 105874 47589 94346 4713 93301 7738 91777 6849 85895 24863 82668 12741 79534 9269 77572 6389 77428 32244 77166 9105 76574 12874 76104 4808 73145 24940 67576 6400 66648 12880 65260 45595 64153 32475 63932 6332 60420 2379 54987 14420 54299 12297 52710 6830 52274 30722 51229 26496 49623 6057 49488 13036 48045 3320 47141 2554 46894 5384 46790 6855 44998 32613 44090 2516 42977 36444 42588 36992 40757 7018 39099 46606 38890 7385 38714 5650 38622 9158 38466 8997 38451 8560 37635 2856 37494 5089 36477 6730 36057 7132 35626 12301 35556 8612 34938 24835 34652 31549 34547 22394 34546 6713 34274 12978 34214 8359 33944 22561 33886 44957 33763 286 33548 1136 32513 17816 32481 55430 32439 7545 31919 4780 30890 27695 30410 21788 29843 15557 29639 28812 29512 1221 29463 35805 28758 41440 28602 47794 28533 43447 28494 16265 28268 34296 28196 25019 28000 4847 27701 13354 27498 9146 27370 27699 27211 6799 27021 13768 27017 20773 26858 17676 26622 41592 26610 44178 26233 25847 25962 2609 25751 3301 25732 15975 25730 8376 25526 29256 24531 19066 24342 6983 23560 25405 23445 33774 23185 17511 23058 34170 22868 17430 22670 8926 22003 19029 21958 4538 21860 12576 21596 25653 21496 10036 21009 29386 20915 8972 20640 9808 20429 22833 20375 13999 20102 11398 19931 30496 19906 34984 19839 5778 19635 9824 19469 12912 19350 17858 19327 9143 18896 9329 18878 7029 18757 8968 18667 56046 18658 8708 18309 6877 18040 25513 17877 11830 17718 4802 17699 11556 17619 2119 16924 28787 16772 4230 16672 174 16373 7497 16370 53006 16309 29550 16079 17762 15825 23352 15718 40244 15673 29049 15590 17964 15571 4725 15289 13213 15122 25144 15026 33182 14698 56040 14615 35819 14613 10474 14555 20860 14410 21003 14408 48159 14381 2514 14219 26599 14174 9790 14149 9125 14147 22773 14138 33812 14132 2527 14107 12260 14043 3303 14001 29182 13969 20825 13881 18566 13865 25515 13863 56041 13710 20738 13612 25490 13511 9605 13360 1785 13333 23752 13147 35041 13048 9845 12989 39501 12986 9924 12938 3292 12902 2828 12886 6703 12831 2518 12704 20676 12328 20115 12325 46475 12306 12486 12216 4323 12209 15589 12206 32748 12180 701 12105 36137 12083 3209 11967 8551 11934 23889 11916 4775 11903 13977 11821 3329 11771 10299 11768 22822 11740 16086 11705 9116 11705 3651 11570 25124 11533 33070 11531 12683 11483 6739 11475 15003 11459 45951 11423 6871 11364 131090 11244 24158 11217 16276 11115 5432 11079 5410 10726 2497 10678 17623 10651 3239 10497 10796 10416 7657 10393 8402 10388 12332 10380 58085 10380 17621 10332 5483 10300 22368 10280 3595 10272 45609 10207 9617
On Thu, 28 Mar 2013 14:16:58 -0400, Jared Mauch said:
I wanted to share PER-ASN data for those that are interested in this generally. If you are a contact for these ASNs, you can e-mail me from your corporate address to get access to the list.
Thank you for many of you that have secured hosts!!!!
I'm embarrassed to admit we appear to have on the order of 150 or so misconfigured hosts. LARTs will be dispatched. :)
On Thu, 28 Mar 2013, Jared Mauch wrote:
I wanted to share PER-ASN data for those that are interested in this generally. If you are a contact for these ASNs, you can e-mail me from your corporate address to get access to the list.
Interesting. My guess is that at least some of these have a lot of CPEs with global resolving turned on on the WAN port. So yes, I think it's very important to have these lists emailed regularily to alert people who might not check themselves all the time. The ISP operated CPEs should hopefully be possible to fix within weeks of notice (hopefully settings and/or software is controlled by the ISP). -- Mikael Abrahamsson email: swmike@swm.pp.se
On Tue, Mar 26, 2013 at 7:38 AM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Jared Mauch" <jared@puck.nether.net>
b) locking down your recursive servers to networks you control
Sure. But OpenDNS, Google, and the other providers of recursive servers for edge cases can't do that anymore?
Of cos they can. But they take the security of their open recursive servers very seriously. 99.99999% of the open recursors dont, hence the problem.
On Tue, 26 Mar 2013 07:43:15 -0700, Tom Paseka said:
On Tue, Mar 26, 2013 at 7:38 AM, Jay Ashworth <jra@baylink.com> wrote:
Sure. But OpenDNS, Google, and the other providers of recursive servers for edge cases can't do that anymore?
Of cos they can. But they take the security of their open recursive servers very seriously. 99.99999% of the open recursors dont, hence the problem.
And what, *exactly* do they do different from the other 5-9's? So far, I've seen lots of people say "close that shit down", but only 2 actual URLs posted that basically both say "only do recursion for IP addresses within your ASN". That's at least a *bit* more helpful than just telling us to close it down. Unfortunately, we already know now to do that - but that isn't the problem that some of us are looking to solve, which is "queries from your own users mobile devices that are currently *outside* your ASN". (And *please* make note that although the fine networking staff of AS1312 can probably figure this out on our own once we're supplied with a big enough pile of square tuits and a belt sander, there's a *lot* of AS's out there that are going to need a tad more hand-holding...)
https://developers.google.com/speed/public-dns/docs/security Cheers, Harry On 03/26/2013 11:07 AM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 26 Mar 2013 07:43:15 -0700, Tom Paseka said:
On Tue, Mar 26, 2013 at 7:38 AM, Jay Ashworth <jra@baylink.com> wrote:
Sure. But OpenDNS, Google, and the other providers of recursive servers for edge cases can't do that anymore?
Of cos they can. But they take the security of their open recursive servers very seriously. 99.99999% of the open recursors dont, hence the problem.
And what, *exactly* do they do different from the other 5-9's?
So far, I've seen lots of people say "close that shit down", but only 2 actual URLs posted that basically both say "only do recursion for IP addresses within your ASN". That's at least a *bit* more helpful than just telling us to close it down. Unfortunately, we already know now to do that - but that isn't the problem that some of us are looking to solve, which is "queries from your own users mobile devices that are currently *outside* your ASN".
(And *please* make note that although the fine networking staff of AS1312 can probably figure this out on our own once we're supplied with a big enough pile of square tuits and a belt sander, there's a *lot* of AS's out there that are going to need a tad more hand-holding...)
On Tue, 26 Mar 2013 12:59:25 -0400, Harry Hoffman said:
https://developers.google.com/speed/public-dns/docs/security
Thanks :)
On 2013-03-26, at 11:07, Valdis.Kletnieks@vt.edu wrote:
which is "queries from your own users mobile devices that are currently *outside* your ASN".
This problem made more sense to me ten years ago than it does now. What mobile devices do you support that don't acquire a suitable local DNS resolver using DHCP or PPP? Honest question. I presume you wouldn't bring it up if it wasn't a real problem. Joe
On Tue, 26 Mar 2013 13:09:53 -0400, Joe Abley said:
What mobile devices do you support that don't acquire a suitable local DNS resolver using DHCP or PPP?
Pretty much all devices are *able* to acquire a DNS resolver via DHCP.
Honest question. I presume you wouldn't bring it up if it wasn't a real problem.
The problem starts when you don't *trust* DHCP to hand you a pointer to a *working* DNS resolver (anybody who's had a hotel net hand them a DNS that's either busted or MITMs your queries knows what I mean, and I hope I don't have to explain about the fun involved in using wireless anywhere near a DefCon or Black Hat conference). And yes, unless you turn on DNSSEC you don't have much defense against a hotel net or rogue net that decides to spoof replies to your queries to your home DNS server Now in day-to-day production, it's *mostly* a non-issue, because many/most of the people who hard-code our DNS into their mobile configs will also fire up a VPN to our campus. Unfortunately, that leaves us a lot of interesting to diagnose corner cases involving DNS lookups that happen between when they boot the device and when they launch the VPN (for instance, coding a DNS name rather than an IP for the VPN endpoint :)
participants (45)
-
Alain Hebert -
Ben Aitchison -
bmanning＠vacation.karoshi.com -
Chris Adams -
Chris Boyd -
Christopher Morrow -
Damian Menscher -
David Conrad -
Dobbins, Roland -
Doug Barton -
Harry Hoffman -
Jack Bates -
Jamie Bowden -
Jared Mauch -
Jay Ashworth -
Jimmy Hess -
Joe Abley -
Joe Greco -
joel jaeggli -
John Kristoff -
John Levine -
Jon Lewis -
Leo Bicknell -
Marco Davids -
Mark Andrews -
Masataka Ohta -
Matthew Petach -
Mattias Ahnberg -
Michael DeMan -
Mikael Abrahamsson -
Mike Simkins -
Måns Nilsson -
nanog＠mitteilung.com -
Nick Hilliard -
Owen DeLong -
Patrick W. Gilmore -
Paul Ferguson -
Rich Kulawiec -
Saku Ytti -
Scott Noel-Hemming -
Steve Hillier -
Tom Paseka -
Tony Finch -
Valdis.Kletnieks＠vt.edu -
William Herrin