Given the recent DNS amplification attacks, I've audit and updated our authoritative servers. We are using 9.6.0-P1 now. I've been using the cyrmu templates, but one thing I see is that the dns queries to the . hint file are still occuring and are not being denied by our servers. For example: 27-Jan-2009 15:00:22.963 queries: client 64.57.246.146#64176: view external-in: query: . IN NS + 27-Jan-2009 15:00:23.118 queries: client 64.57.246.146#33146: view external-in: query: . IN NS + the named.conf has: ... ... ... view "external-in" in { match-clients { any; }; recursion no; additional-from-auth no; additional-from-cache no; zone "." in { type hint; file "db.cache"; }; ... ... since you can't put a "allow-query { none; };" in a hint zone, what can I do to deny the query to the . zone file? ---- Matthew Huff       | One Manhattanville Rd OTA Management LLC | Purchase, NY 10577 http://www.ox.com | Phone: 914-460-4039 aim: matthewbhuff  | Fax:   914-460-4139

In message <200901272116.n0RLGiJA002250@ns1.konadogs.net>, Nate Itkin writes:

On Tue, Jan 27, 2009 at 03:04:19PM -0500, Matthew Huff wrote:

...

< ... snip ... > dns queries to the . hint file are still occuring and are not being denied by our servers. For example: 27-Jan-2009 15:00:22.963 queries: client 64.57.246.146#64176: view external-in: query: . IN NS + < ... snip ... > since you can't put a "allow-query { none; };" in a hint zone, what can I d o to deny the query to the . zone file?

AFAIK, that's about the best you can do with the DNS configuration. You've mitigated the amplification value, so hopefully the perpetrator(s) will drop you. If you're willing to keep up with the moving targets, the next level is an inbound packet filter. Add to your inbound ACL:

deny udp host 64.57.246.146 neq 53 any eq 53

Which pre-supposes that 64.57.246.146 os not emitting queries of its own. BCP 140 looked at this problem and concluded that sending REFUSED was the best general guidance that can be given. While BCP 140 applies to recursive servers, returning REFUSED to queries which are not within the namespace served by authoritative servers is entirely consistant with BCP 140.

Also on this topic: Coincident with this DNS DOS, I started seeing inbound PTR queries from various hosts on 10.0.0.0/8 (which are blackholed by my DNS servers). They receive no response, yet they persist. Anyone have thoughts on their part in the scheme?

If you don't implement BCP 38 you don't block bogus traffic. Unless you are using 10.0.0.0/8 then you aren't implementing BCP 38 either. If you were you wouldn't be seeing queries from 10.0.0.0/8. Mark

Best wishes, Nate Itkin

-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org

At 03:16 PM 1/27/2009, Nate Itkin wrote:

On Tue, Jan 27, 2009 at 03:04:19PM -0500, Matthew Huff wrote:

...

< ... snip ... > dns queries to the . hint file are still occuring and are not being denied by our servers. For example: 27-Jan-2009 15:00:22.963 queries: client 64.57.246.146#64176: view external-in: query: . IN NS + < ... snip ... > since you can't put a "allow-query { none; };" in a hint zone, what can I do to deny the query to the . zone file?

AFAIK, that's about the best you can do with the DNS configuration. You've mitigated the amplification value, so hopefully the perpetrator(s) will drop you. If you're willing to keep up with the moving targets, the next level is an inbound packet filter. Add to your inbound ACL:

deny udp host 64.57.246.146 neq 53 any eq 53

Also on this topic: Coincident with this DNS DOS, I started seeing inbound PTR queries from various hosts on 10.0.0.0/8 (which are blackholed by my DNS servers). They receive no response, yet they persist. Anyone have thoughts on their part in the scheme?

Best wishes, Nate Itkin

I'm not seeing those PTR queries for 10.0.0.0/8, but then my perimeter ingress/egress filters (BCP 38) toss most of that kind of junk before my DNS servers ever see it. I agree that is as far as one can go with BIND, right now. However, that isn't making the perpetrators cease and desist. I am seeing ongoing query attempts coming in and refusal packets going back out, and the targets don't seem to change until I do something to block them. So mitigating the amplification factor does not seem of interest to these perpetrators. On the contrary, even REFUSED responses can aggregate with some amplified responses to enhance the apparent DoS goal. Thus BCP 140 seems to be less than completely effective because it is less than universally applied (i.e., older versions of BIND or misconfigured BIND.) I think the same situation is true with BCP 38: less than universally applied. So do I wait for universal application of these BCPs, or do I take responsibility for doing what I can to make my network resources less appealing for abuse? I choose the latter, and that is why went to the effort of blocking this abusive traffic before it reaches my authoritative-only DNS servers. Nevertheless, I also agree with a point made last week that trying to keep up with the changing targets is a game of whack-a-mole that is and will continue to be a drain on network management resources -- if the detection and response continues to be deployed manually. This is why I wrote some Perl for my authoritative-only servers to automate detection and response at the server level. Granted it isn't a permanent solution, but at least it is a place to start. I appended that code below for those who are interested in it. Notes ----- * Does not blindly block all queries from an IP address, as would be the case with an ACL. * Assumes BIND 9.4+ configured for authoritative-only role and configured to respond with REFUSE for queries to zones for which it is not authoritative. * Uses BIND 9.4+ syslog messages and a state table as inputs. * Alters IPtables rules and a state table as outputs. * Runs as a periodic cron job (currently every 10 minutes). * Performs a logtail on BIND 9.4+ syslog messages to detect abusive queries matching a pattern. * Implements a minimum detection threshold to reduce the false positive rate (currently 1/minute a.k.a. 10/cycle) * Implements a state table with last-seen timestamps to maintain state between job runs. * Implements an expiry mechanism (currently 24 hours) which is extended each time a source is re-detected. Dependencies ------------ * Linux 2.4+ kernel * BIND 9.4+ * An account sufficiently privileged to read local syslog and modify IPtables rules. * Logtail * Cron */10 * * * * /usr/local/sbin/dns-reflecter-finder * IPtables key rules: iptables -N dns-reflecter iptables -A {INPUT} -p udp --sport ! 53 --dport 53 -j dns-reflecter (place this above your general ALLOW rules for DNS) #!/usr/bin/perl # use strict; use Data::Dumper; my $BASENAME = "/bin/basename"; my $LN = "/bin/ln"; my $LOGTAIL = "/usr/sbin/logtail"; my $IPTABLES = "/sbin/iptables"; my $LOGDIR = "/var/log"; my $RUNDIR = "/var/run"; my $Progname = $0; $Progname = `$BASENAME $Progname`; chomp ($Progname); my $IPtablesChain = "dns-reflecter"; my $RealLogFile = "$LOGDIR/messages"; my $LogFile = "$LOGDIR/$Progname.log"; my $DBfile = "$RUNDIR/$Progname.dat"; my $DetectPeriod = 600; # This should match the cron period my $DetectThold = $DetectPeriod / 60; my $ExpiryPeriod = 60 * 60 * 24; my $Now = time(); my $Debug = 0; if ($#ARGV >= 0 && $ARGV[0] eq "-debug") { $Debug++; } # # Set up the symlink to the real log file. # unless (-l $LogFile) { if ($Debug) { print "$LN -s $RealLogFile $LogFile\n"; } else { system ($LN, "-s", $RealLogFile, $LogFile); } } # # Find all unread log entries that are refused ". IN NS" queries with # source port number != 53; # my %IPaddrsMatched = (); my $ipaddr = ""; my $portnum = ""; if (open (LOGS, "$LOGTAIL $LogFile|")) { while (<LOGS>) { chomp; if (/named\[\d+\]: client ([\d\.]+)\#(\d+): .* \'\.\/NS\/IN\' denied$/o) { $ipaddr = $1; $portnum = $2; if ($portnum != 53) { $IPaddrsMatched{$ipaddr}++; } } } close (LOGS); } else { die "Cannot logtail $LogFile: $!\n"; } if ($Debug) { print Dumper (\%IPaddrsMatched), "\n"; } # # Delete IP addresses from the running if they hit with # frequency less than the threshold. In this case 1 every # 60 seconds. # foreach $ipaddr (keys %IPaddrsMatched) { if ($IPaddrsMatched{$ipaddr} < $DetectThold) { delete ($IPaddrsMatched{$ipaddr}); } } # # If our db file exists, read it into a hash. # my %IPaddrsCached = (); my $when = 0; if (-f $DBfile) { if (open (DB, $DBfile)) { while (<DB>) { chomp; # # Decode the IP address and the Unix epoch timestamp # when it was last found in logs. # ($ipaddr, $when) = split (/\s+/o); # # If the entry has been quiescent for less than the # expire time, retain it. If not, skip it. # if ($Now - $when <= $ExpiryPeriod) { $IPaddrsCached{$ipaddr} = $when; } } close (DB); } else { die "Cannot read from $DBfile: $!\n"; } } if ($Debug) { print Dumper (\%IPaddrsCached), "\n"; } # # Refresh last-seen timestamps for IP addresses detected # during this run, and add new entries for IP addresses # not previously seen (i.e., never before seen, or # previously seen and expired). # foreach $ipaddr (keys %IPaddrsMatched) { $IPaddrsCached{$ipaddr} = $Now; } # # Write out the updated db file. Overwrite if # previously existed. # if (open (DB, ">$DBfile")) { foreach $ipaddr (sort keys %IPaddrsCached) { print DB "$ipaddr\t$IPaddrsCached{$ipaddr}\n"; } close (DB); } else { die "Cannot write to $DBfile: $!\n"; } # # Flush the dedicated IPtables rule chain. # if ($Debug) { print "$IPTABLES -F $IPtablesChain\n"; } else { system ($IPTABLES, "-F", $IPtablesChain); } # # Add to the dedicated IPtables rule chain all the entries # just written to the db as DROP rules. # foreach $ipaddr (sort keys %IPaddrsCached) { if ($Debug) { print "$IPTABLES -A $IPtablesChain -s $ipaddr -j DROP\n"; } else { system ($IPTABLES, "-A", $IPtablesChain, "-s", $ipaddr, "-j", "DROP"); } } exit 0; -- Douglas C. Stephens | UNIX/Windows/Email Admin System Support Specialist | Network/DNS Admin Information Systems | Phone: (515) 294-6102 Ames Laboratory, US DOE | Email: stephens@ameslab.gov

Mark Andrews wrote:

In message <6.2.3.4.2.20090127162808.02d4ae50@imap.ameslab.gov>, "Douglas C. St ephens" writes:

...

At 03:16 PM 1/27/2009, Nate Itkin wrote:

...

On Tue, Jan 27, 2009 at 03:04:19PM -0500, Matthew Huff wrote:

...

< ... snip ... > dns queries to the . hint file are still occuring and are not being denied by our servers. For example: 27-Jan-2009 15:00:22.963 queries: client 64.57.246.146#64176: view external-in: query: . IN NS + < ... snip ... > since you can't put a "allow-query { none; };" in a hint zone, what can I do to deny the query to the . zone file?

AFAIK, that's about the best you can do with the DNS configuration. You've mitigated the amplification value, so hopefully the perpetrator(s) will drop you. If you're willing to keep up with the moving targets, the next level is an inbound packet filter. Add to your inbound ACL:

deny udp host 64.57.246.146 neq 53 any eq 53

Also on this topic: Coincident with this DNS DOS, I started seeing inbound PTR queries from various hosts on 10.0.0.0/8 (which are blackholed by my DNS servers). They receive no response, yet they persist. Anyone have thoughts on their part in the scheme?

Best wishes, Nate Itkin I'm not seeing those PTR queries for 10.0.0.0/8, but then my perimeter ingress/egress filters (BCP 38) toss most of that kind of junk before my DNS servers ever see it.

I agree that is as far as one can go with BIND, right now. However, that isn't making the perpetrators cease and desist. I am seeing ongoing query attempts coming in and refusal packets going back out, and the targets don't seem to change until I do something to block them. So mitigating the amplification factor does not seem of interest to these perpetrators. On the contrary, even REFUSED responses can aggregate with some amplified responses to enhance the apparent DoS goal. Thus BCP 140 seems to be less than completely effective because it is less than universally applied (i.e., older versions of BIND or misconfigured BIND.) I think the same situation is true with BCP 38: less than universally applied. So do I wait for universal application of these BCPs, or do I take responsibility for doing what I can to make my network resources less appealing for abuse?

I choose the latter, and that is why went to the effort of blocking this abusive traffic before it reaches my authoritative-only DNS servers. Nevertheless, I also agree with a point made last week that trying to keep up with the changing targets is a game of whack-a-mole that is and will continue to be a drain on network management resources -- if the detection and respons e continues to be deployed manually. This is why I wrote some Perl for my authoritative-only servers to automate detection and response at the server level. Granted it isn't a permanent solution, but at least it is a place to start. I appended that code below for those who are interested in it.

Which will just make the attacks evolve. It's pretty easy to design a amplifing DNS attack which is almost indetectable unless you know which addresses are being targeted. This one is highly visible in the logs.

A much more productive task would be to trace back the offending traffic and to put into place policies which require BCP 38 deployment by those you connect to.

Mark

Are we still seeing DNS DDoS attack?

On Wed, 28 Jan 2009, jay@miscreant.org wrote:

Quoting John Martinez <jmartinez@zero11.com>:

...

Are we still seeing DNS DDoS attack?

Yep. I'm seeing ~2 queries/sec targetting 64.57.246.146.

Also seeing requests from 76.9.16.171 every 1 minute 2 seconds.

I run a small personal nameserver and even I am seeing requests for that address 64.57.246.146 at ~1/sec. How many people have upgraded to the latest version of Bind 9? Reason I ask is that when I do my nightly port scan of my server, I no longer see named listening to udp on a random high order port (for replies I believe?). Almost the next day, I started hearing about/seeing these DNS attacks. Previous nmap scan showed: 53/tcp open domain 53/udp open|filtered domain 33591/udp open|filtered unknown Now nmap shows: 53/tcp open domain 53/udp open|filtered domain The listen port (> 32767 I believe) is no longer there with BIND 9.4.3-P1. The port was bound at startup time and did not change as long as named was still running. -- Steve Equal bytes for women.

I still see a few new ones each day, here is my current bind acl for blocking them: acl blacknet { 69.50.142.11/32; 66.230.160.1/32; 66.230.128.15/32; 76.9.16.171/32; 63.217.28.226/32; 206.71.158.30/32; 64.57.246.146/32; 67.192.144.0/32; }; These have all been seen in the last few days, verified by hand. DZ -----Original Message----- From: John Martinez [mailto:jmartinez@zero11.com] Sent: Wednesday, 28 January 2009 11:59 AM Cc: nanog@nanog.org Subject: Re: Tightened DNS security question re: DNS amplification attacks. Are we still seeing DNS DDoS attack? If you have received this email in error, please notify the sender immediately and erase all copies of the email and any attachments to it. The information contained in this email and any attachments may be private, confidential and legally privileged or the subject of copyright. If you are not the addressee it may be illegal to review, disclose, use, forward, or distribute this email and/or its contents. Unless otherwise specified, the information in the email and any attachments is intended as a guide only and should not be relied upon as legal or technical advice or regarded as a substitute for legal or technical advice in individual cases. Opinions contained in this email or any of its attachments do not necessarily reflect the opinions of ACMA.

You all may wish to check your logs for 202.108.12.112, it could be a new target; although I only saw two requests from it. -- Charles Morris cmorris@cs.odu.edu, cmorris@occs.odu.edu Network Security Administrator, Software Developer Office of Computing and Communications Services, CS Systems Group Old Dominion University http://www.cs.odu.edu/~cmorris

"Douglas C. Stephens" <stephens@ameslab.gov> writes:

... I choose the latter, and that is why went to the effort of blocking this abusive traffic before it reaches my authoritative-only DNS servers.

this is an odd implementation choice. the 1PPS query stream is still using your line even with this defense in place. the 1PPS response stream and the CPU cycles it takes to generate that stream isn't a burden on you (compared to the 1PPS query stream that you can't stop anyway). so the only reason to block these appears to be so that you don't participate in the attack, or in other words to cut down the burden on the victim. with me so far? the burden on the victim isn't going to drop materially by what you did since hundreds of thousands of other servers are not going to block it. but if the victim is a recursive nameserver, then your blocking them will mean that they cannot access the zones you're authoritative for. so, no positive, but some negative, for the only person in the whole equation who can be affected by the blocking you're doing. i don't get it. with a cost:benefit matrix like that one, why is anybody blocking this traffic? (i note with some alarm that ten years after i shot it down, i still get queries to rbl.maps.vix.com, so the possibility that the blocks some folks might put in place to ...manage?... this attack will have a long term bad effect on our heirs and assigns. i know your perl script will expire things 60 seconds after they stop spewing, but i fear that others are blocking these in hardcode. (looking for ". IN NS" as the q-tuple pattern is not a solution, since the bad guys can pretty trivially change the question they ask into one you're willing to answer.) (REFUSED is nice because it's small, but the bad guys aren't lacking for bots to transmit spoofed packets from, they Do Not Need amplification, and all forms of error rate limiting are also new attack vectors.) (is there a name-and-shame registry for networks that do/don't implement BCP38? if not i guess i'll start one and hope that various auditors will use google and notice me.) -- Paul Vixie

Once upon a time, David Andersen <dga@cs.cmu.edu> said:

Actually, ". IN NS" is a particularly useful thing for them to do, because it's an almost globally guaranteed response that will get a large response and be in cache.

That's only true on servers that aren't well-configured.

"<tld>. IN NS", of course, but the set of things that work well for such an attack are relatively limited.

Try "aol.com. MX", "hotmail.com. MX", any domain with a big SPF TXT record, etc. There's nothing really special about ". NS". If somebody is serving cached data to the world (even if they aren't recursing for the world), there are any number of things that are likely in the cache. And, since most people have SMTP servers, it is often easy to "prime" somebody's cache, since the SMTP servers often use the same DNS servers. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.

Pretty soon we need an RBL for DNS-oriented DDoS attacks. =)

in the classic sense, you're wrong. in a neoclassic sense: "maybe". let me explain. the original RBL was designed to reject TCP/25 (SMTP) transactions based on source address reputation. we had a false start where we blackholed these addresses as destinations, figuring that if they couldn't hear our ACK then we would never get into TCP with them. then we decided to reject inside the SMTP server and that's what the world has settled on. but throughout, we have been able to bind a reputation to an IP address and act in some way based on that reputation because TCP more or less requires that a real IP address be used. we're seeing cracks at the edges of this model now, because so many core routers have login: cisco; password: cisco, and it's now trivial for any spammer to inject BGP that either lights up unallocated space or cuts out a piece of somebody else's allocated block. this makes it possible to very temporarily and untraceably speak TCP from addresses that have no reputation (if they're unallocated) or that have a good reputation (if they're cutouts). DNS-oriented attacks are of a completely different kind. today's attacks were precisely described in <http://www.icann.org/en/committees/security/sac004.txt> (which wasn't news in october 2002 but somebody had to write it down so i did). the important statement out of that 4-page document is: "Source addresses that appear at Border or Interior connections are nonrepudiable by nature..." and that statement bears on the question of RBL for DNS-oriented DDoS attacks since the address we'd have to assign a reputation to is the victim, so all we can do is make an attack worse (by denying service to the victim's real traffic.) i've pondered whether a network reputation service based on morality rather than behaviour could possibly work. in other words an RBL-like entity that did not prevent attacks, but rather, which denied service to collaborators. if someone claims they can't deploy BCP38 and if their network or nameserver is then used as a launch point for spoofed packets toward reflectors and amplifiers, then would anyone be willing to deny service to them -- to paint them as having a negative reputation even though their "sin" is laziness or cluelessness rather than malevolent intent? and if someone claims they can't turn off open recursion and they get used as an amplifier, would anyone here be willing to deny that recursive server access to their authority server, not on the basis that it might attack them, but strictly to pressure an upgrade? note, i'm speaking as a concerned internet citizen here, not as an ARIN trustee or as ISC's president. i really want to know if folks would be willing to shun eachother not on the basis of evil but rather complacency.

http://www.internetpulse.net/

From the outages list:

Cogent is currently experiencing problems on their backbone in Chicago, manifesting as packet loss and latency. The master ticket # is 853582. On Wed, 2009-01-28 at 12:27 -0800, John Martinez wrote:

http://www.internetpulse.net/

-- "Prediction is very difficult, especially about the future." Niels Bohr -- Ray Sanders Linux Administrator Village Voice Media Office: 602-744-6547 Cell: 602-300-4344

Wil Schultz wrote:

https://puck.nether.net/pipermail/outages/2009-January/001101.html

-wil

On Jan 28, 2009, at 12:27 PM, John Martinez wrote:

...

http://www.internetpulse.net/

Thank you all.

Ryan Werber wrote:

That ticket was opened yesterday, and we have been hit very hard with it. This problem started on Monday night - I don't know what they did, but we lost all of our Toronto sites in the middle of the night for a good bit - so I assume maintenance - Then all h*ll broke loose over the last couple days.

...

https://puck.nether.net/pipermail/outages/2009-January/001101.html

-wil

On Jan 28, 2009, at 12:27 PM, John Martinez wrote:

...

http://www.internetpulse.net/

Ryan Werber Sr. Network Engineer Epik Networks

We saw the issue starting to show up in our monitors at 8:55 EST.

On 2009-01-28 at 19:30 +0000, Paul Vixie wrote:

DNS-oriented attacks are of a completely different kind. today's attacks were precisely described in <http://www.icann.org/en/committees/security/sac004.txt> (which wasn't news in october 2002 but somebody had to write it down so i did). the important statement out of that 4-page document is: "Source addresses that appear at Border or Interior connections are nonrepudiable by nature..." and that statement bears on the question of RBL for DNS-oriented DDoS attacks since the address we'd have to assign a reputation to is the victim, so all we can do is make an attack worse (by denying service to the victim's real traffic.)

I'd be willing to drop DNS queries, without even sending back a REFUSED response, if they come in with RD set from an IP in a list X, where X might be an RBL. I'd be perfectly happy to have X list every root server, gTLD server and ccTLD server, as a starting point, on the basis that none of those should ever be sending out RD queries, so refusing to reply to those addresses should have no impact. Perhaps if operators start to do this, anyone still running critical infrastructure authoritative DNS servers which perform recursive queries would finally split roles. Smaller outfits might get away with an auth server which does recursion too, for a finite set of IPs (eg, "localhost"), if they have issues obtaining IP addresses. Anyone providing nameservers for gTLD or ccTLD zones should not have this problem. (My personal nameserver is in the smaller outfit category here). Now, implementing the RBL against only recursive queries is a separate issue; without nameserver support, you're obviously down to packet filtering. bind supports complete blackholes, but not RD blackholes, AFAIK, but you'd be in a better position than me to say what's coming in bind. iptables can apparently perform payload inspection, but pf definitely can't (at this time). By this, I mean filtering on udp[10] & 0xf9 = 1 [1] Regards, -Phil [1] That's from the tcpdump rule I'm using to glance at this traffic; intf=ifname0 # wire ethernet device ipv4=192.0.2.1 # local IP tcpdump -vvvnpi $intf -Xs 1500 "( (dst host $ipv4 and dst port 53 and (udp[10] & 0xf9 = 1)) or (src host $ipv4 and src port 53 and (udp[10:2] & 0xfc80 = 0x8000)) )" Queries: QR=0, Opcode=0, RD=1 Responses: QR=1, Opcode=0, AA=0 And all assuming that we're only worried about UDP queries, since a TCP query implies the three-way handshake and if that's susceptible to spoofing then there are routing issues too.

In message <20090128232123.GA66921@redoubt.spodhuis.org>, Phil Pennock writes:

Sorry to follow up to myself; a few more moments reviewing before sending were warranted.

On 2009-01-28 at 15:11 -0800, Phil Pennock wrote:

...

I'd be perfectly happy to have X list every root server, gTLD server and ccTLD server, as a starting point, on the basis that none of those should ever be sending out RD queries,

Before I get grilled on this point: it's not strictly true, since obviously things like looking up the IPs of secondary servers to send NOTIFY requests to may use recursive DNS.

Only if you have configured a forwarder. Nameserver make non- recursive queries by default.

Okay, unless you're running a nameserver which secondaries from the gTLD/ccTLD/root servers, you have no reason to see RD packets from those servers. Hopefully that's accurate enough to appease people who'll otherwise concentrate on that point and lose sight of what I was trying to show -- that *most* people could easily make use of such an RBL, if the nameservers supported using an external file for ignoring RD queries without dropping all traffic.

As people upgrade Bind naturally, the number of reflectors that could participate in an attack would go down. Get the OS vendors to use default configs which set a Bind option to maintain the file automatically and you're getting most of the way there, by sheer number of DNS servers.

-Phil

The most common reason for recursive queries to a authoritative server is someone using dig, nslookup or similar and forgeting to disable recursion on the request. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org

On 2009-01-29 at 14:01 +0100, Florian Weimer wrote:

* Mark Andrews:

...

The most common reason for recursive queries to a authoritative server is someone using dig, nslookup or similar and forgeting to disable recursion on the request.

Useful to know, thanks. So someone performing diagnostics on one of the root/gTLD/ccTLD servers would need to remember to dig +norec when checking visibility? Are manual diagnostics going out from the source IP of such auth nameservers considered common? In any case, it's a small enough, and hopefully clued enough, sample of admins that it shouldn't be a problem. Any organisation seeking to add their auth nameservers to a public RBL of such IPs will have to accept the same constraint on needing clued staff. No tears shed at that.

dnscache in "forward only" mode also sets the RD bit, and apparently does not restrict itself to the configured forwarders list. (This is based on a public report, not on first-hand knowledge.)

Unless any of the root/gTLD/ccTLD nameservers are also running dnscache, it should be safe to drop UDP RD packets from those source IP addresses, as previously described. -Phil

At 09:21 PM 1/27/2009, Paul Vixie wrote:

"Douglas C. Stephens" <stephens@ameslab.gov> writes:

...

... I choose the latter, and that is why went to the effort of blocking this abusive traffic before it reaches my authoritative-only DNS servers.

this is an odd implementation choice. the 1PPS query stream is still using your line even with this defense in place. the 1PPS response stream and the CPU cycles it takes to generate that stream isn't a burden on you (compared to the 1PPS query stream that you can't stop anyway). so the only reason to block these appears to be so that you don't participate in the attack, or in other words to cut down the burden on the victim. with me so far?

the burden on the victim isn't going to drop materially by what you did since hundreds of thousands of other servers are not going to block it. but if the victim is a recursive nameserver, then your blocking them will mean that they cannot access the zones you're authoritative for. so, no positive, but some negative, for the only person in the whole equation who can be affected by the blocking you're doing.

i don't get it. with a cost:benefit matrix like that one, why is anybody blocking this traffic? (i note with some alarm that ten years after i shot it down, i still get queries to rbl.maps.vix.com, so the possibility that the blocks some folks might put in place to ...manage?... this attack will have a long term bad effect on our heirs and assigns. i know your perl script will expire things 60 seconds after they stop spewing, but i fear that others are blocking these in hardcode.

(looking for ". IN NS" as the q-tuple pattern is not a solution, since the bad guys can pretty trivially change the question they ask into one you're willing to answer.)

(REFUSED is nice because it's small, but the bad guys aren't lacking for bots to transmit spoofed packets from, they Do Not Need amplification, and all forms of error rate limiting are also new attack vectors.)

(is there a name-and-shame registry for networks that do/don't implement BCP38? if not i guess i'll start one and hope that various auditors will use google and notice me.) -- Paul Vixie

Paul, Arrrg. I checked and you're right about the ACL blocking potentially legitimate queries from the victim's IP address of authoritative zones served from my authoritative-only DNS servers. This is an issue if the victim is running a recursive DNS server at that IP address. However, is this really an issue if the victim is running a non- recursive DNS server or another kind of server at that IP address? In the latter case, I think not. In the former case, maybe, if it is an authoritative-only DNS server that doesn't call its own system stub resolver, or if it is a forwarding DNS server configured to query my authoritative-only servers (though goodness knows why someone would want to do that). Obviously my code did not vet the detected IP address for these cases, and I don't see any way to do so. The 1 pps query stream is not a burden to my line, nor to my servers. My goal was to make it more difficult for these perpetrators to relay this garbage off my authoritative-only servers, and thus reduce the abusive traffic being reflected towards their target(s). As you said, the "bad guys" aren't lacking for bots, and they don't need, nor seem interested in, amplification. I hadn't noticed BIND having any handles to turn for this. Maybe I'll work on that, or just rebuild my kernels to include the u32 IPtables module. As for why to block near the reflector? Given the cost/benefit you pointed out, and using my tool or other simpler ACLs, you're right, it doesn't make sense. However, using proper DPI tools (which obviously my tool was not), why shouldn't those who are able to block near the reflector? A variation on the IPtables u32 rule I saw posted elsewhere, or a handle to turn in BIND, based on signatures of known abusive query patterns would accomplish for DNS attack vector what anti-virus has been doing for decades for workstation and server systems, would it not? (Yes, yes, I know, signature-based tech is bloated and dying, and behavioral tech is the wave of the future, but right now signatures are all there is for this attack vector.) To address another critique of the posting of my tool, evolution of this form of attack is inevitable. For example, several people have suggested that the next variation will be to change the query data from asking for a root referral to asking for some other zone for which my authoritative-only servers are not authoritative. I'm not so sure, though, about the "bad guys" changing their query into something my authoritative-only servers should answer and not REFUSE. I think changing to querying for a zone my servers are authoritative for would come after simple permutations of their initial static query signature. Though it may be a recent innovation to use a botnet to carry out these attacks, the vector itself is not new, and they started with a simple static query signature. Their botnet CinC channels seem sophisticated enough to divide labor to send out spam, but are they sophisticated enough to determine which zones I host authoritative-only and then hammer out spoofed-source-address queries for only those zones? If not, then a signature-based DPI blocking tool would likely have an appreciable dampening effect. If so, then at least it would raise the bar, as has most other anti-virus and anti-spyware tech. Anyway, I think the "bad guys", if they cared at all, will already be aware of the fact that the community has identified their attack signature. This is because either they are monitoring the down-ness of their targets, or because folks on this list and others have been posting details about identified targets since last week. In some way they must be aware when targets are identified by the community, since the targets keep changing as they are found and dealt with. Finally, I agree with Mark Andrews that BCP 38 is the ultimate best response, right now, and that a cooperative community of network operators working together is the best way to track down from where this garbage is coming. At least that is when said community is small enough and clueful enough. I should wonder, though, (as have others on this list) if the operators of the network(s) where these spoofed query packets originate are complying with BCP 38? Or if the operators of the networks upstream are complying with it? Also, that is assuming that the packets are originating from sufficiently stubby networks where BCP 38 can be applied. Yep, I'm full of cheery thoughts like these. -- Douglas C. Stephens | UNIX/Windows/Email Admin System Support Specialist | Network/DNS Admin Information Systems | Phone: (515) 294-6102 Ames Laboratory, US DOE | Email: stephens@ameslab.gov

Quoting Matthew Huff <mhuff@ox.com>:

Given the recent DNS amplification attacks, I've audit and updated our authoritative servers. We are using 9.6.0-P1 now. I've been using the cyrmu templates, but one thing I see is that the dns queries to the . hint file are still occuring and are not being denied by our servers. For example:

27-Jan-2009 15:00:22.963 queries: client 64.57.246.146#64176: view external-in: query: . IN NS + 27-Jan-2009 15:00:23.118 queries: client 64.57.246.146#33146: view external-in: query: . IN NS +

the named.conf has:

... ... ...

view "external-in" in { match-clients { any; }; recursion no; additional-from-auth no; additional-from-cache no;

zone "." in { type hint; file "db.cache"; }; ... ...

since you can't put a "allow-query { none; };" in a hint zone, what can I do to deny the query to the . zone file?

---- Matthew Huff       | One Manhattanville Rd OTA Management LLC | Purchase, NY 10577 http://www.ox.com | Phone: 914-460-4039 aim: matthewbhuff  | Fax:   914-460-4139

Hi Matthew, I'm using the following with 9.5.1: view "external" { recursion no; allow-query-cache { none; }; zone "." IN { type hint; file "/var/named/named.ca"; And my logs indicate that the requests for . IN NS are being denied: Jan 28 08:40:38 web1 named[12337]: client 64.57.246.146#33453: view external: query (cache) './NS/IN' denied Jan 28 08:40:39 web1 named[12337]: client 67.192.144.0#41794: view external: query (cache) './NS/IN' denied Cheers, Jay

chuck goolsbee wrote:

On Tue, 27 Jan 2009 13:54:10 -0500, wingying wrote:

...

A quick question, what is the common bandwidth for out-of-band access? Thanks.

Clearwire + POTS as a backup.

POTS + CDMA cellular for me. There's a lot of ways to do it. It really depends on what you want to do, what's available, what's reliable, and what you can afford. Like your primary circuits, if you need reliable OOB access pick more than one method just in case. ~Seth

Leigh Porter wrote:

And be careful.. It's easy to have simple local passwords, a dial-in modem and then get pwned.. Were I used to be, we had encrypted modems for POP dialin.

I have my POTS modem set up to accept PPP too so there's the option of being a little more secure than a plain text terminal, assuming the world didn't completely blow up. ~Seth