Re: DNS deluge for x.p.ctrc.cc
Hi, NANOGers. ] other cctld servers have seen what are effectively ddos. rob thomas ] seems to have the most clue on this, so i hope this troll will entice ] him to speak. Did someone say "troll?" :) Yes, this is a real problem. These attacks have exceeded several gigabits per second in size, and during one attack 122K DNS name servers were abused as amplifiers. Ouch! This abuse can be mitigated. Here are a few tips. Limit recursion to trusted netblocks and customers. Do not permit your name servers to provide recursion for the world. If you do, you will contribute to one of these attacks. Watch for queries to your name servers that ask for "ANY" related to a DNS RR outside of the zones for which you are authoritative. This DNS RR will be LARGE. Limit UDP queries to 512 bytes. This greatly decreases the amplification affect, though it doesn't stop it. Scan your IP space for name servers that permit recursive queries. It's amazing just how many of these name servers exist. Refer to the following guides for some excellent insight and suggestions. <http://www.us-cert.gov/reading_room/DNS-recursion121605.pdf> <http://cc.uoregon.edu/cnews/winter2006/recursive.htm> <http://dns.measurement-factory.com/surveys/sum1.html> Note we have our own Secure BIND Template which will help on the BIND side of life. <http://www.cymru.com/Documents/secure-bind-template.html> If you need assistance with any of this, have endured one of these attacks, or have any other questions, please don't hesitate to ping on us at team-cymru@cymru.com. We're here to assist! Thanks! Rob. -- Rob Thomas Team Cymru http://www.cymru.com/ ASSERT(coffee != empty);
Note we have our own Secure BIND Template which will help on the BIND side of life.
<http://www.cymru.com/Documents/secure-bind-template.html>
If you need assistance with any of this, have endured one of these attacks, or have any other questions, please don't hesitate to ping on us at team-cymru@cymru.com. We're here to assist!
... and if you have used the Secure BIND Template in the past but perhaps haven't perused it lately, please take this opportunity to examine the change log helpfully supplied at the top of the page. In particular, some prefixes have been removed from the Secure BIND Template's embedded bogon list; if you don't keep that information current on authoritative nameservers, you'll ignore requests from caching nameservers in FORMER bogon-list space that is now VALID space. Remember - bogon lists aren't just for routers, and must be kept up to date. Stephen
Once upon a time, Rob Thomas <robt@cymru.com> said:
Limit recursion to trusted netblocks and customers. Do not permit your name servers to provide recursion for the world. If you do, you will contribute to one of these attacks.
One thing to note: we've discovered that on some common DSL routers, the internal DNS caching server is on by default and answers requests on the outside IP address. IIRC some even do it when configured for NAT. So, even when you disable outside recursion, things you may not think of on the inside of your network may still allow outside DNS recursion. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On Fri, 24 Feb 2006, Chris Adams wrote:
One thing to note: we've discovered that on some common DSL routers, the internal DNS caching server is on by default and answers requests on the outside IP address. IIRC some even do it when configured for NAT.
So, even when you disable outside recursion, things you may not think of on the inside of your network may still allow outside DNS recursion.
Efficient Networks DSL routers suffer from this problem if DNS servers are defined in the DHCP server config on the router. It's more of a DNS proxy though. It doesn't do any caching. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
] other cctld servers have seen what are effectively ddos. rob thomas ] seems to have the most clue on this, so i hope this troll will entice ] him to speak.
Did someone say "troll?" :)
Yes, this is a real problem. These attacks have exceeded several gigabits per second in size, and during one attack 122K DNS name servers were abused as amplifiers. Ouch!
This abuse can be mitigated. Here are a few tips.
<there has -GOT- to be a better name for this>
Limit recursion to trusted netblocks and customers. Do not permit your name servers to provide recursion for the world. If you do, you will contribute to one of these attacks.
<recursion is a fundamental DNS design feature, restricting it to "walled gardens" cripples its usefullness>
Watch for queries to your name servers that ask for "ANY" related to a DNS RR outside of the zones for which you are authoritative. This DNS RR will be LARGE.
<a valid concern, w/ the following caveat: LARGE, relative to current traffic>
Limit UDP queries to 512 bytes. This greatly decreases the amplification affect, though it doesn't stop it.
<limiting UDP to 512 has other, unwanted effects, edns0 for one... crippling ENUM, DNSSEC, IPv6, etc... is this really what is wanted?>
Scan your IP space for name servers that permit recursive queries. It's amazing just how many of these name servers exist.
<yup... again, a feature that has made the DNS as useful as it has become>
Refer to the following guides for some excellent insight and suggestions.
<http://www.us-cert.gov/reading_room/DNS-recursion121605.pdf> <http://cc.uoregon.edu/cnews/winter2006/recursive.htm> <http://dns.measurement-factory.com/surveys/sum1.html>
Note we have our own Secure BIND Template which will help on the BIND side of life.
<http://www.cymru.com/Documents/secure-bind-template.html>
If you need assistance with any of this, have endured one of these attacks, or have any other questions, please don't hesitate to ping on us at team-cymru@cymru.com. We're here to assist!
Thanks! Rob. -- Rob Thomas Team Cymru http://www.cymru.com/ ASSERT(coffee != empty);
ok, so i'm being a bit of a curmudgion here but just how, if we throttle DNS to the minimum suite for todays services, can we be expected to add new features/services? grump grump grump... -- (grumpy) bill
bmanning@vacation.karoshi.com wrote:
Limit recursion to trusted netblocks and customers. Do not permit your name servers to provide recursion for the world. If you do, you will contribute to one of these attacks.
<recursion is a fundamental DNS design feature, restricting it to "walled gardens" cripples its usefullness>
I don't really think that preventing every Tom, Dick, and Harry from using my nameserver to look up domains I'm not authoritative for is going to cripple DNS. They really should have their own severs that do that for them, or they should use the ones provided to them by their ISP.
Hey, Bill. As many say, you own your network, and are free to run it as you see fit. :) That said, please be aware that if you leave your name servers open to recursive query requests from any source, you WILL unwittingly help to amplify these attacks. It's the same as ICMP directed broadcast and the like. I don't like it. You don't like it. The miscreants love it. It's always a balancing act. Thanks, Rob. -- Rob Thomas Team Cymru http://www.cymru.com/ ASSERT(coffee != empty);
As many say, you own your network, and are free to run it as you see fit. :) That said, please be aware that if you leave your name servers open to recursive query requests from any source, you WILL unwittingly help to amplify these attacks.
and there is discussion on treating this similarly to how open smtp relays are treated today. randy
I've taken the liberty of following up on this thread on a different mailing list (dns-operations@lists.oarci.net), since I'd like to explore it at a depth and breadth that would seem overly obsessive on a general purpose ops list like nanog. <http://lists.oarci.net/mailman/listinfo/> is the entry point if you aren't subscribed and want to be. (There are about 50 folks on that list, which I'm calling "critical mass" for the purpose of starting the first real discussion over there.) -- Paul Vixie
vixie@vix.com (Paul Vixie) (hey, that's me!) writes:
<http://lists.oarci.net/mailman/listinfo/> ... (There are about 50 folks on that list, which I'm calling "critical mass" for the purpose of starting the first real discussion over there.)
oops. 154 as of this morning, i guess i wasn't paying attention. note that the list is open, and the subscriber list is open to those who are subscribed. this isn't a vetted insider-only kind of mailing list like OARC would normally run, it's a public service side-effect kind of thing. -- Paul Vixie
On Sat, 25 Feb 2006, Rob Thomas wrote:
As many say, you own your network, and are free to run it as you see fit. :) That said, please be aware that if you leave your name servers open to recursive query requests from any source, you WILL unwittingly help to amplify these attacks. It's the same as ICMP directed broadcast and the like.
This has been an issue for years. Before the DDoSers started using open recursive DNS servers as a modern way to "smurf", spammers were abusing them by registering a domain, setting up DNS, loading the data into open recursive servers (by sending them queries), and then pointing the domains at those recursive servers...getting free DNS service and misdirecting complaints. The argument that DNS servers have always been open to recursion (so we shouldn't change it) sounds a lot like the open SMTP relay issue 5-10 years ago. It took years, but all but a few wingnuts seem to have finally caught on to the idea that open SMTP relays are a bad idea...enough so that spammers had to move on and adapt to open proxies, and then to botted systems / trojan proxies. Besides, don't the DNS specs dictate that a proper DNS resolver will try again with TCP if the server tells it the UDP reply was truncated? ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Sat, Feb 25, 2006 at 08:41:01AM +0000, bmanning@vacation.karoshi.com wrote:
robt wrote: [snip]
Limit recursion to trusted netblocks and customers. Do not permit your name servers to provide recursion for the world. If you do, you will contribute to one of these attacks.
<recursion is a fundamental DNS design feature, restricting it to "walled gardens" cripples its usefullness>
The bad guys abused open SMTP relaying and we couldn't use it anymore.* They've moved to the next thing that is widely open and will be abusable for a long time while some folks clamp down quickly, others argue against it, etc. Until we can factor out the bad guys, the diminishing returns on playing whack-a-mole will force us all to install more functional equivalent of signs saying "restrooms are for customers only". And no I don't like it either. Cheers, Joe * well, except those who wish to be marginalized. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On 25-Feb-2006, at 03:41, bmanning@vacation.karoshi.com wrote:
Limit UDP queries to 512 bytes. This greatly decreases the amplification affect, though it doesn't stop it.
<limiting UDP to 512 has other, unwanted effects, edns0 for one... crippling ENUM, DNSSEC, IPv6, etc... is this really what is wanted?>
Expanding on this slightly, since I think this merits more discussion -- if there was widespread filtering of 53/udp packets to 512 bytes, I can see two consequences: 1. A temporary decrease in attack traffic: the malware authors will adapt, and the floods will continue except with smaller packets. For attacks launched from drones, the amplification arguably isn't as important to the attacker as it might be for attacks which have single sources. 2. In future, increased use of things like DNSSEC, dynamic updates and IPv6 (with attendant AAAA records) are going to make legitimate, EDNS0, large 53/udp packets more common, and crippling the transport for EDNS0 is going to cause migraines for helpdesks across the world. As a temporary mitigation tool today, when the volume of legitimate, large-packet EDNS0 traffic is near-zero, blocking big 53/udp packets might *sound* reasonable. However, we all know how permanent temporary filters can be. Crippling EDNS0 transport in the future seems like a very high price to pay for what might be a very temporary, short-term reduction in attack traffic. Joe
On Sun, 26 Feb 2006, Joe Abley wrote:
As a temporary mitigation tool today, when the volume of legitimate, large-packet EDNS0 traffic is near-zero, blocking big 53/udp packets might *sound* reasonable. However, we all know how permanent
how are you certain that the udp/53 1500 byte packet is 'dns'? and not kazaa/gnutella/bittorrent/vpn-in-udp-53 ? It seems that filtering the TRAFFIC is short sighted on several fronts :( deciding if you will/won't be part of the global-recursive-dns-server 'problem' is entirely different though.
temporary filters can be. Crippling EDNS0 transport in the future seems like a very high price to pay for what might be a very temporary, short-term reduction in attack traffic.
seems like global tcp/139|tcp/445 filters, or bogon filters... bits put into configs 'now' and completely forgotten about 'tomorrow' :(
christopher.morrow@verizonbusiness.com ("Christopher L. Morrow") writes:
seems like global tcp/139|tcp/445 filters, or bogon filters... bits put into configs 'now' and completely forgotten about 'tomorrow' :(
speaking of which, f-root has about 35 nodes world wide, and about a third to a half of them aren't reachable by udp/161, and the blockage is not in our immediate neighbors but rather on transit paths. this is due to the cisco snmp vulnerability five years or so ago. filtering in the core to protect vulnerable edges has to be done a LOT more carefully than that. (BCP38 is an example of how to do it well, but apparently impractically?) i'm not following up on the dns related parts of this, since dns-operations@ seems to be pulling some of the dns related load today and i don't want to say the same thing in both places. see this URL for details: http://lists.oarci.net/pipermail/dns-operations/2006-February/author.html -- Paul Vixie
i'm not following up on the dns related parts of this, since dns-operations@ seems to be pulling some of the dns related load today and i don't want to say the same thing in both places. see this URL for details:
http://lists.oarci.net/pipermail/dns-operations/2006-February/author.html -- Paul Vixie
hum... i subscribed to this dns-operations@ list some days back and have yet to see any postings. i guess i'm not worthy. --bill
# hum... i subscribed to this dns-operations@ list some days back as what? #in2.oarc:amd64# bin/list_members dns-operations | grep -i manning #in2.oarc:amd64# bin/list_members dns-operations | grep -i ep.net #in2.oarc:amd64# # and have yet to see any postings. i guess i'm not worthy. i apologize to the gallery for cc'ing nanog, but bill did (by mistake, one hopes) and i want to make it clear that bill is completely worthy, and the list is completely open, and if a subscription doesn't work, you don't have to send mail to nanog to get it fixed, admin@oarc.isc.org can help fix things. and fyi: #in2.oarc:amd64# bin/list_members dns-operations | wc -l 190
In message <Pine.GSO.4.62.0602241629470.21514@qentba.nf23028.arg>, Rob Thomas w rites:
Limit UDP queries to 512 bytes. This greatly decreases the amplification affect, though it doesn't stop it.
Unfortunately, the intention of the DNS developers is just the opposite. Things like DNSSEC require larger packet sizes; in fact, there's a DNS extension (EDNS0) whose purpose, among others, it to permit this. --Steven M. Bellovin, http://www.cs.columbia.edu/~smb
participants (13)
-
bmanning＠vacation.karoshi.com -
Chris Adams -
Christopher L. Morrow -
Joe Abley -
Joe Provo -
Jon Lewis -
Nicholas Suan -
Paul Vixie -
Paul Vixie -
Randy Bush -
Rob Thomas -
Stephen Stuart -
Steven M. Bellovin