maybe a dumb idea on how to fix the dns problems i don't know....
Paul, Sorry if this is real stupid for some reason because I don't think about DNS all day (I'm the ldap dude) but since we have faster networks and faster cpus today, what would be the harm in switching to use TCP for DNS clients? The latency on the web isn't dns anymore ever it seems to me..... Wouldn't that eliminate the ability to poison clients' caches? any why wouldn't persistent client-server dns connections make sense? any stupid little bsd or linux box can handle several thousand connections today pretty easily if tuned correctly by some reasonably competent primate CP -- Chris Paul Rex Consulting, Inc 157 Rainbow Drive #5703, Livingston, TX 77399-1057 email: chris.paul@rexconsulting.net * web: http://www.rexconsulting.net* phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
TCP would work, but it makes it more difficult to do Anycast, which works well with UDP and DNS. Chuck -----Original Message----- From: Chris Paul [mailto:chris.paul@rexconsulting.net] Sent: Saturday, August 09, 2008 5:18 PM To: nanog@merit.edu Subject: maybe a dumb idea on how to fix the dns problems i don't know.... Paul, Sorry if this is real stupid for some reason because I don't think about DNS all day (I'm the ldap dude) but since we have faster networks and faster cpus today, what would be the harm in switching to use TCP for DNS clients? The latency on the web isn't dns anymore ever it seems to me..... Wouldn't that eliminate the ability to poison clients' caches? any why wouldn't persistent client-server dns connections make sense? any stupid little bsd or linux box can handle several thousand connections today pretty easily if tuned correctly by some reasonably competent primate CP -- Chris Paul Rex Consulting, Inc 157 Rainbow Drive #5703, Livingston, TX 77399-1057 email: chris.paul@rexconsulting.net * web: http://www.rexconsulting.net* phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
On 9 Aug 2008, at 17:22, Church, Charles wrote:
TCP would work, but it makes it more difficult to do Anycast, which works well with UDP and DNS.
TCP works pretty well with anycast too, if you're careful. It's helpful if your transactions are short-lived. I've seen concern expressed that a server which can handle 100,000 qps over UDP might well fare substantially more poorly if every query arrives using TCP transport. The business of large-scale HTTP is a fairly well-understood problem, however, and has some similar characteristics, so perhaps this is less of a long-term problem. I don't know, I haven't run any experiments to figure out the practical impact on performance of using TCP exclusively. There is, however, the practical consideration that a generation of firewall "administrators" seem to believe that 53/tcp is only ever used for zone transfers, and can safely be closed for all other use. I suspect that a route with better practical implications will be for resolvers to pad queries with additional entropy as EDNS0 options, and to fall back to TCP if EDNS0 is unsupported. That requires some confidence that EDNS0 support in authority servers is widespread, however. draft-vixie-dnsext-dns0x20 describes a shorter-term option for introducing additional entropy into queries using UDP transport, with or without EDNS0. Joe
Why not just require TCP for a lookup if a response with an incorrect TXID is received? You could require TCP for just the one lookup or for some configured interval, say 1 hour. That should slow attackers down substantially. Joe Abley wrote:
On 9 Aug 2008, at 17:22, Church, Charles wrote:
TCP would work, but it makes it more difficult to do Anycast, which works well with UDP and DNS.
TCP works pretty well with anycast too, if you're careful. It's helpful if your transactions are short-lived.
I've seen concern expressed that a server which can handle 100,000 qps over UDP might well fare substantially more poorly if every query arrives using TCP transport. The business of large-scale HTTP is a fairly well-understood problem, however, and has some similar characteristics, so perhaps this is less of a long-term problem. I don't know, I haven't run any experiments to figure out the practical impact on performance of using TCP exclusively.
There is, however, the practical consideration that a generation of firewall "administrators" seem to believe that 53/tcp is only ever used for zone transfers, and can safely be closed for all other use.
I suspect that a route with better practical implications will be for resolvers to pad queries with additional entropy as EDNS0 options, and to fall back to TCP if EDNS0 is unsupported. That requires some confidence that EDNS0 support in authority servers is widespread, however.
draft-vixie-dnsext-dns0x20 describes a shorter-term option for introducing additional entropy into queries using UDP transport, with or without EDNS0.
Joe
On 9 Aug 2008, at 18:10, Matt F wrote:
Why not just require TCP for a lookup if a response with an incorrect TXID is received? You could require TCP for just the one lookup or for some configured interval, say 1 hour. That should slow attackers down substantially.
That sounds like a good way for a remote attacker to make a resolver disable UDP transport for a server, more or less at will. I'm not sure I like the sound of that. Joe
matt@credibleinstitution.org (Matt F) writes:
Why not just require TCP for a lookup if a response with an incorrect TXID is received? You could require TCP for just the one lookup or for some configured interval, say 1 hour. That should slow attackers down substantially.
because TCP is considered optional by many authority DNS server operators. it's only required if you expect AXFR or if you ever emit a TC bit. if you don't want to do TCP then you can rule out the TC bit and AXFR and just not do TCP, and you'll be dead-to-rights within the various DNS protocol RFCs. anyone who insists on reaching such a server by TCP will be shit-outta-luck. however, this suggestion and dozens of others are being workshopped all day every day by actual DNS experts. you may not know about those discussions because they are not occurring on nanog@, where they would be off-topic, like this thread here. please join namedroppers@ops.ietf.org and perhaps dns-operations@lists.oarci.net if you want to discuss DNS protocol matters. please, please, please don't open this can of, um, worms on nanog@ again. not even on a sunday afternoon when just about anything goes. -- Paul Vixie -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Randy Bush wrote:
Paul Vixie wrote:
hey are not occurring on nanog@, where they would be off-topic, like this thread here
you may want to read the aup. by my read they are not off topic.
Also: given how serious the problem is, I'd think that far and wide perspective on this is appropriate. Knowing IETF, there is usually not nearly enough operations perspective than you'd like, and the various working group lists can be pretty daunting for somebody who's day job is to "merely" keep the net running. Mike, delurking
Paul Vixie wrote:
because TCP is considered optional by many authority DNS server operators.
Hey authority DNS server operators. Can you make a change to your servers to always allow TCP client connections? Would this be difficult? What would be the harm?
it's only required if you expect AXFR or if you ever emit a TC bit. if you don't want to do TCP then you can rule out the TC bit and AXFR and just not do TCP, and you'll be dead-to-rights within the various DNS protocol RFCs.
what RFCs forbid TCP for clients? I thought TCP was an option for clients. I'm not spending the rest of my sunday though reading rfcs....... and sure as hell not joining another list because to tell you the truth, I don't really care as much about the typical angry Sunday list poster (talk about redundant statement....) thanks for the thoughts, though Paul. I'll leave the rest of this discussion (should it exist) to others in their forum of choice.... I'm thinking of nice insalade caprese with true mozarella di bufalo right now.... now That's A Sunday!" CP -- Chris Paul Rex Consulting, Inc 157 Rainbow Drive #5703, Livingston, TX 77399-1057 email: chris.paul@rexconsulting.net web: http://www.rexconsulting.net phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
On Aug 9, 2008, at 3:48 PM, Chris Paul wrote:
Paul Vixie wrote:
because TCP is considered optional by many authority DNS server operators.
Hey authority DNS server operators. Can you make a change to your servers to always allow TCP client connections? Would this be difficult? What would be the harm?
SYN flooding?
Paul Vixie wrote:
because TCP is considered optional by many authority DNS server operators.
On Aug 9, 2008, at 3:48 PM, Chris Paul wrote:
Hey authority DNS server operators. Can you make a change to your servers to always allow TCP client connections? Would this be difficult? What would be the harm?
brett@the-watsons.org (brett watson) writes:
SYN flooding?
SYN flooding is a specific instance of "have to hold too much state" whereas the reason for not considering TCP mandatory is the general form of "have to hold too much state". also note, the operators of those nameservers aren't reading nanog@, or indeed any other mailing list where they could all be reached. the installed base is, as usual, an impediment to righteous change. -- Paul Vixie -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On 10 Aug 2008, at 01:45, Paul Vixie wrote:
SYN flooding is a specific instance of "have to hold too much state" whereas the reason for not considering TCP mandatory is the general form of "have to hold too much state".
It may be worth clarifying that "not considering TCP mandatory" above is an implementation/operational choice, and not something that seems to be clearly endorsed by RFC 1035, such as it is. There are a lot of people who insist that TCP transport is used for nothing other than zone transfers in the DNS, and they do so not out of concern over potential TCP state explosion on their servers but instead because "that's what the last guy told me". That kind of reasoning doesn't need a bigger posse. Joe 4.2. Transport The DNS assumes that messages will be transmitted as datagrams or in a byte stream carried by a virtual circuit. While virtual circuits can be used for any DNS activity, datagrams are preferred for queries due to their lower overhead and better performance. Zone refresh activities must use virtual circuits because of the need for reliable transfer. The Internet supports name server access using TCP [RFC-793] on server port 53 (decimal) as well as datagram access using UDP [RFC-768] on UDP port 53 (decimal).
(here we are discussing dns protocol details on nanog@ again. must be sunday.)
From: Joe Abley <jabley@ca.afilias.info>
It may be worth clarifying that "not considering TCP mandatory" above is an implementation/operational choice, and not something that seems to be clearly endorsed by RFC 1035, such as it is.
There are a lot of people who insist that TCP transport is used for nothing other than zone transfers in the DNS, and they do so not out of concern over potential TCP state explosion on their servers but instead because "that's what the last guy told me". That kind of reasoning doesn't need a bigger posse.
Joe
4.2. Transport ...
actually, it does (need a bigger posse). a little further on in RFC 1035 we find this gem: +--- | 4.2.2. TCP usage | ... | Several connection management policies are recommended: | | - The server should not block other activities waiting for TCP | data. | | - The server should support multiple connections. | | - The server should assume that the client will initiate | connection closing, and should delay closing its end of the | connection until all outstanding client requests have been | satisfied. | | - If the server needs to close a dormant connection to reclaim | resources, it should wait until the connection has been idle | for a period on the order of two minutes. In particular, the | server should allow the SOA and AXFR request sequence (which | begins a refresh operation) to be made on a single connection. | Since the server would be unable to answer queries anyway, a | unilateral close or reset may be used instead of a graceful | close. +--- in the era of RFC 1035 the philosophy was "be liberal in what you accept and be conservative in what you generate" because the other people on the network weren't trying to spam, ddos, or poison you. the above text is effectively a "please ddos me" bumper sticker worn across the ass of anyone who implements it. we're having a terrible time with UDP now simply because upstream sockets can now only be used once before they're closed, and any long-running query can tie up a file descriptor for a longish enough time to drain the pool down to the point where new downstream queries can't be accepted because existing upstream queries have not completed. file descriptors are the new "carbon footprint" of DNS. but at least in the UDP case the shortage is experienced only by the initiator. in the TCP case the shortage is experienced by both the initiator and the responder, and the responder is shackled by [4.2.2]. i don't want to completely dismiss the underlying idea of "fall back to TCP if someone guesses a few QID's wrong". however, i dismiss the idea that it's a simple universal solution. on a mailing list called namedroppers@ where i would more likely expect to see a discussion of fine points of DNS protocol, i spake thusly about a week ago, and so far have heard no reply, though i've implemented these recommendations on a server that only keeps 64 descriptors open at a time and it's been incredibly resistant to my poisoning attempts. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On 10 Aug 2008, at 11:56, Paul Vixie wrote:
(here we are discussing dns protocol details on nanog@ again. must be sunday.)
(Or alternatively we could just be discussing DNS operations, something that is entirely on-topic for this list, and conceivably of interest to the many hundreds of people who are subscribed here but not to other dns-specific lists. That was certainly my intent, even if it wasn't yours.)
From: Joe Abley <jabley@ca.afilias.info>
It may be worth clarifying that "not considering TCP mandatory" above is an implementation/operational choice, and not something that seems to be clearly endorsed by RFC 1035, such as it is.
There are a lot of people who insist that TCP transport is used for nothing other than zone transfers in the DNS, and they do so not out of concern over potential TCP state explosion on their servers but instead because "that's what the last guy told me". That kind of reasoning doesn't need a bigger posse.
Joe
4.2. Transport ...
actually, it does (need a bigger posse).
Rhetoric aside, no it doesn't. Choosing not to implement (or permit, as an operational decision) TCP because of concerns about state is what you go on to talk about; what you were actually replying to was the wholesale denial of 53/tcp out of simple ignorance, which I would be surprised to hear you endorse, even if it happens to coincide on this instance with the results of your analysis. Joe
actually, it does (need a bigger posse).
Rhetoric aside, no it doesn't.
Choosing not to implement (or permit, as an operational decision) TCP because of concerns about state is what you go on to talk about; what you were actually replying to was the wholesale denial of 53/tcp out of simple ignorance, which I would be surprised to hear you endorse, even if it happens to coincide on this instance with the results of your analysis.
not doing tcp/53 because the last guy didn't do it is the first step toward not doing tcp/53 because it's amazingly fragile. sorry to cross the streams without a diagram. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
brett watson wrote:
Hey authority DNS server operators. Can you make a change to your servers to always allow TCP client connections? Would this be difficult? What would be the harm?
SYN flooding?
from your clients? We ways of knowing people on our local network are doing this type of thing and turn them off at the switch today. Why are you are doing dns recursion for people outside your network? CP -- Chris Paul Rex Consulting, Inc 157 Rainbow Drive #5703, Livingston, TX 77399-1057 email: chris.paul@rexconsulting.net web: http://www.rexconsulting.net phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
On Sun, Aug 10, 2008 at 01:06:06PM -0700, Chris Paul wrote:
brett watson wrote:
Hey authority DNS server operators. Can you make a change to your servers to always allow TCP client connections? Would this be difficult? What would be the harm?
SYN flooding?
from your clients? We ways of knowing people on our local network are doing this type of thing and turn them off at the switch today. Why are you are doing dns recursion for people outside your network?
The question isn't whether to offer TCP/53 up at the recursive server. The issue is that for you to use TCP/53 from your recursive server, it has to be offered up at the authoritative end. The authoritative server operators have to offer TCP/53 and the firewall administrators between the recursive server and the authoritative servers have to allow the traffic. -rob
The question isn't whether to offer TCP/53 up at the recursive server. The issue is that for you to use TCP/53 from your recursive server, it has to be offered up at the authoritative end.
The authoritative server operators have to offer TCP/53 and the firewall administrators between the recursive server and the authoritative servers have to allow the traffic.
-rob
Yes. This is true. But with a caching resolver being used for most interactive clients (web surfers), this doesn't cause any problem, other than the initial caching. OK I guess the question is this: How many milliseconds now on average does it take for my local dns server to obtain an address which is uncached using recursion up to the authoritative end using UDP And I guess the second question is: How many milliseconds on average would it take for my local dns server to obtain an address which is uncached using recursion up to the authoritative end using TCP. Once it is cached on my local caching server, its a non-issue if I am using some sort of persistent connection to my (non-authoritative) dns caching server. CP -- Chris Paul Rex Consulting, Inc 157 Rainbow Drive #5703, Livingston, TX 77399-1057 email: chris.paul@rexconsulting.net web: http://www.rexconsulting.net phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
On Sat, Aug 9, 2008 at 5:18 PM, Chris Paul <chris.paul@rexconsulting.net> wrote:
Sorry if this is real stupid for some reason because I don't think about DNS all day (I'm the ldap dude) but since we have faster networks and faster cpus today, what would be the harm in switching to use TCP for DNS clients? The latency on the web isn't dns anymore ever it seems to me.....
Latency on in-addr lookups where you typically traverse multiple forward trees to find the NS servers would seriously suck. At best, a TCP-based lookup performs at about a third of the speed of a UDP lookup. Worse unless your implementation is carefully optimized and you make sure that the OS isn't adding options to the front of the handshake. You have at least the whole syn/synack/ack handshake before you can even ask the question. Then there's the server cost associated with keeping that much state... On Sat, Aug 9, 2008 at 6:10 PM, Matt F <matt@credibleinstitution.org> wrote:
Why not just require TCP for a lookup if a response with an incorrect TXID is received? You could require TCP for just the one lookup or for some configured interval, say 1 hour. That should slow attackers down substantially.
Because the attacker is using a sequence of lookups in order to hit one that lets him poison the cache. That is, he looks up a.google.com, then he looks up b.google.com, then c.google.com, etc. until he gets one where the server accepts his fake DNS server record for google.com. To be an effective defense, you'd have to do TCP lookups for the whole scope ({anything}.google.com) for some period of time following the bad ID. That in turn would open up a potential DOS where an attacker could force the DNS server to fall back on TCP for essentially everything, overwhelming it. On Sat, Aug 9, 2008 at 8:25 PM, brett watson <brett@the-watsons.org> wrote:
On Aug 9, 2008, at 3:48 PM, Chris Paul wrote:
Hey authority DNS server operators. Can you make a change to your servers to always allow TCP client connections? Would this be difficult? What would be the harm?
SYN flooding?
SYN flooding is a solved problem. On Sat, Aug 9, 2008 at 6:04 PM, Joe Abley <jabley@ca.afilias.info> wrote:
TCP works pretty well with anycast too, if you're careful. It's helpful if your transactions are short-lived.
Define "careful." It's always possible for someone to find themselves with an equal cost path to two different servers in the anycast set. Add per-packet load balancing at the fork (which is outside the control of the server operator) and what happens is the request times out and the resovler fails over to the other NS record that isn't anycasted. Though the protocol is simple enough that it might be possible to fake it. Build yourself a DNS-only stateless TCP stack for the anycasted address. Have the server send the syn-ack without creating any state. The request will almost certainly be entirely contained in one packet, so when it arrives reply to it without creating any state. Ship off as many packets as you need to reply followed by a Fin. Blindly ack any packet that looks like it needs it. If any packets are lost, there won't be any retransmit (you haven't really established a TCP connection) so the query will time out and retry. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
participants (10)
-
brett watson -
Chris Paul -
Church, Charles -
Joe Abley -
Matt F -
Michael Thomas -
Paul Vixie -
Randy Bush -
Rob Payne -
William Herrin