Re: maybe a dumb idea on how to fix the dns problems i don't know....
list-nanog@pwns.ms wrote:
(I know; you old folks that created this wonderful thing didn't think of that back then.... blah blah blah).
If they had thought of it back then, they would have allowed for a larger TXID, not used TCP. TCP connection setup time is slow; TCP DNS is much slower than UDP DNS. Good point.
But we only care about TCP connection setup time in *interactive* sessions (a human using something like the web). If you have a persistent connection to your dns server from your dns resolver on your browser machine, you just send the request.... no TCP setup there at all. You can even pool connections. We do this stuff in LDAP all the time. How does TCP resolution work in most resolver libraries? A TCP connection for each lookup? That is kind of dumb isn't it, speaking of dumb.... I actually don't know. Not much of a coder, so I'll let you coders check your code and get back to me on that... well.. maybe i'll fire up snort or wireshark and check it out later with some different dns libs.... CP -- Chris Paul Rex Consulting, Inc 157 Rainbow Drive #5703, Livingston, TX 77399-1057 email: chris.paul@rexconsulting.net web: http://www.rexconsulting.net phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
But we only care about TCP connection setup time in *interactive* sessions (a human using something like the web). If you have a persistent connection to your dns server from your dns resolver on your browser machine, you just send the request.... no TCP setup there at all. You can even pool connections. We do this stuff in LDAP all the time.
Again, if we can change the DNS protocol, then it's easy to solve. Securing host->recursive name server is, at the moment, not an issue - each host is a small target, and often has little bandwidth available. Furthermore, stopping IP spoofing of one's own hosts within one's networks is, well, not trivial, but not hugely difficult either.
list-nanog@pwns.ms wrote:
But we only care about TCP connection setup time in *interactive* sessions (a human using something like the web). If you have a persistent connection to your dns server from your dns resolver on your browser machine, you just send the request.... no TCP setup there at all. You can even pool connections. We do this stuff in LDAP all the time.
Again, if we can change the DNS protocol, then it's easy to solve.
Wait a min... This isn't changing the DNS protocol to have or allow persistent dns/tcp connections is it? or is it? Yikes, yes the DNS protocol should be changed then, right now, to allow persistent connections, at least from the interactive client (websurfing machine) resolver to caching name server. CP -- Chris Paul Rex Consulting, Inc 157 Rainbow Drive #5703, Livingston, TX 77399-1057 email: chris.paul@rexconsulting.net web: http://www.rexconsulting.net phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
But we only care about TCP connection setup time in *interactive* sessions (a human using something like the web). If you have a persistent connection to your dns server from your dns resolver on your browser machine, you just send the request.... no TCP setup there at all. You can even pool connections. We do this stuff in LDAP all the time.
How does TCP resolution work in most resolver libraries? A TCP connection for each lookup? That is kind of dumb isn't it, speaking of dumb.... I actually don't know. Not much of a coder, so I'll let you coders check your code and get back to me on that...
well.. maybe i'll fire up snort or wireshark and check it out later with some different dns libs....
Pretending for a moment that it was even possible to make such large scale changes and get them pushed into a large enough number of clients to matter, you're talking about meltdown at the recurser level, because it isn't just one connection per _computer_, but one connection per _resolver stub_ per _computer_ (which, on a UNIX machine, would tend to gravitate towards one connection per process), and this just turns into an insane number of sockets you have to manage. For your average ISP recurser where they only have 50,000 people online at any given time, this could still be half a million open sockets. We already know this sort of thing doesn't scale well. This is very broken in any number of other ways. This message is not intended to imply otherwise. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Joe Greco wrote:
But we only care about TCP connection setup time in *interactive* sessions (a human using something like the web). If you have a persistent connection to your dns server from your dns resolver on your browser machine, you just send the request.... no TCP setup there at all. You can even pool connections. We do this stuff in LDAP all the time.
How does TCP resolution work in most resolver libraries? A TCP connection for each lookup? That is kind of dumb isn't it, speaking of dumb.... I actually don't know. Not much of a coder, so I'll let you coders check your code and get back to me on that...
well.. maybe i'll fire up snort or wireshark and check it out later with some different dns libs....
Pretending for a moment that it was even possible to make such large scale changes and get them pushed into a large enough number of clients to matter, you're talking about meltdown at the recurser level, because it isn't just one connection per _computer_, but one connection per _resolver stub_ per _computer_ (which, on a UNIX machine, would tend to gravitate towards one connection per process), and this just turns into an insane number of sockets you have to manage.
Couldn't the resolver libraries be changed to not use multiple connections? CP -- Chris Paul Rex Consulting, Inc 157 Rainbow Drive #5703, Livingston, TX 77399-1057 email: chris.paul@rexconsulting.net web: http://www.rexconsulting.net phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
Inline.. Chris Paul wrote:
Joe Greco wrote:
But we only care about TCP connection setup time in *interactive* sessions (a human using something like the web). If you have a persistent connection to your dns server from your dns resolver on your browser machine, you just send the request.... no TCP setup there at all. You can even pool connections. We do this stuff in LDAP all the time.
How does TCP resolution work in most resolver libraries? A TCP connection for each lookup? That is kind of dumb isn't it, speaking of dumb.... I actually don't know. Not much of a coder, so I'll let you coders check your code and get back to me on that...
well.. maybe i'll fire up snort or wireshark and check it out later with some different dns libs....
Pretending for a moment that it was even possible to make such large scale changes and get them pushed into a large enough number of clients to matter, you're talking about meltdown at the recurser level, because it isn't just one connection per _computer_, but one connection per _resolver stub_ per _computer_ (which, on a UNIX machine, would tend to gravitate towards one connection per process), and this just turns into an insane number of sockets you have to manage.
Couldn't the resolver libraries be changed to not use multiple connections?
And we'll change to IPv6 tomorrow!
CP
-- Victor Jerlin, CTO Gränslösa System GSYS HB Cell#: +356-9999-0125
Victor Jerlin wrote:
Couldn't the resolver libraries be changed to not use multiple connections?
And we'll change to IPv6 tomorrow! Total apples and oranges. We all have to patch anyhow. This is just code and firewall rules. IPv6 is way more complicated, friend.
-- Chris Paul Rex Consulting, Inc email: chris.paul@rexconsulting.net phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
On Sun, 10 Aug 2008, Chris Paul wrote:
And we'll change to IPv6 tomorrow! Total apples and oranges. We all have to patch anyhow. This is just code and firewall rules. IPv6 is way more complicated, friend.
No - IPv6 is just code (and not even firewall rules). cheers! ========================================================================== "A cat spends her life conflicted between a deep, passionate and profound desire for fish and an equally deep, passionate and profound desire to avoid getting wet. This is the defining metaphor of my life right now."
Pretending for a moment that it was even possible to make such large scale changes and get them pushed into a large enough number of clients to matter, you're talking about meltdown at the recurser level, because it isn't just one connection per _computer_, but one connection per _resolver stub_ per _computer_ (which, on a UNIX machine, would tend to gravitate towards one connection per process), and this just turns into an insane number of sockets you have to manage.
Couldn't the resolver libraries be changed to not use multiple connections?
I think that the text I wrote clearly assumes that there IS only one connection per resolver instance. The problem is that hostname to IP lookup is pervasive in a modern UNIX system, and is probably pretty common on other platforms, too, so you have potentially hundreds or thousands of processes, each eating up additional system file descriptors for this purpose. I cannot think of any reason that init, getty, sh, cron, or a few other things on a busy system would need to use the resolver library - but that leaves a whole ton of things that can and do. Now, of course, you can /change/ how everything works. Stop holding open connections persistently, and a lot of the trouble is reduced. However, anyone who has done *any* work in the area of TCP services that are open to the public will be happy to stamp "Fraught With Peril" on this little project - and to understand why, I suggest you research all the work that has been put into defending services like http, irc, etc. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Joe Greco wrote:
Pretending for a moment that it was even possible to make such large scale changes and get them pushed into a large enough number of clients to matter, you're talking about meltdown at the recurser level, because it isn't just one connection per _computer_, but one connection per _resolver stub_ per _computer_ (which, on a UNIX machine, would tend to gravitate towards one connection per process), and this just turns into an insane number of sockets you have to manage.
Couldn't the resolver libraries be changed to not use multiple connections?
I think that the text I wrote clearly assumes that there IS only one connection per resolver instance. The problem is that hostname to IP lookup is pervasive in a modern UNIX system, and is probably pretty common on other platforms, too, so you have potentially hundreds or thousands of processes, each eating up additional system file descriptors for this purpose.
Well how I read what you first wrote implied that the resolvers are now going to DOS servers with millions of connections due to each resolver stub making a TCP connection... I say this is something that if true, can and should be changed. Now you say that file descriptors on the client are going to run out Isn't that changing the topic? And is that even really a problem? So each process that needs to do a lookup opens a file descriptor for a TCP connection, right? Whereas with UDP we don't have to do this. Is this what I'm hearing you say? That I understand. (Hmm don't udp connections take sockets too? Not sarcastic here.. just asking...) And it is a good point but is this client file descriptor an insurmountable problem? Also, what about the millions of connections to the server? Is that really necessary for a dns resolver on one system to open more than one TCP connection to its caching dns server? I'm not saying that caching dns servers should keep open TCP connections to authoritative name servers! OK? But how much latency do you increase e on that uncached recursive lookup by changing to TCP? CP -- Chris Paul Rex Consulting, Inc email: chris.paul@rexconsulting.net phone, direct: +1, 831.706.4211 phone, toll-free: +1, 888.403.8996 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. Rex Consulting, Inc. is a California Corporation. P Please don't print this e-mail, unless you really need to.
I think that the text I wrote clearly assumes that there IS only one connection per resolver instance. The problem is that hostname to IP lookup is pervasive in a modern UNIX system, and is probably pretty common on other platforms, too, so you have potentially hundreds or thousands of processes, each eating up additional system file descriptors for this purpose.
Well how I read what you first wrote implied that the resolvers are now going to DOS servers with millions of connections due to each resolver stub making a TCP connection... I say this is something that if true, can and should be changed.
Sure. We can introduce a new feature, called a "local recurser," which will do unified name resolution for all lookups asked for by any process on the box. Now, of course, the box enjoys certain benefits, such as being able to remember who "MX nanog.org" is for the second time without having to bother an external recurser. And a hypothetical ability to forward all requests via TCP to the external recurser. Except, why bother, now that you have the capability right on the box, why be dependent on anything else? Might as well just let it resolve all by itself. Of course, the box also enjoys certain other liabilities, such as the next time that all the name servers in the world need to be upgraded, you now have just that many more recursers that are running on unattended autopilot (because heaven knows, most PC's run without a professional admin to keep things up to snuff, and this last problem wasn't exactly the sort of thing you can just "auto update," because someone actually has to verify that there aren't externalities such as NAT devices, etc!) Sounds like a real fun time.
Now you say that file descriptors on the client are going to run out Isn't that changing the topic? And is that even really a problem?
Actually, it's quite a problem, for the server. Try, sometime, having a few thousand file descriptors all open, and then running select() on that fdset. But it's not even really that pleasant on many clients. It's a kernel consumable. You try to avoid introducing additional requirements without a good reason.
So each process that needs to do a lookup opens a file descriptor for a TCP connection, right? Whereas with UDP we don't have to do this. Is this what I'm hearing you say? That I understand. (Hmm don't udp connections take sockets too? Not sarcastic here.. just asking...)
You open and then close it for UDP. You can do that for TCP, too, at a substantial penalty.
And it is a good point but is this client file descriptor an insurmountable problem? Also, what about the millions of connections to the server? Is that really necessary for a dns resolver on one system to open more than one TCP connection to its caching dns server?
There is no "DNS resolver on one system" unless you put one there. At which point, you can safely ask the question of why would you then connect to a caching server (there are good reasons, in some cases). The way libresolv works is that it takes those "nameserver" things listed in resolv.conf and sends requests to them. Since any program that uses the network is likely to be linked to libresolv, you can have lots of different programs on a system, each of which may want to resolve different name resources. There is no monolithic "thing" on a box to do name resolution unless you put one there.
I'm not saying that caching dns servers should keep open TCP connections to authoritative name servers! OK? But how much latency do you increase e on that uncached recursive lookup by changing to TCP?
Since latency would not be extremely high on my list of concerns with this plan, I'll pass and say "I don't really care to speculate." There are many other ways you'll have lit your hair on fire before latency is a big concern. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Joe Greco wrote:
Actually, it's quite a problem, for the server. Try, sometime, having a few thousand file descriptors all open, and then running select() on that fdset. But it's not even really that pleasant on many clients. It's a kernel consumable. You try to avoid introducing additional requirements without a good reason.
I thought that that was why poll(2) was your friend? Not that I'm saying that TCP is the right thing here, but things have evolved on the kernel interface front too, and I'd think that's even more so with (semi) purpose built OS's. It seems to me that DNS is an important enough service that we shouldn't require it least common denominator to creaky 30 year old API's any more than, oh say, routers should. Mike, it's the recursion argument that's really that seems to me to be what is most persuasive of why this is a lot different than http or ldap or most everything else
Unix machines set up by anyone with half a brain run a local caching server, and use forwarders. IE, the nameserver process can establish a persistent TCP connection to its trusted forwarders, if we just let it. That old sneer we used to use against Windows users of not having a "full featured host" and all. Windows stub resolvers multiplex through AD to a MS DNS server; which can easily use TCP to its trusted forwarders; unless they have no DC, which is not so common; in which case they just use standard queries, presumably to a patched ISP host (often a Nominum box). In both cases, the fix is in the local server, which serves only a few (and in a "full featured host" only one) machines using TCP to its forwarder, and the chain repeating itself. I don't see the problem with going to TCP for the recursive queries here. It's akin to the CDN scaling model, which has worked pretty well.
-----Original Message----- From: Joe Greco [mailto:jgreco@ns.sol.net] Sent: Sunday, August 10, 2008 3:14 PM To: Chris Paul Cc: nanog@merit.edu Subject: Re: maybe a dumb idea on how to fix the dns problems i don't know....
But we only care about TCP connection setup time in *interactive* sessions (a human using something like the web). If you have a persistent connection to your dns server from your dns resolver on your browser machine, you just send the request.... no TCP setup there at all. You can even pool connections. We do this stuff in LDAP all the time.
How does TCP resolution work in most resolver libraries? A TCP connection for each lookup? That is kind of dumb isn't it, speaking of dumb.... I actually don't know. Not much of a coder, so I'll let you coders check your code and get back to me on that...
well.. maybe i'll fire up snort or wireshark and check it out later with some different dns libs....
Pretending for a moment that it was even possible to make such large scale changes and get them pushed into a large enough number of clients to matter, you're talking about meltdown at the recurser level, because it isn't just one connection per _computer_, but one connection per _resolver stub_ per _computer_ (which, on a UNIX machine, would tend to gravitate towards one connection per process), and this just turns into an insane number of sockets you have to manage.
For your average ISP recurser where they only have 50,000 people online at any given time, this could still be half a million open sockets. We already know this sort of thing doesn't scale well.
This is very broken in any number of other ways. This message is not intended to imply otherwise.
... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Unix machines set up by anyone with half a brain run a local caching server, and use forwarders. IE, the nameserver process can establish a persistent TCP connection to its trusted forwarders, if we just let it.
Organizations often choose not to do this because doing so involves more risk and more things to update when the next vulnerability appears. In many cases, you are suggesting additional complexity and management requirements. A hosting company, for example, might have 20 racks of machines with 40 machines each, which is 800 servers. If half of those are UNIX, then you're talking about 402 nameservers instead of just 2. Since local bandwidth is "free", this could be seen as a poor engineering choice, and you still had to maintain those two servers for the other (Windows or whatever) boxes anyways. On the upside, you don't need to use a forwarders arrangement unless you really want to... but the benefit of those 400 extra nameserver instances is a bit sketchy. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Joe Greco wrote:
Unix machines set up by anyone with half a brain run a local caching server, and use forwarders. IE, the nameserver process can establish a persistent TCP connection to its trusted forwarders, if we just let it.
Organizations often choose not to do this because doing so involves more risk and more things to update when the next vulnerability appears. In many cases, you are suggesting additional complexity and management requirements. A hosting company, for example, might have 20 racks of machines with 40 machines each, which is 800 servers. If half of those are UNIX, then you're talking about 402 nameservers instead of just 2.
[Customers] <--/UDP/--> [DNS Cache] <--/TCP/--> [DNS servers] Not so? Of course, one shouldn't let the rest of the internet touch your DNS Cache query interface... but that's just obvious. I mentioned this a while ago though, so I demand credit ;P Also, I think there is probably an IETF DNS WG list where this fits on topic (I have no idea what it may be though).
I think Colin just said everything I said, but in 1/10'th the words. And he posted before me. Drats. --Patrick Darden -----Original Message----- From: Colin Alston [mailto:karnaugh@karnaugh.za.net] Sent: Monday, August 11, 2008 8:38 AM To: Joe Greco Cc: nanog@merit.edu Subject: Re: maybe a dumb idea on how to fix the dns problems i don't know.... Joe Greco wrote:
Unix machines set up by anyone with half a brain run a local caching server, and use forwarders. IE, the nameserver process can establish a persistent TCP connection to its trusted forwarders, if we just let it.
Organizations often choose not to do this because doing so involves more risk and more things to update when the next vulnerability appears. In many cases, you are suggesting additional complexity and management requirements. A hosting company, for example, might have 20 racks of machines with 40 machines each, which is 800 servers. If half of those are UNIX, then you're talking about 402 nameservers instead of just 2.
[Customers] <--/UDP/--> [DNS Cache] <--/TCP/--> [DNS servers] Not so? Of course, one shouldn't let the rest of the internet touch your DNS Cache query interface... but that's just obvious. I mentioned this a while ago though, so I demand credit ;P Also, I think there is probably an IETF DNS WG list where this fits on topic (I have no idea what it may be though).
Joe makes some good points here. I'd have to add one caveat though: it depends. It depends on the server. Busy email servers definitely depend on having fast DNS, and benefit *greatly* from a caching DNS server using local sockets instead. Web servers generally don't. Centralized logging servers benefit greatly. Usually, for a pocket of servers like Joe describes, you want some local dedicated DNS servers (e.g. ~800 servers, add 2 more just for local DNS) plus you would want caching DNS servers running directly on your email, logging, etc. servers. Yeah, 400-800 extra caching DNS servers would probably be overkill though! I am intrigued by the idea of persistent connections for those 2 dedicated DNS servers--only for upstream though. You wouldn't need so much security locally (for your 800 clients), I expect. You could use UDP for speed, and not worry too much about poisoning. Expecially if you were using some kind of dedicated professional DNS service that required IPSEC pipes, and had engineers only doing DNS: security, updates, patching, uptime, etc. etc. It would be interesting if such professional services came about Companies that do DNS and that is all they do. Dedicated to the reliability and security of one of the cornerstones of the net. We already went through that with Usenet, email, web hosting, and other of the main services. --Patrick Darden -----Original Message----- From: Joe Greco [mailto:jgreco@ns.sol.net] Sent: Monday, August 11, 2008 8:10 AM To: Tomas L. Byrnes Cc: nanog@merit.edu Subject: Re: maybe a dumb idea on how to fix the dns problems i don't know....
Unix machines set up by anyone with half a brain run a local caching server, and use forwarders. IE, the nameserver process can establish a persistent TCP connection to its trusted forwarders, if we just let it.
Organizations often choose not to do this because doing so involves more risk and more things to update when the next vulnerability appears. In many cases, you are suggesting additional complexity and management requirements. A hosting company, for example, might have 20 racks of machines with 40 machines each, which is 800 servers. If half of those are UNIX, then you're talking about 402 nameservers instead of just 2. Since local bandwidth is "free", this could be seen as a poor engineering choice, and you still had to maintain those two servers for the other (Windows or whatever) boxes anyways. On the upside, you don't need to use a forwarders arrangement unless you really want to... but the benefit of those 400 extra nameserver instances is a bit sketchy. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
participants (9)
-
Cat Okita -
Chris Paul -
Colin Alston -
Darden, Patrick S. -
Joe Greco -
list-nanog＠pwns.ms -
Michael Thomas -
Tomas L. Byrnes -
Victor Jerlin