large organization nameservers sending icmp packets to dns servers.
Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization? Thanks, -Drew
On Mon, 06 Aug 2007 11:53:15 EDT, Drew Weaver said:
Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why?
Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest. Yes, it's a semi-borkken strategy, because it assumes that: 1) ICMP is handled at the same rate as TCP/UDP packets in all the routers involved (so there's no danger of declaring a path "slow" when it really isn't, just becase a router slow-pathed ICMP). 2) That the actual requester of service is reasonably near net-wise to the server handling the end-user's recursive DNS lookup.
On Mon, 06 Aug 2007 11:57:08 -0400 Valdis.Kletnieks@vt.edu wrote:
On Mon, 06 Aug 2007 11:53:15 EDT, Drew Weaver said:
Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why?
Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest.
Yes, it's a semi-borkken strategy, because it assumes that:
1) ICMP is handled at the same rate as TCP/UDP packets in all the routers involved (so there's no danger of declaring a path "slow" when it really isn't, just becase a router slow-pathed ICMP).
This is aimed at hosts, not routers, right? As far as I know, routers don't slow-path forwarded ICMP. Hosts will probably reply to ICMP from their kernel, so it's a faster response than a user-level DNS reply.
2) That the actual requester of service is reasonably near net-wise to the server handling the end-user's recursive DNS lookup.
Right. But there's no particular reason to block it, unless the rate is high enough that it's causing you CPU or network load problems. (If it's your IDS that's getting overloaded, perhaps tell it not to worry unless you see other load issues...) --Steve Bellovin, http://www.cs.columbia.edu/~smb
Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest.
Why would they ping rather than just sending the query to all of the NS and see which one answers first? It's an IP round trip either way. I agree that pinging is harmless, but for this application it seems pointless, too. R's, John
On Mon, 06 Aug 2007 17:21:49 -0000, John Levine said:
Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest.
Why would they ping rather than just sending the query to all of the NS and see which one answers first? It's an IP round trip either way.
If you have sites in San Fran, London, and Tokyo, and you launch a ping from all 3 and see which one gets there first, you'll *know* the RTT from each site. If you just send DNS replies from all 3, you don't have a good way of telling which one got to the destination first. Your method works if *I* want to know which one of the 3 sites is closest (assuming I can identify an DNS server at the 3 sites). The problem of the owner of the 3 sites trying to identify which one I'm closest to isn't symmetric to it.
Why would they ping rather than just sending the query to all of the NS and see which one answers first? It's an IP round trip either way.
If you have sites in San Fran, London, and Tokyo, and you launch a ping from all 3 and see which one gets there first, you'll *know* the RTT from each site.
If you just send DNS replies from all 3, you don't have a good way of telling which one got to the destination first.
Um, unless I seriously misunderstand the client DNS cache wants to know which server is closest. So it sends DNS queries to all three NS at the same time. Then it waits for the answers. Whichever one answers first is the closest. What am I missing? Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.
On Aug 6, 2007, at 1:47 PM, John L wrote:
Why would they ping rather than just sending the query to all of the NS and see which one answers first? It's an IP round trip either way.
If you have sites in San Fran, London, and Tokyo, and you launch a ping from all 3 and see which one gets there first, you'll *know* the RTT from each site.
If you just send DNS replies from all 3, you don't have a good way of telling which one got to the destination first.
Um, unless I seriously misunderstand the client DNS cache wants to know which server is closest. So it sends DNS queries to all three NS at the same time. Then it waits for the answers. Whichever one answers first is the closest. What am I missing?
The client DNS doesn't know there is more than one server. It queries for www.$FOO.com, and the authority for $FOO.com replies with the IP address of the 'closest' web server. This result could be pre-calculated by all the web servers pinging the client DNS. It could be done lots of ways, but that is what we are discussing today. Owen said it worked well for his customers (in a past life), and he has operational experience with this. Can anyone give a serious counter example _from experience_? Or are we just discussing possibilities? -- TTFN, patrick
On Mon, 6 Aug 2007, Patrick W. Gilmore wrote: first I agree that in most cases the 'RTT to client cacheresolver' probably works well enough. That said though...
Owen said it worked well for his customers (in a past life), and he has operational experience with this. Can anyone give a serious counter example _from experience_? Or are we just discussing possibilities?
Sure, 75% of the people that use cache00.ns.uu.net and aren't in the 'mid-atlantic' region... (and someone's blocking echo-request to cache00 but...) Unless there exist exceptions and other metrics for the clients it's probably not very accurate in this instance (or the other cacheXX.ns.uu.net cases really, since they tend to move about the network as required...)
John L wrote:
Um, unless I seriously misunderstand the client DNS cache wants to know which server is closest. So it sends DNS queries to all three NS at the same time. Then it waits for the answers. Whichever one answers first is the closest. What am I missing?
The bind_garbadge_collection_delay. From time to time bind goes to sleep, depending on the size and number of zones the nameserver does host. Maybe it is internal garbadge collection or memory reclaiming. Maybe it is the operating system that causes the delay. That delay is very random. icmp is answered by the network layer. dns is answered after the network layer passes the packet to the operating system, after the operating system passes the packet to the tcp/ip subsystem, after the tcp/ip subsystem passes the packet to the socket subsystem, after the memorymanager swappes bind back into memory - and all the way back. Each of the steps could envolve the memory manager swapping memory to disk or from disk. Each of the delays could be longer that all network delays. I have seen nameservers answering after seconds. I have seen pings to those servers returning after only 30 millisecs. Kind regards Peter and Karin -- Peter and Karin Dambier Cesidian Root - Radice Cesidiana Rimbacher Strasse 16 D-69509 Moerlenbach-Bonsweiher +49(6209)795-816 (Telekom) +49(6252)750-308 (VoIP: sipgate.de) mail: peter@peter-dambier.de mail: peter@echnaton.arl.pirates http://iason.site.voila.fr/ https://sourceforge.net/projects/iason/ http://www.cesidianroot.com/
On 6-Aug-2007, at 16:31, Peter Dambier wrote:
John L wrote:
Um, unless I seriously misunderstand the client DNS cache wants to know which server is closest. So it sends DNS queries to all three NS at the same time. Then it waits for the answers. Whichever one answers first is the closest. What am I missing?
The bind_garbadge_collection_delay.
From time to time bind goes to sleep, depending on the size and number of zones the nameserver does host.
I think you're talking about the performance of BIND 9 as a cache. This is pretty tangential, but there was a good presentation on work being done to fix that performance problem by Michael Graff at the recent DNS ops workshop in Chicago: http://public.oarci.net/files/dnsops-2007/Graff-BIND9-cache.pdf Joe
Hi Guys, All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons... -J -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Valdis.Kletnieks@vt.edu Sent: Monday, August 06, 2007 11:35 AM To: John Levine Cc: nanog@nanog.org Subject: Re: large organization nameservers sending icmp packets to dns servers. On Mon, 06 Aug 2007 17:21:49 -0000, John Levine said:
Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest.
Why would they ping rather than just sending the query to all of the NS and see which one answers first? It's an IP round trip either way.
If you have sites in San Fran, London, and Tokyo, and you launch a ping from all 3 and see which one gets there first, you'll *know* the RTT from each site. If you just send DNS replies from all 3, you don't have a good way of telling which one got to the destination first. Your method works if *I* want to know which one of the 3 sites is closest (assuming I can identify an DNS server at the 3 sites). The problem of the owner of the 3 sites trying to identify which one I'm closest to isn't symmetric to it.
All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons... Then most are incredibly stupid.
Several anti DoS utilities force unknown hosts to initiate a query via TCP in order to be whitelisted. If the host can't perform a TCP query then they get blacklisted. In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems. -Don
On Aug 7, 2007, at 2:14 PM, Donald Stahl wrote:
All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons...
Then most are incredibly stupid.
Those are impressively harsh words. Mind if I ask what operational experience you have with name servers behind firewalls filtering TCP53? I have none, so perhaps you could enlighten us with your vast experience?
Several anti DoS utilities force unknown hosts to initiate a query via TCP in order to be whitelisted. If the host can't perform a TCP query then they get blacklisted.
That trick is so well known, most people turn it off since there has been more than one instance of large, well known organizations suffering spectacular failures by using it. The phrase "worse than the disease" comes to mind.
In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems.
Since we are talking about authorities here, one can control the size of ones responses. Unless, of course, you are so incredibly stupid you can't figure out the difference between an authority and a caching server. -- TTFN, patrick
On 7-Aug-2007, at 14:38, Patrick W. Gilmore wrote:
On Aug 7, 2007, at 2:14 PM, Donald Stahl wrote:
All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons...
Then most are incredibly stupid.
Those are impressively harsh words.
But they are hard to argue with.
In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems.
Since we are talking about authorities here, one can control the size of ones responses.
"Never reply with anything big and hence never set TC" seems like a reasonable, expedient way to circumvent the problem of wholesale 53/ tcp-blocking stupidity. It doesn't make the behaviour any less stupid, though. The "security" argument looks even more bizarre when you consider what the DO bit in a request will do in general to the size of a response, in the case of an authority server which has signed zone data. Joe
From: Joe Abley <jabley@ca.afilias.info> Date: Tue, 7 Aug 2007 15:19:30 -0400 Sender: owner-nanog@merit.edu
On 7-Aug-2007, at 14:38, Patrick W. Gilmore wrote:
On Aug 7, 2007, at 2:14 PM, Donald Stahl wrote:
All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons...
Then most are incredibly stupid.
Those are impressively harsh words.
But they are hard to argue with.
In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems.
Since we are talking about authorities here, one can control the size of ones responses.
"Never reply with anything big and hence never set TC" seems like a reasonable, expedient way to circumvent the problem of wholesale 53/ tcp-blocking stupidity. It doesn't make the behaviour any less stupid, though.
The "security" argument looks even more bizarre when you consider what the DO bit in a request will do in general to the size of a response, in the case of an authority server which has signed zone data.
This has been a pain for me for years. I have tried to reason with security people about this and, while they don't dispute my reasoning, they always end up saying that it is the "standard" practice and that, lacking any evidence of what it might be breaking, it will continue to be blocked. And I don't mean small companies, either. One of the biggest issues I have is with one of the countries largest government funded research labs. Wonder how often DNSSEC might make non-transfer queries tickle this and really break things? (Assuming we ever get wide use of DNSSEC.) -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
This has been a pain for me for years. I have tried to reason with security people about this and, while they don't dispute my reasoning, they always end up saying that it is the "standard" practice and that, lacking any evidence of what it might be breaking, it will continue to be blocked. And I don't mean small companies, either. One of the biggest issues I have is with one of the countries largest government funded research labs. Can someone, anyone, please explain to me why blocking TCP 53 is considered such a security enhancement? It's a token gesture and does nothing to really help improve security. It does, however, cause problems.
You have no way of knowing why a client might want or need to contact you via TCP 53 for DNS- so why would you block them? The fact is most people, to this day, still believe that TCP 53 is only used for axfr's. Someone was only too happy to point out to me that he would never create a record larger than 512 bytes so why should they allow TCP queries? The answer is simple- because they are supposed to be allowed. By disallowing them you are breaking the agreed upon rules for the protocol. Before long it becomes impossible to implement new features because you can't be sure if someone else hasn't broken something intentionally. If you don't like the rules- then change the damned protocol. Stop just doing whatever you want and then complaining when other people disagree with you. -Don
Date: Tue, 7 Aug 2007 16:33:22 -0400 (EDT) From: Donald Stahl <don@calis.blacksun.org>
This has been a pain for me for years. I have tried to reason with security people about this and, while they don't dispute my reasoning, they always end up saying that it is the "standard" practice and that, lacking any evidence of what it might be breaking, it will continue to be blocked. And I don't mean small companies, either. One of the biggest issues I have is with one of the countries largest government funded research labs. Can someone, anyone, please explain to me why blocking TCP 53 is considered such a security enhancement? It's a token gesture and does nothing to really help improve security. It does, however, cause problems.
You have no way of knowing why a client might want or need to contact you via TCP 53 for DNS- so why would you block them?
The fact is most people, to this day, still believe that TCP 53 is only used for axfr's.
Someone was only too happy to point out to me that he would never create a record larger than 512 bytes so why should they allow TCP queries? The answer is simple- because they are supposed to be allowed. By disallowing them you are breaking the agreed upon rules for the protocol. Before long it becomes impossible to implement new features because you can't be sure if someone else hasn't broken something intentionally.
If you don't like the rules- then change the damned protocol. Stop just doing whatever you want and then complaining when other people disagree with you.
Don, You are preaching to the choir...at least in the group. But I have found that security types (I mean those with a police/physical security background) don't must care for these arguments. It usually comes down to "lock and bar every door unless you can prove to them that there is a need to have the door unlocked". Standards and such mean nothing to them. Only evidence that something is broken that has to work will convince them to change something. It's the tcp/53 is evil meme. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
On Tue, Aug 07, 2007 at 01:50:33PM -0700, Kevin Oberman wrote:
that security types (I mean those with a police/physical security background) don't must care for these arguments. It usually comes down to "lock and bar every door unless you can prove to them that there is a need to have the door unlocked".
So these people are also the ones responsible for chaining shut fire doors because "fires never happen in this building, but theft does"? I sure feel safer now! The "need to have the door unlocked" is because that's the way the building is designed to fail its fireproofing. And the need to have the TCP port open is because that's the way the network protocol is designed to fail from UDP. If "this is the way the protocol works" is not enough of an argument, then I'm afraid we're past the point of engineering and into the realm of tea-leaf readers and chicken-entrail-based prognosticators. I'm aware there are such people promoting themselves as security experts. It's rather depressing that those people can still find gainful employment; but in this post-literate age where people prefer to repeat (or listen to) foolish bromides rather than Read the Fine Commentaries that define the protocol, I suppose I ought not to be surprised. Shocked but not surprised, A ---- Andrew Sullivan 204-4141 Yonge Street Afilias Canada Toronto, Ontario Canada <andrew@ca.afilias.info> M2P 2A8 +1 416 646 3304 x4110
On Aug 7, 2007, at 2:23 PM, Andrew Sullivan wrote:
On Tue, Aug 07, 2007 at 01:50:33PM -0700, Kevin Oberman wrote:
that security types (I mean those with a police/physical security background) don't must care for these arguments. It usually comes down to "lock and bar every door unless you can prove to them that there is a need to have the door unlocked".
...
The "need to have the door unlocked" is because that's the way the building is designed to fail its fireproofing. And the need to have the TCP port open is because that's the way the network protocol is designed to fail from UDP.
Ensuring an authoritative domain name server responds via UDP is a critical security requirement. TCP will not create the same risk of a resolver being poisoned, but a TCP connection will consume a significant amount of a name server's resources. ACLs restricting TCP fall-back is fairly common. For example, too many bytes might be placed into a domain's SPF records. While TCP offers a fallback mode of operation for this fairly common error, this fallback does not ensure oversize records are fixed promptly. TCP fallback on such records leaves open an opportunity to stage DDoS attacks when bad actors wishes to take down authoritative name servers while also attempting to poison resolvers. Here again, SPF might offer access to remote resolvers query for the records to be poisoned, isolate query ports, and time poison records. : ( http://www.ietf.org/internet-drafts/draft-ietf-dnsext-forgery- resilience-01.txt -Doug
i normally agree with doug.... dotis@mail-abuse.org (Douglas Otis) writes:
Ensuring an authoritative domain name server responds via UDP is a critical security requirement. TCP will not create the same risk of a resolver being poisoned, but a TCP connection will consume a significant amount of a name server's resources.
...but this is flat out wrong, dead wrong, no way to candy coat it, wrong. -- Paul Vixie
On Aug 8, 2007, at 12:11 PM, Paul Vixie wrote:
dotis@mail-abuse.org (Douglas Otis) writes:
Ensuring an authoritative domain name server responds via UDP is a critical security requirement. TCP will not create the same risk of a resolver being poisoned, but a TCP connection will consume a significant amount of a name server's resources.
...but this is flat out wrong, dead wrong, no way to candy coat it, wrong.
Wanting to understand this comment, I'll expand upon the quoted statement. Resolver's factors affecting DNS security are: - selection of port and transaction IDs - restrictions on outstanding queries for same resource - limits on inbound bandwidth Ignoring uncontrollable factors... Authoritative server factors affecting security are: - time frame for an answer - duration of RR TTLs - number of servers A short time frame for an answer along with longer TTLs are influenced by authoritative servers and also affect spoofing rates. When DNS TCP is used, the transport sequence number further precludes a spoofed TCP answer from being accepted. When a truncated response is returned, TCP fallback may be attempted. When a TCP ICMP refusal is filtered or never sent, but TCP has been blocked, the timeframe alloted for spoofing could entail the entire TCP timeout. However, probability for successful spoofing includes an additional multiplier of 1 / 2^32. This reduction should sufficiently negate an additional timeout duration. TCP requires state and introduces several additional exchanges for a given number of answers. Any effort related to poisoning will likely attempt to delay an answer by adding to the server's overhead. Precluding truncation, and thereby eliminating TCP, should favorably reduce server overhead and increase overall performance. Of course, a more practical method would be to ensure sufficient DNS resources are available by increasing server resources. That said, how many domains allocate a couple of prior generation servers for DNS? -Doug
... but a TCP connection will consume a significant amount of a name server's resources.
...wrong.
Wanting to understand this comment, ...
the resources given a nameserver to TCP connections are tightly controlled, as described in RFC 1035 4.2.2. so while TCP/53 can become unreliable during high load, the problems will be felt by initiators not targets. (this is why important AXFR targets have to be firewalled down to a very small population of just one's own nameservers, and is why important zones have to use unpublished primary master servers, and is why f-root's open AXFR of the root zone is a diagnostic service not a production service.)
On Aug 8, 2007, at 5:35 PM, Paul Vixie wrote:
... but a TCP connection will consume a significant amount of a name server's resources.
...wrong.
Wanting to understand this comment, ...
the resources given a nameserver to TCP connections are tightly controlled, as described in RFC 1035 4.2.2. so while TCP/53 can become unreliable during high load, the problems will be felt by initiators not targets.
The relevant entry in Section 1035 4.2.2 recommends that the server not block other activities waiting for TCP data. This is not exactly a requirement that TCP should fail before UDP. The concern leading to a suggestion that TCP always fail was a bit different. A growing practice treats DNS as a type of web server when used to publish rather bulky script-like resource records. Due to typical sizes, it is rather common to find these records depend upon TCP fallback. This problem occurred with paypal, for example. TCP fallback is especially problematic when these records are given wildcards. Such fallback increases the amplification associated with an exploit related to the use of the script within the record. Of course there are better ways to solve this problem, but few are as certain. -Doug
the resources given a nameserver to TCP connections are tightly controlled, as described in RFC 1035 4.2.2. so while TCP/53 can become unreliable during high load, the problems will be felt by initiators not targets.
The relevant entry in Section 1035 4.2.2 recommends that the server not block other activities waiting for TCP data. This is not exactly a requirement that TCP should fail before UDP.
it is semantically equivilent to such a requirement, in that UDP/53 is an "other activity" performed by name servers. it happens to be implemented this way in all versions of BIND starting in 4.8 or so (when named-xfer was made a separate executable), all versions of Windows DNS, and all current name server implementations i am aware of (including powerdns, nominum ANS, and NSD). so while "not exactly", it's "effectively" a requirement, and i think we ought to design our systems with this constraint as a given.
The concern leading to a suggestion that TCP always fail was a bit different. A growing practice treats DNS as a type of web server when used to publish rather bulky script-like resource records. Due to typical sizes, it is rather common to find these records depend upon TCP fallback. This problem occurred with paypal, for example. TCP fallback is especially problematic when these records are given wildcards. Such fallback increases the amplification associated with an exploit related to the use of the script within the record.
Of course there are better ways to solve this problem, but few are as certain.
i think you're advising folks to monitor their authority servers to find out how many truncated responses are going out and how many TCP sessions result from these truncations and how many of these TCP sessions are killed by the RFC1035 4.2.2 connection management logic, and if the numbers seem high, then they ought to change their applications and DNS content so that truncations no longer result. or perhaps you're asking that EDNS be more widely implemented, that it not be blocked by firewalls or perverted by hotelroom DNS middleboxes, and that firewalls start allowing UDP fragments (which don't have port numbers and therefore won't be allowed by UDP/53 rules). i would agree with either recommendation. but i won't agree that TCP creates stability or load problems for servers.
On Thu, 09 Aug 2007 21:05:26 -0000, Paul Vixie said:
i think you're advising folks to monitor their authority servers to find out how many truncated responses are going out and how many TCP sessions result from these truncations and how many of these TCP sessions are killed by the RFC1035 4.2.2 connection management logic, and if the numbers seem high, then they ought to change their applications and DNS content so that truncations no longer result.
How does the (eventual) deployment of DNSSEC change these numbers? And who's likely to feel *that* pain first?
Valdis.Kletnieks@vt.edu writes:
... advising folks to monitor their authority servers to find out how many truncated responses are going out and how many TCP sessions result from these truncations and how many of these TCP sessions are killed by the RFC1035 4.2.2 connection management logic, and if the numbers seem high, then they ought to change their applications and DNS content so that truncations no longer result.
How does the (eventual) deployment of DNSSEC change these numbers?
DNSSEC cannot be signalled except in EDNS.
And who's likely to feel *that* pain first?
the DNSSEC design seems to distribute pain very fairly. -- Paul Vixie
On Thu, 09 Aug 2007 22:58:40 -0000, Paul Vixie said:
How does the (eventual) deployment of DNSSEC change these numbers?
DNSSEC cannot be signalled except in EDNS.
Right. Elsewhere in this thread, somebody discussed ugly patches to keep the packet size under 512. I dread to think how many different ways of "protecting" DNS are deployed that will break EDNS, and just haven't been noticed because there's little enough *actual* EDNS breakage that it's down in the noise of *other* "random voodoo" breakage at those sites.
And who's likely to feel *that* pain first?
the DNSSEC design seems to distribute pain very fairly.
I actually meant "which 800 pound gorilla is going to try this first and find all the bustifications", but your answer is good too.. :)
On Aug 9, 2007, at 2:05 PM, Paul Vixie wrote: Your comments have helped.
i think you're advising folks to monitor their authority servers to find out how many truncated responses are going out and how many TCP sessions result from these truncations and how many of these TCP sessions are killed by the RFC1035 4.2.2 connection management logic, and if the numbers seem high, then they ought to change their applications and DNS content so that truncations no longer result.
Monitoring is a good recommendation, as many incorrectly estimate record limits. Wildcard resources should also be checked against maximal labels. Fallback may occur with resource records encompassing a bit more than a couple hundred bytes. The assurance TCP will fail first is heartening. How this protection interacts with an emerging exploit could be interesting. I'll try to setup some tests and be less pragmatic.
or perhaps you're asking that EDNS be more widely implemented, that it not be blocked by firewalls or perverted by hotelroom DNS middleboxes, and that firewalls start allowing UDP fragments (which don't have port numbers and therefore won't be allowed by UDP/53 rules).
TCP offers a means to escape UDP related issues. On the other hand, blocking TCP may offer the necessary motivation for having these UDP issues fixed. After all, only UDP should be required. When TCP is designed to readily fail, reliance upon TCP seems questionable. As DNSSEC in introduced, TCP could be relied upon in the growing number of instances where UDP is improperly handled. UDP handling may have been easier had EDNS been limited to 1280 bytes. On the other hand, potentially larger messages may offer the necessary motivation for adding ACLs on recursive DNS, and deploying BCP 38. No pain, no gain might be a motto that applies equally to DNS as it does for weight lifting. -Doug
Your comments have helped.
groovy.
When TCP is designed to readily fail, reliance upon TCP seems questionable.
i caution against being overly cautious about DNS TCP if you're using RFC 1035 section 4.2.2 as your basis for special caution. DNS TCP only competes directly against other DNS TCP. there are only two situations where a DNS TCP state blob is present in a DNS target ("server") long enough to be in any danger: when doing work upstream to fulfill the query, and in zone transfers. when answering DNS TCP queries in an authority server, there is by definition no "upstream work" to be done, other than possible backend database lookups which are beyond the scope of this discussion. these responses will usually be generated synchronous to the receipt of the last octet of a query, and the response will be put into the TCP window (if it's open, which it usually is), and the DNS target ("server") will then wait for the initiator ("client") to close the connection or send another query. (usually it's a close.) when answering DNS TCP zone transfer requests in an authority server, there is a much larger window of doom, during which spontaneous network congestion can close the outgoing TCP window and cause a DNS target ("server") to think that a TCP session is "idle" for the purpose of RFC 1035 section 4.2.2 TCP resource management. while incremental zone transfer is slightly less prone to this kind of doom than full zone transfer, since the sessions are shorter, it can take some time for the authority server to compute incremental zone "diffs", during which the TCP session may appear "idle" through no fault of the DNS initiator ("client") who is avidly waiting for its response. lastly, when answering DNS TCP queries in a recursive caching nameserver, it can take a while (one or more round trips to one or more authority servers) before there is enough local state to satisfy the query, during which time the TCP resources held by that query might be reclaimed under RFC 1035 section 4.2.2's rules. the reason why not to be overly cautious about TCP is that in the case where you're an authority server answering a normal query, the time window during which network congestion could close the outbound TCP window long enough for RFC 1034 section 4.2.2's rules to come into effect, is vanishingly short. so while it's incredibly unwise to depend on zone transfer working from a small number of targets to a large number of initiators, and it is in fact wise to firewall or ACL your stealth master server so that only your designated secondary servers can reach it, none of this comes into play for normal queries to authority servers -- only zone transfers to authority servers. the unmanageable risk is when a recursive caching nameserver receives a query by TCP and forwards/iterates upstream. if this happens too often, then the RFC 1035 section 4.2.2 rules will really hurt. and thus, it's wise, just as you say, to try to make sure other people don't have to use TCP to fetch data about your zone. the counterintuitive thing is that you won't be able to measure the problems at your authority server since that's not where they will manifest. they'll manifest at caching recursive servers downstream.
As DNSSEC in introduced, TCP could be relied upon in the growing number of instances where UDP is improperly handled.
this would be true if TCP fallback was used when EDNS failed. it's not. if EDNS fails, then EDNS will not be used, either via UDP or TCP. so if improper handling of UDP prevents EDNS from working, then EDNS and anything that depends on EDNS, including DNSSEC, will not be used.
UDP handling may have been easier had EDNS been limited to 1280 bytes.
if you mean, had EDNS been limited to nonfragmentation cases, then i think you might mean 576 bytes or even 296 bytes. 1280 is an IPv6 (new era) limit.
On the other hand, potentially larger messages may offer the necessary motivation for adding ACLs on recursive DNS, and deploying BCP 38.
i surely do hope so. we need those ACLs and we need that deployment, and if message size and TCP fallback is a motivator, then let's turn UP the volume.
On Aug 10, 2007, at 4:41 PM, Paul Vixie wrote:
On the other hand, potentially larger messages may offer the necessary motivation for adding ACLs on recursive DNS, and deploying BCP 38.
i surely do hope so. we need those ACLs and we need that deployment, and if message size and TCP fallback is a motivator, then let's turn UP the volume.
There are so many more larger and immediate reasons for doing these things that I seriously doubt message size and TCP fallback on the DNS will have any impact at all in terms of motivating the non- motivated. But, one can always hope. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice Culture eats strategy for breakfast. -- Ford Motor Company
On Fri, 10 Aug 2007 16:11:04 -0700 Douglas Otis <dotis@mail-abuse.org> wrote:
TCP offers a means to escape UDP related issues. On the other hand, blocking TCP may offer the necessary motivation for having these UDP issues fixed. After all, only UDP should be required. When TCP is designed to readily fail, reliance upon TCP seems questionable. As DNSSEC in introduced, TCP could be relied upon in the growing number of instances where UDP is improperly handled.
As a datapoint I ran some tests against a reasonably diverse and sizeable TLD zone I work with in another forum. I queried the name servers listed in the parent to see if I could successfuly query them for their corresponding domain name they are configured for using TCP. Out of about 9,300 unique name servers I failed to receive any answer from about 1700 of them. That is a bit more than an 18% failure rate. John
Hi, On Aug 7, 2007, at 1:33 PM, Donald Stahl wrote:
Can someone, anyone, please explain to me why blocking TCP 53 is considered such a security enhancement? It's a token gesture and does nothing to really help improve security. It does, however, cause problems.
It has been argued that it is a bit harder to download/bootstrap shell code/arbitrary root kit through the latest BIND vulnerability (or whatever) via a 512 UDP packet than it is through TCP.
Someone was only too happy to point out to me that he would never create a record larger than 512 bytes so why should they allow TCP queries? The answer is simple- because they are supposed to be allowed.
Yep. However, as the always amusing Dr. Bernstein points out, if you don't care about zone transfer, DNS-over-TCP is an optional part of the standard (per RFC 1123).
Before long it becomes impossible to implement new features because you can't be sure if someone else hasn't broken something intentionally.
Yep. And then they scream at you when you tickle their brokenness. It sucks. Rgds, -drc P.S. Note that I think blocking TCP/53 is really stupid.
On Aug 7, 2007, at 4:33 PM, Donald Stahl wrote: [...]
If you don't like the rules- then change the damned protocol. Stop just doing whatever you want and then complaining when other people disagree with you.
I think this last part is the key. Remember the old adage: "My network, My rules"? Have we forgotten that? Should I not block ports for MS protocols when a new worm spreads because it would break the E-2-E principal? What about spam filtering? Or a myriad of other things. Everyone here is breaking some RFC somehow. And most of us don't give a rats ass. Which is the way it should be. But when you decide that YOUR violation is MY problem to fix, then you are just being silly. And worse, annoying. Let's all just agree to run our own networks the way we damned well please, as long as we are not hurting anyone else. We just have to define "omplaining to you about things I b0rk'ed by myself" as "hurting you". Which isn't a stretch, support costs money, and costing me money because you screwed up is definitely hurtful. -- TTFN, patrick P.S. To be clear, no, your pr0n site not resolving because I don't support TCP does not qualify as "hurting you" unless I call you to complain about it, even if it loses you revenue.
On Aug 7, 2007, at 4:33 PM, Donald Stahl wrote:
If you don't like the rules- then change the damned protocol. Stop just doing whatever you want and then complaining when other people disagree with you.
I think this last part is the key.
Remember the old adage: "My network, My rules"? Have we forgotten that?
No, that's the point. The Internet is based on cooperation. You can run your network however you want, but if you fail to cooperate, other people will exercise their right to run their network how they want by blacklisting you.
Should I not block ports for MS protocols when a new worm spreads because it would break the E-2-E principal? What about spam filtering? Or a myriad of other things. Everyone here is breaking some RFC somehow. And most of us don't give a rats ass. Which is the way it should be.
Fine, so long as you don't break the promises you make to other networks. If you do that, you wreck the cooperation fabric the Internet is based on.
But when you decide that YOUR violation is MY problem to fix, then you are just being silly. And worse, annoying.
Let's all just agree to run our own networks the way we damned well please, as long as we are not hurting anyone else. We just have to define "omplaining to you about things I b0rk'ed by myself" as "hurting you". Which isn't a stretch, support costs money, and costing me money because you screwed up is definitely hurtful.
When you promise to provide a service to anyone who asks for it and then fail to, you impose costs on other people. Failing to resolve names that you claim you will resolve is just such a failure. It forces other people's resolvers to do extra work to get the information they need or they just can't get it. This is, IMO, the type of cooperation failure that justifies blacklisting. DS
On Aug 8, 2007, at 2:11 AM, David Schwartz wrote:
On Aug 7, 2007, at 4:33 PM, Donald Stahl wrote:
If you don't like the rules- then change the damned protocol. Stop just doing whatever you want and then complaining when other people disagree with you.
I think this last part is the key.
Remember the old adage: "My network, My rules"? Have we forgotten that?
No, that's the point. The Internet is based on cooperation. You can run your network however you want, but if you fail to cooperate, other people will exercise their right to run their network how they want by blacklisting you.
So we are in violent agreement. IOW: Your first word is incorrect. It _IS_ my network, and you agree it is my network, and you agree I am allowed to run it as I please. In return, I agree you can run your network as you please, even if that includes blacklisting me.
Should I not block ports for MS protocols when a new worm spreads because it would break the E-2-E principal? What about spam filtering? Or a myriad of other things. Everyone here is breaking some RFC somehow. And most of us don't give a rats ass. Which is the way it should be.
Fine, so long as you don't break the promises you make to other networks. If you do that, you wreck the cooperation fabric the Internet is based on.
Paying $10 and registering a domain IN NOW WAY means I promised a bazillion people anything. What happened to: "You can run your network however you want"?
But when you decide that YOUR violation is MY problem to fix, then you are just being silly. And worse, annoying.
Let's all just agree to run our own networks the way we damned well please, as long as we are not hurting anyone else. We just have to define "omplaining to you about things I b0rk'ed by myself" as "hurting you". Which isn't a stretch, support costs money, and costing me money because you screwed up is definitely hurtful.
When you promise to provide a service to anyone who asks for it and then fail to, you impose costs on other people. Failing to resolve names that you claim you will resolve is just such a failure. It forces other people's resolvers to do extra work to get the information they need or they just can't get it.
This is, IMO, the type of cooperation failure that justifies blacklisting.
You are very, very confused. When you ask me to resolve a name, _I_ did not cost _you_ anything - just the opposite. This is true whether I send you an A record or not. The idea that you can force me to provide service for you without payment, contract, service in trade, etc., has not been true for a couple decades. The idea that I might, out of the goodness of my heart, provide services for others is still alive and well. But to expect it is only going to cause you all kinds of problems, even from the people who have goodness in their hearts. But hey, feel free to disagree and blacklist away. Your network, your decision. :) -- TTFN, patrick
On Wed, 08 Aug 2007 10:33:56 EDT, "Patrick W. Gilmore" said:
Paying $10 and registering a domain IN NOW WAY means I promised a bazillion people anything.
What happened to: "You can run your network however you want"?
You're totally welcome to run your own network backbone as IPv6-only or X.25/CLNP. However, if you actually want to talk to the outside world, a higher level of cooperation is required. The hard-to-answer question is where the "best" tradeoff of "however you want" and "industry standards and best practices" lies for a given provider, because the answer is of course different for each site.
The answer is simple- because they are supposed to be allowed. By disallowing them you are breaking the agreed upon rules for the protocol. Before long it becomes impossible to implement new features because you can't be sure if someone else hasn't broken something intentionally.
I don't really have a dog in this fight about TCP 53. It does seem to me that it's a bit black and white to treat the RFCs as religious texts. It's important to follow them wherever possible, but frankly they don't foresee the bulk of the future security issues that usually materialize. So if a feature of the RFC isn't working for you security-wise, I believe it's your call to break with it there. As someone else said, don't complain when it breaks other things as well however.
If you don't like the rules- then change the damned protocol. Stop just doing whatever you want and then complaining when other people disagree with you.
I think its possible to disagree without calling other folks stupid... Best Regards, Jason
Date: Tue, 7 Aug 2007 23:32:21 -0600 From: "Jason J. W. Williams" <williamsjj@digitar.com>
The answer is simple- because they are supposed to be allowed. By disallowing them you are breaking the agreed upon rules for the protocol. Before long it becomes impossible to implement new features because you can't be sure if someone else hasn't broken something intentionally.
I don't really have a dog in this fight about TCP 53. It does seem to me that it's a bit black and white to treat the RFCs as religious texts. It's important to follow them wherever possible, but frankly they don't foresee the bulk of the future security issues that usually materialize. So if a feature of the RFC isn't working for you security-wise, I believe it's your call to break with it there. As someone else said, don't complain when it breaks other things as well however.
It is worth noting that we are not talking about just RFCs here, but STD or "Internet Standards". RFCs are a variety of things, but when they become Internet Standards, they are supposed to be mandatory. That said, the STD makes opening TCP/53 non-mandatory as it is labeled as a "SHOULD", not a "MUST". Those blocking tcp/53 maybe stupid to do so, but they are only violating a strong recommendation and not a requirement. As is often pointed out, blocking port 53 will eventually almost certainly break something and I have yet to see a good argument for blocking TCP/53.
If you don't like the rules- then change the damned protocol. Stop just doing whatever you want and then complaining when other people disagree with you.
I think its possible to disagree without calling other folks stupid...
While the folks blocking or suggesting blocking TCP/53 may not be stupid, the act blocking it is. (Intelligent people do stupid things.) -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
On Tue, 7 Aug 2007, Kevin Oberman wrote:
This has been a pain for me for years. I have tried to reason with security people about this and, while they don't dispute my reasoning, they always end up saying that it is the "standard" practice and that, lacking any evidence of what it might be breaking, it will continue to be blocked. And I don't mean small companies, either. One of the biggest issues I have is with one of the countries largest government funded research labs.
Having worked on both sides of the fence, i.e. I was a card-carrying member of both ASIS and NFPA, I used grumbled about the kooky things sysadmins and programmers did in the name of "security" as much as I grumbled about the kooky things security folks did in the name of "security." Heck, if programmers only produced bug-free software and sysadmins kept only well configured systems, security people would have a lot less work to do. What are the industry best practices for keeping DNS servers secure? CERT publishes a document on securing DNS: <http://www.cert.org/archive/pdf/dns.pdf> NIST publishes a document on securing DNS: <http://csrc.nist.gov/fasp/FASPDocs/network-security/NISTSecuringDNS.htm> CMYRU publishes a document on securing DNS: <http://www.cymru.com/Documents/secure-bind-template.html> Microsoft publishes a document on securing DNS: <http://technet2.microsoft.com/WindowsServer/en/Library/0fe406eb-6ca2-4d95-9a18-aede7e931ca21033.mspx> IETF publishes a document on operational (including security) requirements for root DNS servers: <http://www.rfc-editor.org/rfc/rfc2870.txt> While there is a lot in common, they each also have variations and omissions. Especially when it comes to some possibly obscure interactions with many different protocols and applications. The relationships between IP, ICMP, TCP, UDP and DNS seems to be tough for many people to get right. When you add undocumented "common knowledge" and other applications leveraging DNS for all sorts of stuff besides name/address resolution, its the typical programmer generated pile of spaghetti. Its often simplier to wait for something to break before you fix it. I know many sysadmins, programmers and even security people, that use that philosphy to decide which things to work on today. The good thing about security folks (and their cousins, the auditors) is most are compliance driven. So if you get a new Industry Best Practice, often they will become your friend enforcing whatever that says. So what should the Industry Best Practice(s) for DNS servers (root, authoritative and recursive) be? And what should it say about the interaction between IP/ICMP and TCP/UDP? And maybe we'll even get G-Root to follow it.
I can add one more voice to the chorus, not that it will necessarily change anyone's mind. :) When I was at Yahoo! the question of whether to keep TCP open or not had already been settled, since they had found that if they didn't have it open there was some small percentage of users who could not reach them. Given the large total number of {users|dns requests}/day even a small percentage was too much to sacrifice. In addition to that, it was already well established policy that all RR sets should be kept under the 512 byte limit. I took this a step further and worked (together with others) on a patch to restrict the size of DNS answers to < 512 by returning a random selection of any RR set larger than that. Even with all of those precautions, I still measured a non-trivial amount of TCP traffic to our name servers, most of which was for valid requests. BTW, one of the things that a lot of people don't take into account in this little equation is the fact that the size of the QUERY will affect the size of the response. So, given this experience, my conclusions (for whatever they are worth) are: 1. You can restrict 53/TCP on an authoritative name server if you want to, but you will lose traffic because of it. 2. Whether this is an acceptable loss or not is a local policy decision, but you should understand the consequences before you act. 3. No matter what your policy is, you cannot guarantee that employees will never make a mistake and create an RR set larger than 512 bytes. 4. You cannot control the behavior of client software out in the world, no matter how much you rant about it. Others have already brought up the issues of DNSSEC, IPv6, etc. so I won't belabor how important having working TCP _and_ EDNS0 is going to be down the road. And last but not least, the yang of "My network, my rules" has a yin to balance it out, "Be liberal in what you accept ...." hth, Doug -- If you're never wrong, you're not trying hard enough
dougb@dougbarton.us (Doug Barton) writes:
... I took this a step further and worked (together with others) on a patch to restrict the size of DNS answers to < 512 by returning a random selection of any RR set larger than that.
note that this sounds like a DNS protocol violation, and usually is. every time someone sent me a BIND patch adding this kind of deliberate instability (see RFC 1794 for an example) i said "no". -- Paul Vixie
Followups probably should go to the dnsops mailing list. I got tired of things and went back to the original question, and put together my list of what the "minimum" packets needed for full DNS performance on the modern Internet. It is the minimum, based on the security principle deny everything, allow only what is needed. But "needed" is performance based. So it means not relying on fallbacks, timeouts or hoping no one complains. It does not include packets needed for diagnostic or troubleshooting information. It is based on the "modern" Internet so does not included very deprecated packets like Source Quench or unimplemented functions like broadcast DNS queries. It does include current Internet practices for EDNS, Notify, global DNS load balancers and error handling I've seen in recent, i.e. less than 10 years old, DNS, Router and OS software. I didn't included TOS/DSCP and some military options, mainly because I'm not sure what "modern" military networks are currently using. If you are using TOS/DSCP or military options, there are some things you will need to add. <http://www.donelan.com/dnsacl.html> <http://www.donelan.com/dnsacl-min-cisco.html>
On Tue, 07 Aug 2007 14:38:06 EDT, "Patrick W. Gilmore" said:
In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems.
Since we are talking about authorities here, one can control the size of ones responses.
Barely. % dig aol.com txt ; <<>> DiG 9.5.0a6 <<>> aol.com txt ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57320 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 2 ;; QUESTION SECTION: ;aol.com. IN TXT ;; ANSWER SECTION: aol.com. 218 IN TXT "v=spf1 ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/23 ip4:205.188.159.0/24 ip4:64.12.136.0/23 ip4:64.12.138.0/24 ptr:mx.aol.com ?all" aol.com. 218 IN TXT "spf2.0/pra ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/23 ip4:205.188.159.0/24 ip4:64.12.136.0/23 ip4:64.12.138.0/24 ptr:mx.aol.com ?all" ;; AUTHORITY SECTION: aol.com. 439 IN NS dns-02.ns.aol.com. aol.com. 439 IN NS dns-06.ns.aol.com. aol.com. 439 IN NS dns-07.ns.aol.com. aol.com. 439 IN NS dns-01.ns.aol.com. ;; ADDITIONAL SECTION: dns-02.ns.aol.com. 158598 IN A 205.188.157.232 dns-06.ns.aol.com. 158598 IN A 149.174.54.153 ;; Query time: 4 msec ;; SERVER: 2001:468:c80:6101:213:72ff:fefc:d5cc#53(2001:468:c80:6101:213:72ff:fefc:d5cc) ;; WHEN: Tue Aug 7 15:32:36 2007 ;; MSG SIZE rcvd: 512 So tell me what they should do if they wanted to add 1 more byte to those TXT records. Say they want to start announcing IPv6 addresses too... :) They're running just a bit tight on 'authority and 'additional' they can heave over the side - they *already* don't answer with the txt records if you try to do a 'dig aol.com any' because that 512 and the 497 returned on a 'dig aol.com mx' won't fit in one 512-byte packet.
Unless, of course, you are so incredibly stupid you can't figure out the difference between an authority and a caching server.
I wish people would keep straight what direction they're doing the measurement, and for who's benefit. If *XYZ* wants to find which of their servers I'm closest to, they'll most likely be poking at my *caching* nameservers, because that's where my recursive query arrived from[1]. So we're *not* talking about authorities here. We're talking about DNS servers that are quite possibly configured to not talk, or give only partial results via UDP, to queries coming from outside the provider's network (after all, those people probably *should* be using *their* provider's caching DNS, right?) [1] If they *want* to go to the added effort of digging up my PTR, and from that finding the SOA and NS and ask my *authoritative* DNS for timing info, that's their right. Of course, all 3 of my caching servers are *really close* to me netwise - but of the 5 NS entries we list, 3 are intentionally *far away*, being off-campus secondaries. Hell, they're not even in our AS. :)
On Aug 7, 2007, at 3:45 PM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 07 Aug 2007 14:38:06 EDT, "Patrick W. Gilmore" said:
In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems.
Since we are talking about authorities here, one can control the size of ones responses.
Barely.
[SNIP] The point is, if you are the authority, you know how big the packet is. If you know it ain't over 512, then you don't need TCP. Or are you saying you do? Wouldn't it be 'incredibly stupid' for recursive servers to -require- TCP, even for < 512 byte packets?
Unless, of course, you are so incredibly stupid you can't figure out the difference between an authority and a caching server.
I wish people would keep straight what direction they're doing the measurement, and for who's benefit.
If *XYZ* wants to find which of their servers I'm closest to, they'll most likely be poking at my *caching* nameservers, because that's where my recursive query arrived from[1].
So we're *not* talking about authorities here. We're talking about DNS servers that are quite possibly configured to not talk, or give only partial results via UDP, to queries coming from outside the provider's network (after all, those people probably *should* be using *their* provider's caching DNS, right?)
Interesting. You are suggesting that as a content provider, one should rely on measurements from random caching name servers around the Internet, many of which you admit yourself are configured not to respond to addresses outside their network? Pardon me for not considering an idea you admit yourself wouldn't work. But you are right, I totally missed that part of the conversation. Mea Culpa. And in case anyone wasn't clear, yes, of course, running a recursive server that doesn't accept TCP53 will probably result in missing data your users want occasionally. As for being "incredibly stupid", well, as I have said in private, calling a bunch of people rude names without even asking them why they are doing what you think is so stupid is .. uh .. probably not very bright. :) Unless, of course, you want everyone else passing judgement on how you run your network without asking. -- TTFN, patrick
As for being "incredibly stupid", well, as I have said in private, calling a bunch of people rude names without even asking them why they are doing what you think is so stupid is .. uh .. probably not very bright. :) Unless, of course, you want everyone else passing judgement on how you run your network without asking. Breaking the agreed upon rules of a protocol is stupid. Period.
It has nothing to do with judging how one runs their network or any other such nonsense. The RFC's say TCP 53 is fine. If you don't want to follow the rules, fine, but have the temerity to admit that it is stupid. -Don
On Tue, 7 Aug 2007, Donald Stahl wrote:
It has nothing to do with judging how one runs their network or any other such nonsense. The RFC's say TCP 53 is fine. If you don't want to follow the rules, fine, but have the temerity to admit that it is stupid.
I don't want to wade into this particular argument, which doesn't seem to be going anywhere useful. But I think the style of the argument causes some problems that trickle into network operations, and should be addressed. The problem with this argument is that, while it may be entirely correct, it's unlikely to convince the people who matter. The people who matter are the people who write the checks for the networks we work on. Successful managers (and successful engineers) generally get pretty good at doing cost benefit analyses. Since there are many decisions where there isn't one obvious answer, they learn instead to think in terms of each choice providing some benefits and having some costs, and doing the things where the benefits outweigh the costs. In the firewall case, as Kevin said, there are probably people going to the decision makers and talking about the importance of keeping things closed up. Every open firewall rule, they'll say, creates the potential for an attack. Any attack could cause down time, unauthorized sharing of confidential data, loss of files people have spent the last several years working on, and more. Therefore, the cost of an open firewall rule could potentially be millions of dollars. The value of any service enabled by a hole in the firewall had better be more than that. Is this argument valid? Maybe not. But the money people who make the decisions probably don't have the technical expertise to analyse it. Even if they suspect that the case for the policy is overstated, they'll associate some cost with ignoring the advice of their security people, as they probably should. So, what's somebody who objects to such an argument to do? You could go to management and say, "the security people are wrong. The standard says we must open more ports. To not do so would be wrong." But you may not like the choice this presents management with. On one side, they've got you telling them to follow an arbitrary standard, because not doing so would be wrong. On the other side, they're being told that taking your advice could cost millions of dollars. Losing millions of dollars as a result of a refusal to heed warnings would probably get them fired, or worse. Pointing at an arbitrary standard after things had gone wrong probably wouldn't get them very far. Alternatively, you too could start speaking their cost benefit language. You could assail the security peoples' cost figures, although at that point you'd be asking them to distrust other employees and they might wonder if they should distrust you instead. Or you could point out the costs of leaving the port closed, or possible benefits of leaving it open. If you can tell them that some fraction of their customers aren't able to get to them because of the closed port, and that those would be customers represent some large amount of revenue, you'll show that there's actual benefit to having the port open. If that benefit is greater than the potential loss they're being told about, you might actually win the argument. If you have some evidence to back up your numbers, you may have more credibility, and be able to win the argument with lower numbers. Or, you may find that you're not as right as you thought you were. You may find that what you were advocating doesn't seem to have any concrete benefit, and that what the other side was saying has some merit. That may not happen in this case, but sooner or later you'll probably find one where it does. -Steve
Dear colleagues, I apologise for replying twice in the same thread (especially as I tend not to post here very much, on the grounds that I usually don't know what I'm talking about). I feel compelled to object to the below remark, however, because I think it gets at the heart of the problem. On Tue, Aug 07, 2007 at 03:09:58PM -0700, Steve Gibbard wrote:
But you may not like the choice this presents management with. On one side, they've got you telling them to follow an arbitrary standard,
I generally agree with Steve Gibbard's point, which I take to be that understanding the cost-benefit realm in which these discussions happen is both crucial to achieving one's result and may reveal a point of view one hasn't properly considered. I nevertheless object to the suggestion (that I think was not actually part of Steve's main argument, please note) that we are talking about some "arbitrary standard". The RFCs that define DNS are of course arbitrary in the strict sense that they could have been otherwise: RFC103[45] could have said, "512 is the limit, sorry, can't help you, haveanicedaycomeagain." They're arbitrary in the sense that, for instance, the definition of ANSI C is; or that "hook" versus "arrow" for entailment in various formal logic systems is. But that's not the interesting meaning of "arbitrary" in this case. The connotation of "arbitrary" in these discussions is that this is a rule that isn't strictly needed. But the fact of the matter, on the Internet, is that if you don't follow the "arbitrary" standards for a protocol as defined in the RFCs, then you're _not implementing the protocol_. That's what a protocol _is_: a set of arbitrary rules that define how various strangers can implement systems that all comply, without having to talk about it individually. If you try to put 'zMttOOOPS' into a SQL database field defined as INT4, you get an error: it's an arbitrary rule, but one that defines the field. And if you try to turn off TCP for DNS, you get an error too. It's just that you're not the one who happens to see it. This is not some bizarre demand on the part of Internet weenies, demanding that your network comply with their rules. It's just straightforward implementation. As operators, I think we have an obligation to be clear in our representation to our various management: there are things that are required to participate in the Internet as a compliant system. If one rejects those things, then one is not really participating. We are each free to make such a decision; but where a protocol says "TCP and UDP", one doesn't get to make up a rule that says, "Yeah, but not for us." That way lies the end of interoperation. If you don't want inter-networking, then it will work fine. But if you want the benefits, you have to pay the cost of complying with the rules, even when you don't understand or care how they affect you or everybody else. Best regards, A ---- Andrew Sullivan 204-4141 Yonge Street Afilias Canada Toronto, Ontario Canada <andrew@ca.afilias.info> M2P 2A8 +1 416 646 3304 x4110
Forgive my broken formatting, but LookOut, it's Microsoft! Is what we use, period. I have a question related to what you posted below, and it's a pretty simple one: How is answering a query on TCP/53 any MORE dangerous than answering it on UDP/53? Really. I'd like to know how one of these security nitwits justifies it. It's the SAME piece of software answering the query either way. Jamie Bowden -- "It was half way to Rivendell when the drugs began to take hold" Hunter S Tolkien "Fear and Loathing in Barad Dur" Iain Bowen <alaric@alaric.org.uk> -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Steve Gibbard Sent: Tuesday, August 07, 2007 6:10 PM To: Nanog Subject: Re: large organization nameservers sending icmp packets to dns servers. On Tue, 7 Aug 2007, Donald Stahl wrote:
It has nothing to do with judging how one runs their network or any other such nonsense. The RFC's say TCP 53 is fine. If you don't want to follow the rules, fine, but have the temerity to admit that it is stupid.
I don't want to wade into this particular argument, which doesn't seem to be going anywhere useful. But I think the style of the argument causes some problems that trickle into network operations, and should be addressed. The problem with this argument is that, while it may be entirely correct, it's unlikely to convince the people who matter. The people who matter are the people who write the checks for the networks we work on. Successful managers (and successful engineers) generally get pretty good at doing cost benefit analyses. Since there are many decisions where there isn't one obvious answer, they learn instead to think in terms of each choice providing some benefits and having some costs, and doing the things where the benefits outweigh the costs. In the firewall case, as Kevin said, there are probably people going to the decision makers and talking about the importance of keeping things closed up. Every open firewall rule, they'll say, creates the potential for an attack. Any attack could cause down time, unauthorized sharing of confidential data, loss of files people have spent the last several years working on, and more. Therefore, the cost of an open firewall rule could potentially be millions of dollars. The value of any service enabled by a hole in the firewall had better be more than that. Is this argument valid? Maybe not. But the money people who make the decisions probably don't have the technical expertise to analyse it. Even if they suspect that the case for the policy is overstated, they'll associate some cost with ignoring the advice of their security people, as they probably should. So, what's somebody who objects to such an argument to do? You could go to management and say, "the security people are wrong. The standard says we must open more ports. To not do so would be wrong." But you may not like the choice this presents management with. On one side, they've got you telling them to follow an arbitrary standard, because not doing so would be wrong. On the other side, they're being told that taking your advice could cost millions of dollars. Losing millions of dollars as a result of a refusal to heed warnings would probably get them fired, or worse. Pointing at an arbitrary standard after things had gone wrong probably wouldn't get them very far. Alternatively, you too could start speaking their cost benefit language. You could assail the security peoples' cost figures, although at that point you'd be asking them to distrust other employees and they might wonder if they should distrust you instead. Or you could point out the costs of leaving the port closed, or possible benefits of leaving it open. If you can tell them that some fraction of their customers aren't able to get to them because of the closed port, and that those would be customers represent some large amount of revenue, you'll show that there's actual benefit to having the port open. If that benefit is greater than the potential loss they're being told about, you might actually win the argument. If you have some evidence to back up your numbers, you may have more credibility, and be able to win the argument with lower numbers. Or, you may find that you're not as right as you thought you were. You may find that what you were advocating doesn't seem to have any concrete benefit, and that what the other side was saying has some merit. That may not happen in this case, but sooner or later you'll probably find one where it does. -Steve
On Wed, Aug 08, 2007, Jamie Bowden wrote:
Forgive my broken formatting, but LookOut, it's Microsoft! Is what we use, period.
I have a question related to what you posted below, and it's a pretty simple one:
How is answering a query on TCP/53 any MORE dangerous than answering it on UDP/53? Really. I'd like to know how one of these security nitwits justifies it. It's the SAME piece of software answering the query either way.
I'd hazard a guess and say something like "TCP state complexity > UDP state complexity" and that possibly leading to a potential DoS. But then, there's also stuff like stateful firewalls which can more aggressively timeout UDP flows (and not break DNS ones, since they're not exactly long-living!) but die under large TCP loads. And TCP takes CPU to setup/teardown, and requires client-side state. Adrian
On 8-Aug-2007, at 11:59, Jamie Bowden wrote:
I have a question related to what you posted below, and it's a pretty simple one:
How is answering a query on TCP/53 any MORE dangerous than answering it on UDP/53? Really. I'd like to know how one of these security nitwits justifies it. It's the SAME piece of software answering the query either way.
There are people (I believe; this is a little rumour-laden) who take the approach that 53/tcp is actually safer than 53/udp, since the handshake makes it easier to believe the query's source address. The approach I heard about was to reply to UDP-transport queries with some minimal answer set with TC set, and serve a more useful set of information over TCP once the re-query arrives. [I realise that the state involved in handing TCP queries on a busy server is non-trivial, and that there are many aspects to this approach which deserve raised eyebrows.] However, my point is that "TCP is more secure than UDP" also has a posse. Joe
On Aug 8, 2007, at 8:59 AM, Jamie Bowden wrote:
How is answering a query on TCP/53 any MORE dangerous than answering it on UDP/53? Really. I'd like to know how one of these security nitwits justifies it. It's the SAME piece of software answering the query either way.
How many bytes of shell code can you stuff in a 512 byte DNS UDP packet? How many bytes of shell code can you stuff in a TCP DNS connection? Rgds, -drc P.S. I still think blocking TCP/53 is stupid.
On Wed, 8 Aug 2007, David Conrad wrote:
How many bytes of shell code can you stuff in a 512 byte DNS UDP packet?
How many bytes of shell code can you stuff into a 4096 byte EDNS0 UDP packet? :)
P.S. I still think blocking TCP/53 is stupid.
Agreed. -- If you're never wrong, you're not trying hard enough
On Thu, 9 Aug 2007 15:53:12 -0700 (PDT) Doug Barton <dougb@dougbarton.us> wrote:
How many bytes of shell code can you stuff into a 4096 byte EDNS0 UDP packet? :)
Probably a lot. People used to have 4-line signatures with the PGP encryption or DECSS. I have a 152-byte C program that calculates 32K digits of PI. matthew black network services california state university, long beach
On Tue, 7 Aug 2007, Donald Stahl wrote:
As for being "incredibly stupid", well, as I have said in private, calling a bunch of people rude names without even asking them why they are doing what you think is so stupid is .. uh .. probably not very bright. :) Unless, of course, you want everyone else passing judgement on how you run your network without asking. Breaking the agreed upon rules of a protocol is stupid. Period.
actually people break rules all the time, they do it as part of a risk/cost/reward balance. If they decide that blocking port X but not port Y is 'ok' for them who are you to say beyond: "Wow, the blah blah RFC says foo-bar, why would you do what you did?" Some folks decide to block tcp/53 to their nameservers, some don't. it's not stupid, it maybe unwise if they don't know what complications they are setting themselves up for... Similarly answering a different A for each client based on their location and your feelings about them could be considered 'dangerous' or 'concerning' unless you understood what complications that might induce.
It has nothing to do with judging how one runs their network or any other such nonsense. The RFC's say TCP 53 is fine. If you don't want to follow
RFC's say many things, some might be unwise given your view of the world, some may be peachy... It's all about what risk you are willing to take, or that's what it seems like to me :) -Chris
The point is, if you are the authority, you know how big the packet is. If you know it ain't over 512, then you don't need TCP.
Or are you saying you do? Wouldn't it be 'incredibly stupid' for recursive servers to -require- TCP, even for < 512 byte packets?
A TCP query is just as valid as a UDP query. If you claim to provide DNS for a zone but fail to respond to valid queries, you are breaking your promise. It's not whether or not you need TCP. It's that if you promise to provide a service, you should in fact provide that service. DS
On Tue, 07 Aug 2007 16:10:17 EDT, "Patrick W. Gilmore" said:
The point is, if you are the authority, you know how big the packet is. If you know it ain't over 512, then you don't need TCP.
Right. But remember the discussion is that *we* (for some value of "we") are querying some *other* nameserver, and we *don't* know a priori how big the packet will be, until they send us a packet with the truncate bit set, and then we get to find out if their config is sane....
Interesting. You are suggesting that as a content provider, one should rely on measurements from random caching name servers around the Internet, many of which you admit yourself are configured not to respond to addresses outside their network? Pardon me for not considering an idea you admit yourself wouldn't work.
Oddly enough, it *does* seem to work fairly well - given the number of content providers that are running global-scale load balancers that ping the source of queries to figure out how far they are. What I'm trying to figure out is how authoritative DNS servers enter into it in the first place, since the behavior as originally discussed was content providers who poke (presumably) caching servers.
On Tue, 7 Aug 2007, Valdis.Kletnieks@vt.edu wrote:
they *already* don't answer with the txt records if you try to do a 'dig aol.com any' because that 512 and the 497 returned on a 'dig aol.com mx' won't fit in one 512-byte packet.
Wrong! You're probably not getting the txt records because you don't have them in your cache. See the following four queries for an example. (First three from my cache, fourth from AOL.) ; <<>> DiG 9.3.1 <<>> aol.com any ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15075 ;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 4, ADDITIONAL: 4 ;; QUESTION SECTION: ;aol.com. IN ANY ;; ANSWER SECTION: aol.com. 1790 IN NS dns-06.ns.aol.com. aol.com. 1790 IN NS dns-07.ns.aol.com. aol.com. 1790 IN NS dns-01.ns.aol.com. aol.com. 1790 IN NS dns-02.ns.aol.com. aol.com. 2802 IN MX 15 mailin-04.mx.aol.com. aol.com. 2802 IN MX 15 mailin-01.mx.aol.com. aol.com. 2802 IN MX 15 mailin-02.mx.aol.com. aol.com. 2802 IN MX 15 mailin-03.mx.aol.com. ;; AUTHORITY SECTION: aol.com. 1790 IN NS dns-02.ns.aol.com. aol.com. 1790 IN NS dns-06.ns.aol.com. aol.com. 1790 IN NS dns-07.ns.aol.com. aol.com. 1790 IN NS dns-01.ns.aol.com. ;; ADDITIONAL SECTION: dns-01.ns.aol.com. 115155 IN A 64.12.51.132 dns-02.ns.aol.com. 115817 IN A 205.188.157.232 dns-06.ns.aol.com. 12385 IN A 149.174.54.153 dns-07.ns.aol.com. 116305 IN A 64.236.1.107 ;; Query time: 7 msec ;; SERVER: 131.111.8.42#53(131.111.8.42) ;; WHEN: Wed Aug 8 16:18:21 2007 ;; MSG SIZE rcvd: 339 ; <<>> DiG 9.3.1 <<>> aol.com. txt ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17924 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 2 ;; QUESTION SECTION: ;aol.com. IN TXT ;; ANSWER SECTION: aol.com. 300 IN TXT "v=spf1 ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/23 ip4:205.188.159.0/24 ip4:64.12.136.0/23 ip4:64.12.138.0/24 ptr:mx.aol.com ?all" aol.com. 300 IN TXT "spf2.0/pra ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/23 ip4:205.188.159.0/24 ip4:64.12.136.0/23 ip4:64.12.138.0/24 ptr:mx.aol.com ?all" ;; AUTHORITY SECTION: aol.com. 1679 IN NS dns-06.ns.aol.com. aol.com. 1679 IN NS dns-07.ns.aol.com. aol.com. 1679 IN NS dns-01.ns.aol.com. aol.com. 1679 IN NS dns-02.ns.aol.com. ;; ADDITIONAL SECTION: dns-01.ns.aol.com. 115044 IN A 64.12.51.132 dns-02.ns.aol.com. 115706 IN A 205.188.157.232 ;; Query time: 80 msec ;; SERVER: 131.111.8.42#53(131.111.8.42) ;; WHEN: Wed Aug 8 16:20:12 2007 ;; MSG SIZE rcvd: 512 ;; Truncated, retrying in TCP mode. ; <<>> DiG 9.3.1 <<>> aol.com. any ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1265 ;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 4, ADDITIONAL: 4 ;; QUESTION SECTION: ;aol.com. IN ANY ;; ANSWER SECTION: aol.com. 298 IN TXT "v=spf1 ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/23 ip4:205.188.159.0/24 ip4:64.12.136.0/23 ip4:64.12.138.0/24 ptr:mx.aol.com ?all" aol.com. 298 IN TXT "spf2.0/pra ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/23 ip4:205.188.159.0/24 ip4:64.12.136.0/23 ip4:64.12.138.0/24 ptr:mx.aol.com ?all" aol.com. 1677 IN NS dns-07.ns.aol.com. aol.com. 1677 IN NS dns-01.ns.aol.com. aol.com. 1677 IN NS dns-02.ns.aol.com. aol.com. 1677 IN NS dns-06.ns.aol.com. aol.com. 2689 IN MX 15 mailin-02.mx.aol.com. aol.com. 2689 IN MX 15 mailin-03.mx.aol.com. aol.com. 2689 IN MX 15 mailin-04.mx.aol.com. aol.com. 2689 IN MX 15 mailin-01.mx.aol.com. ;; AUTHORITY SECTION: aol.com. 1677 IN NS dns-06.ns.aol.com. aol.com. 1677 IN NS dns-07.ns.aol.com. aol.com. 1677 IN NS dns-01.ns.aol.com. aol.com. 1677 IN NS dns-02.ns.aol.com. ;; ADDITIONAL SECTION: dns-01.ns.aol.com. 115042 IN A 64.12.51.132 dns-02.ns.aol.com. 115704 IN A 205.188.157.232 dns-06.ns.aol.com. 12272 IN A 149.174.54.153 dns-07.ns.aol.com. 116192 IN A 64.236.1.107 ;; Query time: 1 msec ;; SERVER: 131.111.8.42#53(131.111.8.42) ;; WHEN: Wed Aug 8 16:20:14 2007 ;; MSG SIZE rcvd: 707 ; <<>> DiG 9.3.1 <<>> any aol.com. @64.12.51.132 +notcp +bufsize=1024 ; (1 server found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45020 ;; flags: qr aa rd; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 20 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;aol.com. IN ANY ;; ANSWER SECTION: aol.com. 60 IN A 64.12.50.151 aol.com. 60 IN A 205.188.142.182 aol.com. 300 IN TXT "v=spf1 ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/23 ip4:205.188.159.0/24 ip4:64.12.136.0/23 ip4:64.12.138.0/24 ptr:mx.aol.com ?all" aol.com. 300 IN TXT "spf2.0/pra ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/23 ip4:205.188.159.0/24 ip4:64.12.136.0/23 ip4:64.12.138.0/24 ptr:mx.aol.com ?all" aol.com. 3600 IN MX 15 mailin-03.mx.aol.com. aol.com. 3600 IN MX 15 mailin-04.mx.aol.com. aol.com. 3600 IN MX 15 mailin-01.mx.aol.com. aol.com. 3600 IN MX 15 mailin-02.mx.aol.com. aol.com. 3600 IN NS dns-07.ns.aol.com. aol.com. 3600 IN NS dns-01.ns.aol.com. aol.com. 3600 IN NS dns-02.ns.aol.com. aol.com. 3600 IN NS dns-06.ns.aol.com. aol.com. 3600 IN SOA dns-01.ns.aol.com. hostmaster.aol.net. 2007080800 1800 300 604800 600 ;; ADDITIONAL SECTION: mailin-01.mx.aol.com. 300 IN A 205.188.158.121 mailin-01.mx.aol.com. 300 IN A 205.188.159.57 mailin-01.mx.aol.com. 300 IN A 64.12.137.184 mailin-01.mx.aol.com. 300 IN A 64.12.137.249 mailin-02.mx.aol.com. 300 IN A 205.188.155.89 mailin-02.mx.aol.com. 300 IN A 205.188.157.25 mailin-02.mx.aol.com. 300 IN A 64.12.137.89 mailin-02.mx.aol.com. 300 IN A 64.12.137.168 mailin-03.mx.aol.com. 300 IN A 205.188.157.217 mailin-03.mx.aol.com. 300 IN A 64.12.138.120 mailin-03.mx.aol.com. 300 IN A 64.12.138.153 mailin-04.mx.aol.com. 300 IN A 64.12.138.57 mailin-04.mx.aol.com. 300 IN A 64.12.138.88 mailin-04.mx.aol.com. 300 IN A 64.12.139.249 mailin-04.mx.aol.com. 300 IN A 205.188.159.216 dns-01.ns.aol.com. 3600 IN A 64.12.51.132 dns-02.ns.aol.com. 3600 IN A 205.188.157.232 dns-06.ns.aol.com. 3600 IN A 149.174.54.153 dns-07.ns.aol.com. 3600 IN A 64.236.1.107 ;; Query time: 83 msec ;; SERVER: 64.12.51.132#53(64.12.51.132) ;; WHEN: Wed Aug 8 16:24:53 2007 ;; MSG SIZE rcvd: 988 Tony. -- f.a.n.finch <dot@dotat.at> http://dotat.at/ IRISH SEA: SOUTHERLY, BACKING NORTHEASTERLY FOR A TIME, 3 OR 4. SLIGHT OR MODERATE. SHOWERS. MODERATE OR GOOD, OCCASIONALLY POOR.
On Tue, 7 Aug 2007, Donald Stahl wrote:
All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons... Then most are incredibly stupid.
Several anti DoS utilities force unknown hosts to initiate a query via TCP in order to be whitelisted. If the host can't perform a TCP query then they get blacklisted.
How is that an "anti DoS" technique when you actually need to return an answer via UDP in order to force next request via TCP? Or is this techinque based on premise that an attacker will not spoof packets and thus will send flood of DNS requests to server from same IP (set of ips)? If so the result would be that attacker could in fact use TCP just as well as UDP. -- William Leibzon Elan Networks william@elan.net
On Aug 8, 2007, at 6:20 PM, "william(at)elan.net" <william@elan.net> wrote:
On Tue, 7 Aug 2007, Donald Stahl wrote:
All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons... Then most are incredibly stupid.
Several anti DoS utilities force unknown hosts to initiate a query via TCP in order to be whitelisted. If the host can't perform a TCP query then they get blacklisted.
How is that an "anti DoS" technique when you actually need to return an answer via UDP in order to force next request via TCP? Or is this techinque based on premise that an attacker will not spoof packets and thus will send flood of DNS requests to server from same IP (set of ips)? If so the result would be that attacker could in fact use TCP just as well as UDP.
The anti-ddos box sends back a UDP reply with the TCP bit sent and no data. Which, I believe, violates the RFC. (But it is too hard to look up on my iPhone. :) If so, guess that makes those boxes 'stupid'. -- TTFN, patrick
On Wed, Aug 08, 2007 at 03:20:56PM -0700, william(at)elan.net <william@elan.net> wrote a message of 23 lines which said:
How is that an "anti DoS" technique when you actually need to return an answer via UDP in order to force next request via TCP?
Because there is no amplification: the UDP response packet can be very small.
On Thu, 9 Aug 2007, Stephane Bortzmeyer wrote:
On Wed, Aug 08, 2007 at 03:20:56PM -0700, william(at)elan.net <william@elan.net> wrote a message of 23 lines which said:
How is that an "anti DoS" technique when you actually need to return an answer via UDP in order to force next request via TCP?
Because there is no amplification: the UDP response packet can be very small.
actually because it forces authentication of the source (authentication being that the source is a real-live host asking for dns services). Beyond that trick, the deviecs I've seen/used also catalog the rates of queries from individual hosts and force a cached answer to be generated locally if the loads get too high (per source).. Sorry this is a bit late to the punch :)
On Mon, 6 Aug 2007, John Levine said:
Why would they ping rather than just sending the query to all of the NS and see which one answers first? It's an IP round trip either way.
ping targets are of the caching nameserver type, not the authoritative nameserver type. Duane W.
On Aug 6, 2007, at 10:21 AM, John Levine wrote:
Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest.
Why would they ping rather than just sending the query to all of the NS and see which one answers first? It's an IP round trip either way.
I agree that pinging is harmless, but for this application it seems pointless, too.
Well... we're talking about recursive resolvers. There's not really a simple way for a third party to measure the round trip time to the recursive resolver at the dns level. It may not respond to external queries at all, and even if it does, what query would you send that would cause an immediate reply without any additional processing or network latency at the resolver? There's lots of tricks you can play to do this, but most of them are no better than a simple ICMP ping. Cheers, Steve
On Mon, 06 Aug 2007 12:13:03 EDT, "Steven M. Bellovin" said:
1) ICMP is handled at the same rate as TCP/UDP packets in all the routers involved (so there's no danger of declaring a path "slow" when it really isn't, just becase a router slow-pathed ICMP).
This is aimed at hosts, not routers, right? As far as I know, routers don't slow-path forwarded ICMP. Hosts will probably reply to ICMP from their kernel, so it's a faster response than a user-level DNS reply.
Well, they don't *directly* slow-path it. But we've seen *plenty* of cases of "multi-hop performance as indicated by ICMP Echo Request/Reply doesn't at all match throughput/latency as indicated by TCP-level stats" mentioned on this list...
But why would they care where the nameserver is? Point 2 would seem to be a little stupid a thing to assume. Also, what happens if, at that moment, the ICMP packet is stuck in a queue for a few ms making the shortest route longer. -- Leigh Valdis.Kletnieks@vt.edu wrote:
On Mon, 06 Aug 2007 11:53:15 EDT, Drew Weaver said:
Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why?
Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest.
Yes, it's a semi-borkken strategy, because it assumes that:
1) ICMP is handled at the same rate as TCP/UDP packets in all the routers involved (so there's no danger of declaring a path "slow" when it really isn't, just becase a router slow-pathed ICMP).
2) That the actual requester of service is reasonably near net-wise to the server handling the end-user's recursive DNS lookup.
On Aug 6, 2007, at 9:13 AM, Leigh Porter wrote:
But why would they care where the nameserver is? Point 2 would seem to be a little stupid a thing to assume. Also, what happens if, at that moment, the ICMP packet is stuck in a queue for a few ms making the shortest route longer.
While point 2 is a bad assertion if you depend completely upon it, it's not necessarily a bad starting point if you have no other data to go on. 1. 90+% of resolvers are topologically proximate to either the requestor, or, the requestors NAT box that you will have to talk to anyway. 2. At the GLB level, you really don't have any data other than the IP address of the resolver upon which to base your GLB decision. Since you'll be right 90+% of the time, and, only sub-optimal, not broken the other <10% of the time, it generally works OK. 3. When I worked for Netli, before they were acquired in what I would call a much less than ethical transaction, we maintained an exception table for cases where we learned that the DNS resolver was not topologically proximate to the requestors that flowed through it. We also spent a fair amount of time explaining the benefits of having the resolver be topologically proximate to our customers and their customers. The Netli system was designed to be quite gentle in the amount of probing it did, but, we did occasionally get messages from people with paranoid IDS boxes. Usually, once we explained that our efforts were directed at improving the quality of service to their users, and how the system worked and how little traffic we sent their way to accomplish this, they were happy to reconfigure their alarm preferences. I don't have first hand knowledge of anyone elses use of these kinds of ICMP probes, but, I would say that generally, they are somewhat useful and mostly harmless. Owen
On Monday 06 August 2007 16:53, Drew Weaver wrote:
Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why?
Some of the DNS load balancing schemes do this, I assume to work out how far away your server is so they can give geographically relevant answers. If you are geographically close to your recursive name server, it might even work.
And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization?
I tend to favour filtering some types of ICMP packets and not others, the packets required for ping hold little fear for me (and are kind of useful), but YMMV. My ICMP filtering experience is not DNS specific, you might be able to do better with DNS server specific rule, but that is too much like micromanagement for me, others may know a lot more on this.
Drew Weaver wrote:
Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization?
Wearing my Mozilla hat here... I blogged about this (blog.mozilla.com/mrz, somewhere there) and Asa blog'd about it over at http://weblogs.mozillazine.org/asa/archives/2007/03/trying_to_load.html . Mozilla uses Citrix Netscalers and we're currently using dynamic proximity for load balancing between data centers. After Asa's post, we found poorly documentation that led to misconfiguration of the probe settings. I've cut down the number of probes (default was icmp, udp and tcp:80 to a nameserver) and instead of the ~10 complaints a day I was getting, I get many one a month. If you're still annoyed by the probes, ping me off-list. - mz
On Mon, 6 Aug 2007, Drew Weaver wrote:
Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization?
They use ICMP/Echo Request to calculate a rough surrogate latency estimate for potential users of that caching DNS server so they can return different DNS answers depending on your network topology. Yes its an approximation, and can be wrong. Some networks even re-route ICMP/Echo to a completely different box which just responsed to pings; so it may not even go to the same place. Given all those caveats, many times its still the best guess you can make. ICMP/ECHO is a separate protocol which is easy to filter if you want to, without affecting "normal" TCP/UDP/etc packets. But then expect to get "worse" default DNS answers from those same sites attempting to optimize their DNS answers. It would be cool if people ran NTP port 123 on their DNS servers, and then we could get extreme measurements. But then I'm sure someone would point out 62 flaws with that. In the end, its up to each network operator to make its own decision. If your DNS servers aren't being negatively impacted, and it helps your users get better answers, you might keep it. If the answers are reversed, you might drop them. My IDS is badly tuned.... Well maybe there is a fix for that.
On Mon, 6 Aug 2007, Drew Weaver wrote:
Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization?
If someone else mentioned this I didn't see it, so I think it's worthwhile to point out that in the case of Yahoo! at least the pinging isn't done by nsX.yahoo.com, but rather it's done by Akamai after you get to the point in the chain where the "real" answer is going to come from one of their servers. Someone else already did point out that in spite of the existence of outliers, the shape of the bell-shaped curve remains the same, and if this method didn't actually work then it wouldn't be in such widespread use. hth, Doug -- If you're never wrong, you're not trying hard enough
participants (35)
-
Adrian Chadd -
Andrew Sullivan -
Chris L. Morrow -
David Conrad -
David Schwartz -
Donald Stahl -
Doug Barton -
Douglas Otis -
Drew Weaver -
Duane Wessels -
Jamie Bowden -
Jason J. W. Williams -
Joe Abley -
John Kristoff -
John L -
John Levine
-
Kevin Oberman -
Leigh Porter -
Matthew Black -
matthew zeier -
Owen DeLong -
Patrick W. Gilmore -
Paul Vixie -
Paul Vixie -
Peter Dambier -
Roland Dobbins -
Sean Donelan -
Simon Waters -
Stephane Bortzmeyer -
Steve Atkins -
Steve Gibbard -
Steven M. Bellovin -
Tony Finch -
Valdis.Kletnieks＠vt.edu -
william(at)elan.net