Hello, has someone experience in anycast ipv4 networks (to support DNS)? Regards Lutz
InterNetX - Lutz Muehlig wrote:
Hello,
has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc. I assume quite a number of people know how to do it, especially as several root DNS servers abuse it. Simple recipe: - Box with: - Your favourite OS - Quagga or OpenBGPd - Your favourite DNS server - Announce the IP of the anycast node in BGP - Monitor the DNS server, when it does not work kill your local BGPd and notify the admins that it broke That is it. Probably with the above couple of things, google a bit and find the rest. Greets, Jeroen
has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc.
- Yes but as for DNS, anycast is essentially used for user requests (UDP) not to perform zone transfer(TCP).
How-to with working configurations for Linux+Quagga: <www.linuxsa.org.au/meetings/2006-07/anycast-dns.odt> Further information is found in RFC3258: <http://tools.ietf.org/html/rfc3258> Remember to disable PMTU discovery on your DNS servers if you are using this. Paul
On Fri, 26 Mar 2010 09:40:39 EDT, Max Larson Henry said:
- Yes but as for DNS, anycast is essentially used for user requests (UDP) not to perform zone transfer(TCP).
DNS uses TCP for more than just XFR. For instance, if you're running a resolver that doesn't do EDNS0, and you hit an (increasingly common) DNSSEC signed reply, it's going to be over 512 bytes and the lack of EDNS0 will cause it to re-ask via TCP. Just mentioning it because the sort of sites that think TCP==XFR are the sort most likely to be running firewalls that munch the EDNS0 bits, and are setting themselves up for big surprises in the very near future.
In message <4828.1269611568@localhost>, Valdis.Kletnieks@vt.edu writes:
--==_Exmh_1269611568_4209P Content-Type: text/plain; charset=us-ascii
On Fri, 26 Mar 2010 09:40:39 EDT, Max Larson Henry said:
- Yes but as for DNS, anycast is essentially used for user requests (UDP) not to perform zone transfer(TCP).
DNS uses TCP for more than just XFR. For instance, if you're running a resolver that doesn't do EDNS0, and you hit an (increasingly common) DNSSEC signed reply, it's going to be over 512 bytes and the lack of EDNS0 will cause it to re-ask via TCP.
DNSSEC depends on EDNS and DO being set in the EDNS OPT record, so won't get DNSSEC records, except in response to * queries, for non EDNS queries.
Just mentioning it because the sort of sites that think TCP==XFR are the sort most likely to be running firewalls that munch the EDNS0 bits, and are setting themselves up for big surprises in the very near future.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Max Larson Henry wrote:
> has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc.
- Yes but as for DNS, anycast is essentially used for user requests (UDP) not to perform zone transfer(TCP).
Also that would work, unless you have a very unstable routing table that makes the node swap all the time. Please also note that if a DNS answer does not fit inside a a UDP packet (default 512, MTU with EDNS0) that the fallback is TCP mode... John Payne wrote: [..]
Can't really tell if you're being serious here due to caffeine underrun.
As it is already almost 15:00 in Europe (and it is a friday), take a guess ;) Also note the next line I wrote and the point to the google, which you now have done for the person, who probably is also having a lazy friday afternoon ;) Greets, Jeroen
On Mar 26, 2010, at 6:55 AM, Jeroen Massar wrote:
Max Larson Henry wrote:
has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc.
- Yes but as for DNS, anycast is essentially used for user requests (UDP) not to perform zone transfer(TCP).
Also that would work, unless you have a very unstable routing table that makes the node swap all the time.
It doesn't require an unstable routing table. There is a small set of locations that could hit routers with multipath that may "balance" the anycast packets down divergent paths. Essentially, these are the topological midpoints between any two (or more) anycast servers. Owen
On 2010-03-26, at 10:04, Owen DeLong wrote:
It doesn't require an unstable routing table. There is a small set of locations that could hit routers with multipath that may "balance" the anycast packets down divergent paths.
Essentially, these are the topological midpoints between any two (or more) anycast servers.
As with most things, there are risks and benefits to distributing services using anycast. There are also risks and benefits from not doing so. Far too many words to bother people with here on this list, but the ;login: article I include a link to earlier (<http://www.usenix.org/publications/login/2008-02/openpdfs/abley.pdf>) attempts to discuss ways in which people can decide whether anycast suits their own particular circumstances ("Horses for Courses"). Joe
On Mar 26, 2010, at 6:40 AM, Max Larson Henry wrote:
has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc.
- Yes but as for DNS, anycast is essentially used for user requests (UDP) not to perform zone transfer(TCP).
-M
The number of DNS queries for user requests that are in UDP in IPv4 is nearly all. When you start seeing hosts with more than a couple of AAAA records, you will start seeing more user requests in TCP. FWIW, World of Warcraft authentication actually involves some hosts with enough A records to require TCP. Owen
On 2010-03-26, at 06:40, Max Larson Henry wrote:
has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc.
- Yes but as for DNS, anycast is essentially used for user requests (UDP) not to perform zone transfer(TCP).
As others have mentioned, TCP can generally be used for any DNS query, not just AXFR. This becomes more important as DNS responses get bigger, e.g. responses from root servers due to the root zone containing DNSSEC information, see <http://www.root-dnssec.org/>. If your nameserver can't be reached over TCP, it's likely that there are people who can't talk to your nameserver. This means your DNS records can't be found. This is a bad thing. Here, in glorious LOLCAPS: ALWAYS MAKE SURE YOUR DNS SERVER CAN BE REACHED OVER TCP TCP IS NOT JUST FOR ZONE TRANSFERS FIX YOUR FIREWALLS :-) Joe
From: Joe Abley <jabley@hopcount.ca> Date: Fri, 26 Mar 2010 10:06:02 -0700
On 2010-03-26, at 06:40, Max Larson Henry wrote:
has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc.
- Yes but as for DNS, anycast is essentially used for user requests (UDP) not to perform zone transfer(TCP).
As others have mentioned, TCP can generally be used for any DNS query, not just AXFR.
This becomes more important as DNS responses get bigger, e.g. responses from root servers due to the root zone containing DNSSEC information, see <http://www.root-dnssec.org/>.
If your nameserver can't be reached over TCP, it's likely that there are people who can't talk to your nameserver. This means your DNS records can't be found. This is a bad thing.
Here, in glorious LOLCAPS:
ALWAYS MAKE SURE YOUR DNS SERVER CAN BE REACHED OVER TCP TCP IS NOT JUST FOR ZONE TRANSFERS FIX YOUR FIREWALLS
:-)
Fix your security officers! I have talked to multiple security officers (who are generally not really knowledgeable on networks) who had 53/tcp blocked and none have yet agreed to change it. The last one told me that blocking 53/tcp is "standard industry practice" as per his firewall training. Point out what RFCs said simply bounced off of him. He said that if the protocols would not handle blocked 53/tcp, the protocols would have to be changed. Opening the port was simply not open to discussion. They don't seen to really care if things are broken and seem to feel that it is up to "the network" to accommodate their idea of normal firewall configuration. I will say that these were at federal government facilities. I hope the commercial world is a bit more in touch with reality. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
On 30 Mar 2010, at 07:59, Randy Bush <randy@psg.com> wrote:
I have talked to multiple security officers (who are generally not really knowledgeable on networks) who had 53/tcp blocked and none have yet agreed to change it.
patience. when things really start to break, and the finger of fate points at them, clue may arise.
36 days until all root servers have DNSSEC data, at which point large replies become normal. Tony (on his iPod). -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/
I have talked to multiple security officers (who are generally not really knowledgeable on networks) who had 53/tcp blocked and none have yet agreed to change it. patience. when things really start to break, and the finger of fate points at them, clue may arise. 36 days until all root servers have DNSSEC data, at which point large replies become normal.
are end user tools, i.e. a web click a button, available so they can test if they are behind a clueless security id10t? is there good simple end user docco they are somewhat likely to find when things break for them? i.e. what can we do to maximize the odds that the victim will quickly find the perp, as opposed to calling our our tech support lines? randy
Randy Bush (randy) writes:
i.e. what can we do to maximize the odds that the victim will quickly find the perp, as opposed to calling our our tech support lines?
Ah yes, there was the second good reason for actually helping netops and security officers :) Tools: https://www.dns-oarc.net/oarc/services/replysizetest https://www.dnssec-deployment.org/wiki/index.php/Tools_and_Resources, under troubleshooting: http://labs.ripe.net/content/testing-your-resolver-dns-reply-size-issues http://secspider.cs.ucla.edu/ Info sheets: http://www.afnic.fr/actu/nouvelles/240/l-afnic-invite-les-responsables-techn... (click English, top right) ... plenty of links there too. Cheers, Phil
I must observe that these are not really the links you'd want to give your end users to check out. Their audience is very different. While the article on RIPE Labs comes close, they don't really answer the "does it work or does it not?" question with a green/red light, and they don't provide a good explanation to the audience Randy is referring to. Robert On 2010.03.30. 11:29, Phil Regnauld wrote:
Randy Bush (randy) writes:
i.e. what can we do to maximize the odds that the victim will quickly find the perp, as opposed to calling our our tech support lines?
Ah yes, there was the second good reason for actually helping netops and security officers :)
Tools:
https://www.dns-oarc.net/oarc/services/replysizetest
https://www.dnssec-deployment.org/wiki/index.php/Tools_and_Resources, under troubleshooting: http://labs.ripe.net/content/testing-your-resolver-dns-reply-size-issues http://secspider.cs.ucla.edu/
Info sheets:
http://www.afnic.fr/actu/nouvelles/240/l-afnic-invite-les-responsables-techn... (click English, top right)
... plenty of links there too.
Cheers, Phil
Robert Kisteleki (robert) writes:
I must observe that these are not really the links you'd want to give your end users to check out. Their audience is very different. While the article on RIPE Labs comes close, they don't really answer the "does it work or does it not?" question with a green/red light, and they don't provide a good explanation to the audience Randy is referring to.
Fair enough. Some simple "check your DNS reply size test [what is this ?]" page ought to be set up, with a simple explanagtion. "checkmydns.org" is available. If I get 5 minutes... :)
* Phil Regnauld:
Fair enough. Some simple "check your DNS reply size test [what is this ?]" page ought to be set up, with a simple explanagtion. "checkmydns.org" is available. If I get 5 minutes... :)
Reply sizes are a red herring. You need something that looks at the result of ./IN/DNSKEY, ./IN/RRSIG, ./IN/NSEC. At least one of these queries should return data, some of the time. (Unfortunately, the test is probabilistic.) Then you know that your resolver can receive data from the signed root and will not cease to work when all the roots serve the signed zone. Other tests can't tell you that. If your resolver is DNSSEC-aware, you can force cache misses by using random query names with a non-existing TLD. This variant of the test is much easier to carry out.
On Tue, Mar 30, 2010 at 05:43:25PM +0900, Randy Bush wrote:
I have talked to multiple security officers (who are generally not really knowledgeable on networks) who had 53/tcp blocked and none have yet agreed to change it. patience. when things really start to break, and the finger of fate points at them, clue may arise. 36 days until all root servers have DNSSEC data, at which point large replies become normal.
are end user tools, i.e. a web click a button, available so they can test if they are behind a clueless security id10t?
no - in part because using a browser to debug DNS involves a third app (and likly a third/forth) platform. the nifty OARC testpoint is nearly worthless for real operations, since its not located at/near a DNS authoritative source. the K testpoint is good, I should prolly put back the one off B.
is there good simple end user docco they are somewhat likely to find when things break for them?
not yet. in part because out of the few simple parts, many, many combinations of failure can occur. ) MTU strictures: v6/v4 tunneling v6/v4 MTU clamping ) Fragmenation UDP ) Port blocking ) Resolver Behaviour EDNS awareness
i.e. what can we do to maximize the odds that the victim will quickly find the perp, as opposed to calling our our tech support lines?
thats a tough call. as tech support staff, we are almost always an outside observer on the path btwn the victim and the perp. troubleshooting is going to be problematic.
randy
On Tue, 30 Mar 2010 15:59:08 +0900, Randy Bush said:
I have talked to multiple security officers (who are generally not really knowledgeable on networks) who had 53/tcp blocked and none have yet agreed to change it.
patience. when things really start to break, and the finger of fate points at them, clue may arise.
How many years did it take for firewalls to quit screwing with the ECN bits?
Randy Bush (randy) writes:
patience. when things really start to break, and the finger of fate points at them, clue may arise.
When this issue was brought up on the OARC dns-operations list, and it was suggested to make some simply factsheets (a bit like ICANN's IPv6 http://www.icann.org/announcements/factsheet-ipv6-26oct07.pdf), this was poo-pooed as being useless and a waste of time. Since the final victim is the end user, I still think it's worth the effort to try and make security officers and similar network operators aware of the issues and what they can do to mitigate potential problems. See for example: http://www.afnic.fr/actu/nouvelles/240/l-afnic-invite-les-responsables-techn... (Yes, there's an English version too). Cheers, Phil
"Kevin Oberman" <oberman@es.net> writes:
He said that if the protocols would not handle blocked 53/tcp, the protocols would have to be changed. Opening the port was simply not open to discussion.
Let me guess: They also completely blocked ICMP. I always tell these customers to switch to IPv6 real fast and to turn of ICMPv6 to make their networks really secure. ;-)
I will say that these were at federal government facilities. I hope the commercial world is a bit more in touch with reality.
You can find clueless people everywhere. Jens -- ------------------------------------------------------------------------- | Foelderichstr. 40 | 13595 Berlin, Germany | +49-151-18721264 | | http://www.quux.de | http://blog.quux.de | jabber: jenslink@guug.de | -------------------------------------------------------------------------
"Kevin Oberman" <oberman@es.net> writes:
He said that if the protocols would not handle blocked 53/tcp, the protocols would have to be changed. Opening the port was simply not open to discussion.
Do they also believe that all DNS replies are less than 512 bytes? :-) Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS. MODERATE OR GOOD.
"Kevin Oberman" <oberman@es.net> writes:
He said that if the protocols would not handle blocked 53/tcp, the protocols would have to be changed. Opening the port was simply not open to discussion.
Do they also believe that all DNS replies are less than 512 bytes? :-)
Sure, why not. The phrase "simply not open to discussion" in this context can be accurately read as "we were told this was good, but couldn't possibly defend the line of reasoning since we have no clue what it was." It's like debating PMTU brokenness with someone who feels that blocking all ICMP is a Really Smart Clever Good Thing To Do, because someone told them all about evil ICMP. Sometimes, the happiest solution is to let the pain rain. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Mon, 29 Mar 2010, Kevin Oberman wrote:
Fix your security officers!
I have talked to multiple security officers (who are generally not really knowledgeable on networks) who had 53/tcp blocked and none have yet agreed to change it. The last one told me that blocking 53/tcp is "standard industry practice" as per his firewall training. Point out what RFCs said simply bounced off of him. He said that if the protocols would not handle blocked 53/tcp, the protocols would have to be changed. Opening the port was simply not open to discussion.
They don't seen to really care if things are broken and seem to feel that it is up to "the network" to accommodate their idea of normal firewall configuration.
I will say that these were at federal government facilities. I hope the commercial world is a bit more in touch with reality.
If you are with a US Federal agency having problems like this, or any other issues with your DNSSEC deployment, please include them in the DNSSEC survey every US Federal executive agency has been tasked to submit by the Office of Management and Budget. http://iase.disa.mil/stigs/stig/dns_stig_v4r1_20071017.pdf If, for example, an organization placed an authoritative server in its DMZ but limited zone transfers to servers behind the firewall, then it could allow inbound and outbound UDP 53 to and from the DMZ name server to allow queries, but deny TCP 53 in both directions to prohibit zone transfers. This configuration, however, is strongly discouraged because it may prevent legitimate DNS transactions that involve large responses (e.g., a DNSSEC signature). In general, limitations on queries, zone updates and transfers should be implemented in the name server's configuration rather than through configuration of firewalls, routers, or other communications devices.
On Mar 26, 2010, at 9:24 AM, Jeroen Massar wrote:
InterNetX - Lutz Muehlig wrote:
Hello,
has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc.
Can't really tell if you're being serious here due to caffeine underrun. http://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf Slide 23 seems quite appropriate. http://www.ripe.net/ripe/meetings/ripe-50/presentations/ripe50-plenary-tue-a... has links to other work on this. It certainly seems to work "well enough".
I assume quite a number of people know how to do it, especially as several root DNS servers abuse it.
Simple recipe: - Box with: - Your favourite OS - Quagga or OpenBGPd - Your favourite DNS server - Announce the IP of the anycast node in BGP - Monitor the DNS server, when it does not work kill your local BGPd and notify the admins that it broke
That is it. Probably with the above couple of things, google a bit and find the rest.
Greets, Jeroen
* Jeroen Massar:
Simple recipe: - Box with: - Your favourite OS - Quagga or OpenBGPd - Your favourite DNS server - Announce the IP of the anycast node in BGP - Monitor the DNS server, when it does not work kill your local BGPd and notify the admins that it broke
This is fine if you just use BGP to indirectly hook into your IGP. For global anycast, you need a sufficiently short prefix with no non-anycast services on it. This can be difficult to justify for newcomers. -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
On Fri, 26 Mar 2010 14:24:21 +0100 Jeroen Massar <jeroen@unfix.org> wrote:
InterNetX - Lutz Muehlig wrote:
Hello,
has someone experience in anycast ipv4 networks (to support DNS)?
"Never been done" "Dangerous" "TCP does not work" etc etc etc.
I assume quite a number of people know how to do it, especially as several root DNS servers abuse it.
Simple recipe: - Box with: - Your favourite OS - Quagga or OpenBGPd - Your favourite DNS server - Announce the IP of the anycast node in BGP - Monitor the DNS server, when it does not work kill your local BGPd and notify the admins that it broke
That is it. Probably with the above couple of things, google a bit and find the rest.
I was involved in building an anycast setup where we had two anycast DNS /32 addresses. Each of them was the backup for the other i.e. each DNS server was announcing both /32s via BGP, with opposite weights. If one failed, the other DNS server then took over the failed DNS cache's queries, and as it was also already an operational DNS server for one of the anycast addresses, it's DNS cache was already hot. For load balancing, we alternated the order of announcing the DNS server addresses in e.g. PPP IPCP, DHCP. That worked somewhat surprisingly well - the peak query per second values on each of them were no more than about 10% different. One trap we got caught by was stateful firewalling on the host. We knew to up the number of stateful connections, however on that particular Linux distro there were two places it was set - /etc/sysctl.conf and in the iptables configuration. We only knew about the first, so when the firewall rules were updated the number of supported stateful connections was dropped down to too low a level. It wasn't funny to have one DNS server stop answering queries, and have it's own monitoring script fail itself, switch all the traffic to the other one and then have that die too for the same reason. Lots of gnashing of teeth until we worked out . The final and better solution was to stop doing stateful firewalling on DNS queries, using the iptables 'raw' table. Regards, Mark.
On 2010-03-26, at 06:21, InterNetX - Lutz Muehlig wrote:
has someone experience in anycast ipv4 networks (to support DNS)?
This is a general reference that tries hard not to be DNS-specific: http://www.ietf.org/rfc/rfc4786.txt These are two papers written whilst at ISC describing many aspects of how ISC did what you are talking about with the F root nameserver: http://ftp.isc.org/isc/pubs/tn/isc-tn-2003-1.txt http://ftp.isc.org/isc/pubs/tn/isc-tn-2004-1.txt This is a rendering of the OSPF/IGP anycast tech note above as a USENIX paper: http://www.usenix.org/events/usenix04/tech/sigs/full_papers/abley/abley.pdf This is a more sarcastic article about anycast that was published in the USENIX publication ;login: http://www.usenix.org/publications/login/2008-02/openpdfs/abley.pdf This is a presentation about building nameserver clusters that was presented at NANOG 34: http://nanog.org/meetings/nanog34/presentations/abley.nameservers.pdf Joe
participants (21)
-
bmanning@vacation.karoshi.com
-
Florian Weimer
-
Florian Weimer
-
InterNetX - Lutz Muehlig
-
Jens Link
-
Jeroen Massar
-
Joe Abley
-
Joe Greco
-
John Payne
-
Kevin Oberman
-
Mark Andrews
-
Mark Smith
-
Max Larson Henry
-
Owen DeLong
-
Paul Ryland
-
Phil Regnauld
-
Randy Bush
-
Robert Kisteleki
-
Sean Donelan
-
Tony Finch
-
Valdis.Kletnieks@vt.edu