Numbering nameservers and resolvers
Hi Folks, I am needing to renumber some core infrastructure - namely, my nameservers and my resolvers - and I was wondering if the collective wisdom still says heck yes keep this stuff all on seperate subnets away from eachother? Anyone got advice either way? Should I try to give sequential numbers to my resolvers for the benefit of consultants ... like .11, .22 and .33 for my server ips? Mike-
Composed on a virtual keyboard, please forgive typos. On Aug 16, 2010, at 7:49, Mike <mike-nanog@tiedyenetworks.com> wrote:
Hi Folks,
I am needing to renumber some core infrastructure - namely, my nameservers and my resolvers - and I was wondering if the collective wisdom still says heck yes keep this stuff all on seperate subnets away from eachother? Anyone got advice either way? Should I try to give sequential numbers to my resolvers for the benefit of consultants ... like .11, .22 and .33 for my server ips?
1) Use different prefixes. A single prefix going down should not kill your entire network. (Nameservers and resolvers being unreachable breaks the whole Internet as far as users are concerned.) 2) Consider trading secondary NS with another AS. This is for authorities only, recursive NSes should be on-net only. 3) Try not to use the first /24 in a large prefix. See as7007 incident for why, although that is probably less likely today. 4) Using easily memorized numbers for at least one authority & one resolved will help your NOC, but should not override other considerations. That's a start, I'm sure others will have more suggestions. -- TTFN, patrick
Once upon a time, Patrick W. Gilmore <patrick@ianai.net> said:
1) Use different prefixes. A single prefix going down should not kill your entire network. (Nameservers and resolvers being unreachable breaks the whole Internet as far as users are concerned.)
How do you do this in the IPv6 world, where I get a single /32? Will others accept announcements of two /33s to better handle things like this? -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On Aug 16, 2010, at 6:03 AM, Chris Adams wrote:
Once upon a time, Patrick W. Gilmore <patrick@ianai.net> said:
1) Use different prefixes. A single prefix going down should not kill your entire network. (Nameservers and resolvers being unreachable breaks the whole Internet as far as users are concerned.)
How do you do this in the IPv6 world, where I get a single /32? Will others accept announcements of two /33s to better handle things like this?
The better solution is to trade secondary services with some other provider. Sure, it's a bit of a pain keeping up with the new zones to be added and old zones to be removed back and forth, but, it's a great way to have your authoritative servers truly diverse and independent. Owen
On Mon, Aug 16, 2010 at 06:08:02AM -0700, Owen DeLong wrote:
On Aug 16, 2010, at 6:03 AM, Chris Adams wrote:
Once upon a time, Patrick W. Gilmore <patrick@ianai.net> said:
1) Use different prefixes. A single prefix going down should not kill your entire network. (Nameservers and resolvers being unreachable breaks the whole Internet as far as users are concerned.)
How do you do this in the IPv6 world, where I get a single /32? Will others accept announcements of two /33s to better handle things like this?
The better solution is to trade secondary services with some other provider. Sure, it's a bit of a pain keeping up with the new zones to be added and old zones to be removed back and forth, but, it's a great way to have your authoritative servers truly diverse and independent.
At $JOB[3], where I was responsible for this sort of thing, a small amount of shell scripting behind inetd on the master[1], and slightly more shell scripting behind cron on the secondaries[2], and all our problems were solved for all time. - Matt [1] Read /etc/named/zones/* mangled the (standardised) filenames to get a list of the zones, and dumped it on stdout, which went out on a high port that inetd was listening on. [2] nc to the master on the relevant high port, read the list and write out an automated named.conf fragment. Also use a bit of md5sum to detect when the list changed, so we know when to reload named on the slave. [3] Subscript, not footnote.
nowadays, i'd simply put them all on the same /24 which you simply announce on different pops tcp/zonetransfer not working reliably is no longer a problem as you simply retreive those directly from the database over a seperate ip, no more old-fashioned bind related crap. so 1 /24 prefix, with one ip for your authorative nameserver, and maybe one for a resolver if needed, and the rest you leave unused.. this you simply put right next to the routers where you pick up your transit for transport to your own facilities (bet you have some rackspace and power left there too ;) making the network itself redundant rather than the nameserver... not to mention ofcourse that you fit these nameservers with solid state hdd's and ramdisks for the changing files and no moving parts so they last forever, and that whatever nameserver software you run is either an init child with respawn.. as these boxes are actually an integrated solid state router+nameserver, they have a normal static ip for the bgp/ospf session/routing and therefore can use this ip to retreive information themselves from the database and other nameservers once more and more parties buy/build routers with sufficient ram and therefore can handle larger routing tables (it's 2010 people, move on ;) you can also make the prefix smaller, let's say a /29.. our own setup is not yet a proper example here btw, so no bashing on that, but this is what our next setup will look like. kinda like ripes k-root, just used for ordinary authorative servers/resolvers pretty much plug and play (with ospf, with bgp it requires some additional configging ;) and nuke resistant, just the way we like it. this whole "you have to put 2 nameservers on two seperate subnets at two different locations" seems a bit.. pre-1993 to me. plus, why only 2, why not... 20 or so, all in different parts of the world and let bgp handle the rest. -- Greetings, Sven Olaf Kamphuis, CB3ROB Ltd. & Co. KG ========================================================================= Address: Koloniestrasse 34 VAT Tax ID: DE267268209 D-13359 Registration: HRA 42834 B BERLIN Phone: +31/(0)87-8747479 Germany GSM: +49/(0)152-26410799 RIPE: CBSK1-RIPE e-Mail: sven@cb3rob.net ========================================================================= <penpen> C3P0, der elektrische Westerwelle ========================================================================= Confidential: Please be advised that the information contained in this email message, including all attached documents or files, is privileged and confidential and is intended only for the use of the individual or individuals addressed. Any other use, dissemination, distribution or copying of this communication is strictly prohibited. On Tue, 17 Aug 2010, Matthew Palmer wrote:
On Mon, Aug 16, 2010 at 06:08:02AM -0700, Owen DeLong wrote:
On Aug 16, 2010, at 6:03 AM, Chris Adams wrote:
Once upon a time, Patrick W. Gilmore <patrick@ianai.net> said:
1) Use different prefixes. A single prefix going down should not kill your entire network. (Nameservers and resolvers being unreachable breaks the whole Internet as far as users are concerned.)
How do you do this in the IPv6 world, where I get a single /32? Will others accept announcements of two /33s to better handle things like this?
The better solution is to trade secondary services with some other provider. Sure, it's a bit of a pain keeping up with the new zones to be added and old zones to be removed back and forth, but, it's a great way to have your authoritative servers truly diverse and independent.
At $JOB[3], where I was responsible for this sort of thing, a small amount of shell scripting behind inetd on the master[1], and slightly more shell scripting behind cron on the secondaries[2], and all our problems were solved for all time.
- Matt
[1] Read /etc/named/zones/* mangled the (standardised) filenames to get a list of the zones, and dumped it on stdout, which went out on a high port that inetd was listening on.
[2] nc to the master on the relevant high port, read the list and write out an automated named.conf fragment. Also use a bit of md5sum to detect when the list changed, so we know when to reload named on the slave.
[3] Subscript, not footnote.
Sven, On Aug 17, 2010, at 8:11 AM, Sven Olaf Kamphuis wrote:
this whole "you have to put 2 nameservers on two seperate subnets at two different locations" seems a bit.. pre-1993 to me. plus, why only 2, why not... 20 or so, all in different parts of the world and let bgp handle the rest.
There's an important component that is missing from the above. It's one thing to have a single nameserver hosted in such a manner, but through operational integration and history there are still a lot of domain names that are not fault tolerant. I remember "in recent years" a ccTLD that ended up without functioning services as a result of poor nameserver site selection. Ideally you would have a system with two geographically diverse nameservers for a domain, under seperate (routing) administrative control. One of my former employers backhauled all their legacy nameservers to a single site, eg: e[0-2].ns.voyager.net. While they were originally on diverse subnets and geographical locations, this appears to have changed. Selecting a site outside of your control is valuable. When I was hostmaster@cic.net, we "traded" with mr.net. These days, if I were in the same role, I would want to have three instead of two. Asia, Europe and US someplace. If US only, east, west and central. If you look at ntt.net, our "off-net" resolver is 69.36.249.36 This means if there is a ntt meltdown, there's a good chance you can still resolve related names off-net. - Jared
One of my former employers backhauled all their legacy nameservers to a = single site, eg: e[0-2].ns.voyager.net.
While they were originally on diverse subnets and geographical = locations, this appears to have changed.
As one of the people who originally worked on that setup, I'll note that they're all being announced by 7321, which is certainly not the ideal for purposes of diversity, but I notice some variation in the ping times, so it's not clear to me that there's a reliable basis for expecting that they're not at least diverse geographically. The original locations were in Dayton, Kalamazoo, and New Berlin, all of which are several ms away from Chicago, and while several of the facilities have been shuffled around or closed, it's not clear that there aren't still nameservers in those states. AS diversity wasn't there for all that long to begin with; I seem to recall that a lot of it was being announced from 8011 as the integration efforts went on. It seems to me that for very small or very large organizations, there are significant benefits to finding AS diversity, but for mid-size ones, the picture is a bit less clear. In the Voyager case, the existence of separate networks was something that came along as more of a bonus and side effect of acquistions, and nameserver engineering took advantage of it, but network engineering's goal was to get all the networks integrated and connected, so eventually things got rolled into 8011. That would definitely count as somewhat suboptimal from the point of nameserver reliability, but the network grew generally more reliable since there weren't twenty slightly different ways of doing things and lots of legacy crud that neteng needed to "just know". While that did a lot to increase the overall reliability of the network, it certainly is putting your eggs all in one basket, and then you have to be ready for the hazards. We had, for example, this guy in Michigan who liked to load up routers in remote locations with unreleased versions of Cisco code that he'd get from his contacts at Cisco, which led to several cases of network downtime when they didn't work as expected (or at all). I believe that Wisconsin network engineering was generally fearful that one day it would turn ugly and something bad would happen that would take down the whole network; this is the downside to having less compartmentalization. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Tue, Aug 17, 2010 at 08:52:20AM -0400, Jared Mauch wrote:
Selecting a site outside of your control is valuable. When I was hostmaster@cic.net, we "traded" with mr.net. These days, if I were in the same role, I would want to have three instead of two. Asia, Europe and US someplace. If US only, east, west and central.
While this is good advice, I think it is also important to consider your customer base. I could easily host an authoritative nameserver for my domains in Japan, but I elected not to do so, because most of the end users who would be querying it are in Canada, and, with one nameserver in Canada and one in Japan, they would get a long RTT on DNS queries roughly half the time. -Phil
nowadays, i'd simply put them all on the same /24 which you simply announce on different pops
tcp/zonetransfer not working reliably is no longer a problem as you simply retreive those directly from the database over a seperate ip, no more old-fashioned bind related crap.
tcp/zonetransfer can also be configured to run off of a different IP address, for example, the native IP of the box. This works just fine. In BIND, you're looking for transfer-source ${qaddr} port ${qport}; IIRC. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Once upon a time, Sven Olaf Kamphuis <sven@cb3rob.net> said:
tcp/zonetransfer not working reliably is no longer a problem as you simply retreive those directly from the database over a seperate ip, no more old-fashioned bind related crap.
TCP is not just for zone transfers (especially in the age of DNSSEC and still-broken firewalls). -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On Aug 17, 2010, at 8:56 AM, Chris Adams wrote:
Once upon a time, Sven Olaf Kamphuis <sven@cb3rob.net> said:
tcp/zonetransfer not working reliably is no longer a problem as you simply retreive those directly from the database over a seperate ip, no more old-fashioned bind related crap.
TCP is not just for zone transfers (especially in the age of DNSSEC and still-broken firewalls).
Yeah. there's a lot of bad networking voodoo out there. I was on the NY State Thruway in recent weeks, and noticed a few things: 1) Don't query their website for an AAAA record, nor attempt to report it to the state. They say "we don't support IPv6" - not understanding sending back a SERVFAIL is bad 2) Don't expect 1.1.1.1 to work, they use that as a HTTPS portal, so you not only get broken IP, but a broken certificate login page 3) Comcast will sometimes reply from a "different" IP than you sent the query if the dns query fails in such a manner. - Jared
On Tue, 17 Aug 2010 12:11:56 +0000 (UTC) Sven Olaf Kamphuis <sven@cb3rob.net> wrote:
nowadays, i'd simply put them all on the same /24 which you simply announce on different pops
I would raise a red flag of caution with this approach especially for services that need to be reachable outside your network If there is a a snafu with said /24 prefix, particularly outside your own routing domain, a reachability problem could persist for an extended period and you'd be in a difficult position to solve it on your own. For instance, if it flaps and someone, for better or worse, dampens that route, that could mean an extended outage for all those hosts until the damping period timer expires. On a related note, some systems and folks have taken multiple unique origin ASNs as a measure of diversity. In pratice, unless there is some odd AS path mangling going on for your specific routes, which is unlikely, one can properly instrument diversity using a single origin ASN with multiple prefixes. Its the path and the prefix that matters, much less the ASN. John
On 08/17/2010 05:11, Sven Olaf Kamphuis wrote:
tcp/zonetransfer not working reliably is no longer a problem
TCP is a MUST for DNS. It's used as a fallback in the normal resolution process if an answer can't fit in a UDP packet for whatever reason. This is true even for common things like large A record lists, but is only becoming more frequent in the age of DNSSEC, AAAA, etc. It is unfortunately even more necessary than we had hoped it would be due to many local network operators not "getting the memo" regarding EDNS. hth, Doug -- Improve the effectiveness of your Internet presence with a domain name makeover! http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso
In IPv6 you should be able to advertise up to /48 with no problem... Arie On Mon, Aug 16, 2010 at 4:03 PM, Chris Adams <cmadams@hiwaay.net> wrote:
Once upon a time, Patrick W. Gilmore <patrick@ianai.net> said:
1) Use different prefixes. A single prefix going down should not kill your entire network. (Nameservers and resolvers being unreachable breaks the whole Internet as far as users are concerned.)
How do you do this in the IPv6 world, where I get a single /32? Will others accept announcements of two /33s to better handle things like this?
-- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On Aug 16, 2010, at 9:03 AM, Chris Adams wrote:
Once upon a time, Patrick W. Gilmore <patrick@ianai.net> said:
1) Use different prefixes. A single prefix going down should not kill your entire network. (Nameservers and resolvers being unreachable breaks the whole Internet as far as users are concerned.)
How do you do this in the IPv6 world, where I get a single /32? Will others accept announcements of two /33s to better handle things like this?
Oooooooooobviously I was not thinking clearly, since I was only considering v4. =) But you do what you can. Some people have only a single v4 prefixes as well. If you can't use more than one prefix, then don't. Other good suggestions were things like ensuring the default exit point for each NS was a different vector. If you have only one exit point, that is not possible. But it does not mean the suggestion is bad. -- TTFN, patrick
On Aug 16, 2010, at 9:03 AM, Chris Adams wrote:
Once upon a time, Patrick W. Gilmore <patrick@ianai.net> said:
1) Use different prefixes. A single prefix going down should not kill your entire network. (Nameservers and resolvers being unreachable breaks the whole Internet as far as users are concerned.)
How do you do this in the IPv6 world, where I get a single /32? Will others accept announcements of two /33s to better handle things like this?
Everyone I know that is clued in this arena goes out and (even as a $sfi_network) BUYS transit/colo/somethingelse from another network (or does it via trade/barter/part of peering agreements, you get the idea). There are also people you can outsource this stuff to as well to make sure it's done right. Honestly, I'm surprised both at how many and how few people do this. Those that have the network capability pay someone else to do DNS at times, and those who should pay to have a reliable service hack it on some poor network infrastructure. Some people even provide an API or web interface to manage secondary dns servers to help out those in need. Try clicking here: http://www.google.com/search?q=free+secondary+dns - Jared
On Aug 16, 2010, at 12:49 AM, Mike wrote:
Hi Folks,
I am needing to renumber some core infrastructure - namely, my nameservers and my resolvers - and I was wondering if the collective wisdom still says heck yes keep this stuff all on seperate subnets away from eachother? Anyone got advice either way? Should I try to give sequential numbers to my resolvers for the benefit of consultants ... like .11, .22 and .33 for my server ips?
Resolvers being easily memorable is nice, since they get keyed in by IP. Authority servers are referred to by name, so IP matters less. Definitely keep an authority server in another prefix if you can, and resolvers in different prefixes is also nice -- but that's more a question of redundancy, not numbering. Other than that, go dense. Addresses are starting to get scarce. Aria Stewart
On Sun, 15 Aug 2010 23:49:05 PDT, Mike said:
I am needing to renumber some core infrastructure - namely, my nameservers and my resolvers - and I was wondering if the collective wisdom still says heck yes keep this stuff all on seperate subnets away from eachother? Anyone got advice either way
Microsoft used to have all their DNS servers on one /24. Nine years later, you can still use Google on just 'microsoft dns server failure subnet' and find this on the second page of over a million hits: http://www.wired.com/techbiz/media/news/2001/01/41423 (OK, so our local resolvers are in one /24, but it's a bridged VLAN across our entire campus, the servers are physically in buildings several miles apart, and if you can't reach at least one of them, it probably means our campus core network is hosed enough that you're not going to do anything with a DNS response anyhow... Our authoritative servers are split across 2 different AS's in 2 different states.) Whatever gave you the idea that collective wisdom could *possibly* have moved away from "spread it out as far as you can to avoid single points of failure"?
On 8/16/2010 2:49 AM, Mike wrote:
from eachother? Anyone got advice either way? Should I try to give
If you have a dedicated subnet for /32s (e.g., router loopback interfaces), i'd pick from there. if you eventually require geo-redundancy or want to load balance your queries, it's much neater injecting them into your igp rather than having a few /32's injected from an otherwise nice clean /24. I am also a fan of keeping your recursive and authoritative ip addresses separate. Not only is this much more modular, it can be more secure; see http://cr.yp.to/djbdns/separation.html -- Jeremy Kister http://jeremy.kister.net./
for authoritatuve servers, i try to have one on a very different backbone on a distant continent. i make deals with friends. there have been just too many failures where servers were in the same facility, or behind the same routing, or on a single backbone. see rfc 2182. for customer- and infrastructure-facing caches, i try for as different routing domains and peering/x-stream exits as i can while keeping them easily reachable by their clients. randy
For resolvers, I guess it would make sense to advertise them as /32s as dynamic prefixes coming from some SLB device... You can have multiple VIPs, each representing a different POP/network domain... Arie On Mon, Aug 16, 2010 at 9:49 AM, Mike <mike-nanog@tiedyenetworks.com> wrote:
Hi Folks,
I am needing to renumber some core infrastructure - namely, my nameservers and my resolvers - and I was wondering if the collective wisdom still says heck yes keep this stuff all on seperate subnets away from eachother? Anyone got advice either way? Should I try to give sequential numbers to my resolvers for the benefit of consultants ... like .11, .22 and .33 for my server ips?
Mike-
On 2010-08-16 08:49, Mike wrote:
Hi Folks,
I am needing to renumber some core infrastructure - namely, my nameservers and my resolvers - and I was wondering if the collective wisdom still says heck yes keep this stuff all on seperate subnets away from eachother? Anyone got advice either way? Should I try to give sequential numbers to my resolvers for the benefit of consultants ... like .11, .22 and .33 for my server ips?
Take a IPv4 /24, /28, whatever size you might think you need and an IPv6 /64 and make it your 'service prefix', then anycast this inside your network and do the standard 'bgp daemon on the box, monitor the local service' trick and kill the announcement when the service does not work, presto. As for the actually numbers, just keep them simple. Using port-numbers (53 = DNS, 25 = SMTP etc etc etc) where possible is easy for at least the more technical folks, of course IPv4 only goes up to 255, IPv6 does not have that issue. Greets, Jeroen
On 8/15/2010 11:49 PM, Mike wrote:
Hi Folks,
I am needing to renumber some core infrastructure - namely, my nameservers and my resolvers - and I was wondering if the collective wisdom still says heck yes keep this stuff all on seperate subnets away from eachother?
Authoritative name servers should be on different networks, preferably in entirely different facilities. You've already had good suggestions about swapping secondary service, etc. Resolving name servers should be separate from authoritative ones, but there is no reason that they can't be on the same subnet(s). It's still a good idea to have more than one resolver on each local network, but there is also no reason they can't be on the same subnet as well. For larger and/or highly performance sensitive installations anycasting the resolvers (so that you only need 1 IP in resolv.conf) is becoming more popular.
Anyone got advice either way? Should I try to give sequential numbers to my resolvers for the benefit of consultants ... like .11, .22 and .33 for my server ips?
This sounds more like a local preference issue. Presumably the people who don't type into config files for a living will have their hosts configured with DHCP, and those who do will know how to copy and paste. :) hth, Doug PS, if you need more formal help, see the URL below. -- ... and that's just a little bit of history repeating. -- Propellerheads Improve the effectiveness of your Internet presence with a domain name makeover! http://SupersetSolutions.com/
On 16/08/2010 08:49, Mike wrote:
I am needing to renumber some core infrastructure - namely, my nameservers and my resolvers - and I was wondering if the collective wisdom still says heck yes keep this stuff all on seperate subnets away from eachother? Anyone got advice either way? Should I try to give sequential numbers to my resolvers for the benefit of consultants ... like .11, .22 and .33 for my server ips?
We have 4 authoritative nameservers with a management backend to make sure that their records are in sync. The servers are located on 3 separate continents, originated on 4 different ASNs, numbered from 4 different /8's and not sharing any common data centre or power infrastructure. The software platform is still a single point of failure and some people have recommended a mix of software vendors for additional redundancy. With resolvers the approach is a bit different: You want an easy to remember address and also an address that will not be subject to renumbering in the future. Even though they shouldn't we see many users statically configuring their DNS resolvers. A dedicated prefix for each resolver would be my first choice. You can then move that prefix to different hardware if necessary even if the routing to the hardware changes. A dedicated prefix also allows you to anycast the service if required. Since this is only internal routing it doesn't need to be a full /24. I have also found it helpful to have the upstream queries originating from IPs in separate prefixes and this is quite easy to move around transparently to users or even in an emergency. On IPv6 I have reserved 4 x /48s for DNS resolvers. The prefixes were chosen to be short and easy to remember and they are routed to existing resolvers. The :1 of each prefix is added to the loopback on the resolver. -- Graham Beneke
participants (19)
-
Aria Stewart -
Arie Vayner -
Chris Adams -
Doug Barton -
Graham Beneke -
Jared Mauch -
Jeremy Kister -
Jeroen Massar -
Joe Greco -
John Kristoff -
Lyndon Nerenberg -
Matthew Palmer -
Mike -
Owen DeLong -
Patrick W. Gilmore -
Phil Vandry -
Randy Bush -
Sven Olaf Kamphuis -
Valdis.Kletnieks＠vt.edu