Broken domain statistics...
Hi all, I'd finally had it up to here with people coming from misconfigured domains trying to connect to my servers and filling up the logs with 'Host name mismatch..' errors, so I decided to put together a bunch of scripts to try and see exactly how widespread the problem of bogus DNS info is. What I found was kind of surprising. Here're the raw stats for my test (done on the entire com zone): Total domains checked: 1401150 Domains with NO good nameservers (all responded non-auth): 236008 Domains with NO good nameservers (some timed out): 107482 Domains with at least one bad non-auth (but most/all answering):99211 Bringing that into percentages, about 17% of all domains in COM have NO good nameservers listed. If one adds in the nameservers that timed out, that number goes up to 25%, and adding in domains with at least one bad nameserver brings the number up to 32% of the domains in .com that have bad nameserver info registered. Please note, it appears that it's not entirely accurate to view the nameservers that timed out as necessarily 'bad' in my test - several known good nameservers timed out during the runs, and I only had a retry of 1 (so the nameservers got 1 chance to give the correct data within 4 seconds). It's very interesting, however, that the number of domains that had all listed nameservers respond, all of which responded non-authoritatively (i.e. 'I don't know about this domain') is so high. Here is my testing methodology: I read in the entire com zone, and when I found a line containing dom IN NS server I would spawn off a 'dig ANY dom @server +retry=1', and parse the output to see if it contained 'aa', the authoritative flag. If it did, it was a good domain. If it didn't, it was a bad domain. Timeouts and non-authoritative responses were counted separately. I then had three variables for each domain - goodResponses, badResponses, and timeouts. Domains where badResponses and timeouts were both 0 were considered 'good'. Domains where goodResponses and timeouts were both 0 were considered to have 'no good nameservers (but all responded)'. Else, domains where goodResponses was 0 were considered bad (noting that some queries timed out). Beyond that, if there was 1 or more badResponses, it was listed in the 'at least one bad non-auth NS' list. The processes that did this would fork out about 80 processes per host to run the digs; I was on a reasonably fast connection, so bandwidth shouldn't have become a problem as far as increasing the timeouts I got. I split the com zone file into 200,000 line sections and ran one section per host. I then stopped the stats collection server after every few runs to gather statistics. The queries occurred over a period of 12 hours between 3pm and 3am Pacific Time, Tuesday 2/9 - Wednesday 2/10. I've put up the code, results, and the logs of non-auth queries and timed-out queries at ftp1.dal.net:/pub/misc/domain-test/. The files haven't (and won't) propogate to the mirror sites. (Note, this machine will be switching IPs sometime this week, so there may be a period of an hour or two when the machine will be unreachable.) If I've got some sort of flaw in my logic, please let me know; I'm willing to correct it and run the test again. But it looks right. I haven't tested the net/edu/org domains, but I suspect that since folks using those are slightly more clued than the folks using .com, the numbers of bogus domains will be lower. And if it is, it means that 17% of the folks on the internet are paying for domains that don't work. Either that, or something else is broken. I'm posting here because I feel it is an operational issue; that, plus I feel there're more folks here who can and will hammer at InterNIC to start doing something to enforce their policies that require real, authoritative nameservers. One last request - if you plan to use this data somewhere, *please* listen to any responses that may show up here explaining how it might be wrong, and *please* go through my methodology and find out for yourself if it looks right. I don't want to be responsible for any false/overinflated claims out there }:P . And please provide context, too, especially where the 'nameserver timed out' statistics are concerned. Anyhow, there it is. -dalvenjah -- Dalvenjah FoxFire (aka Sven Nielsen) Stupid people shouldn't breed. Founder, the DALnet IRC Network e-mail: dalvenjah@dal.net WWW: http://www.dal.net/~dalvenjah/ whois: SN90 Try DALnet! http://www.dal.net/
In message <19980211132424.19891@dragonlair.dal.net>, Dalvenjah FoxFire writes:
And if it is, it means that 17% of the folks on the internet are paying for domains that don't work. Either that, or something else is broken.
An alternate interpretation might be that 17% (or some portion thereof) of the domains in .COM are held by domain speculators who have no intention of paying for them and can't be bothered to provide DNS for them.
I'm posting here because I feel it is an operational issue; that, plus I feel there're more folks here who can and will hammer at InterNIC to start doing something to enforce their policies that require real, authoritative nameservers.
If nothing else, it might deter casual domain speculators. God forbid they actually have to configure something before they can register a domain... No doubt someone would just automate the process, though.
Actually, they are getting rid of that requirement:
8.The requirement for operational service from two DNS servers has been deleted.
http://rs.internic.net/domain-info/nic-rev03.html dan On Wed, 11 Feb 1998, Michael K. Sanders wrote:
In message <19980211132424.19891@dragonlair.dal.net>, Dalvenjah FoxFire writes:
I'm posting here because I feel it is an operational issue; that, plus I feel there're more folks here who can and will hammer at InterNIC to start doing something to enforce their policies that require real, authoritative nameservers.
If nothing else, it might deter casual domain speculators. God forbid they actually have to configure something before they can register a domain... No doubt someone would just automate the process, though.
Tons of people have used our domain name servers without permission. Of course, we don't set up a domain name without compensation... Dirk On Wed, Feb 11, 1998 at 06:42:49PM -0700, Michael K. Sanders wrote:
In message <19980211132424.19891@dragonlair.dal.net>, Dalvenjah FoxFire writes:
And if it is, it means that 17% of the folks on the internet are paying for domains that don't work. Either that, or something else is broken.
An alternate interpretation might be that 17% (or some portion thereof) of the domains in .COM are held by domain speculators who have no intention of paying for them and can't be bothered to provide DNS for them.
I'm posting here because I feel it is an operational issue; that, plus I feel there're more folks here who can and will hammer at InterNIC to start doing something to enforce their policies that require real, authoritative nameservers.
If nothing else, it might deter casual domain speculators. God forbid they actually have to configure something before they can register a domain... No doubt someone would just automate the process, though.
Tons of people have used our domain name servers without permission.
I periodically audit the zones which claim to be served here. For those which have been delegated lamely, I create a *primary* zone @ 7200 IN SOA my.server. hostmaster.my.server. ( 9401090 ; serial 7200 ; refresh every two hours 3600 ; retry every hour 2592000 ; expire in 30 days 7200 ) ; default TTL of one day NS MY.SERVER. MX 0 lame.delegation.to.MY.SERVER. * MX 0 lame.delegation.to.MY.SERVER. randy
Tons of people have used our domain name servers without permission.
I periodically audit the zones which claim to be served here. For those which have been delegated lamely, I create a *primary* zone
How do you find them all? You could check your DNS logs for lame delegations and collect a list, but that's not all that great. I agree that the Internic should check nameservers before putting up a domain, even though it's more resource intensive. In addition to controlling speculators, it might just prevent or at least detect honest mistakes. The CA-Domain registration authority used to do this but I don't think they do it anymore. While they're at it, I should be able to NAK a registration or domain modification so that it is cancelled if I don't want it on my nameservers. -Phil
On 02/12/98, Phillip Vandry <vandry@Mlink.NET> wrote:
How do you find them all? You could check your DNS logs for lame delegations and collect a list, but that's not all that great.
Get your nameserver's NIC handle (for example, my own ns.cybernothing.org is NS21329-HST), and try: whois "server NS21329-HST" This'll give you a list of up to 256 domains registered on that nameserver. If you have more, I hear the 'NIC will give you a list if you ask. -- J.D. Falk <jdfalk@vix.com> Vixie Enterprises http://www.vix.com/
On Thursday February 12, 1998, Phillip Vandry <vandry@Mlink.NET> had this to say about "Re: Broken domain statistics...":
Tons of people have used our domain name servers without permission.
I periodically audit the zones which claim to be served here. For those which have been delegated lamely, I create a *primary* zone
How do you find them all? You could check your DNS logs for lame delegations and collect a list, but that's not all that great.
You can find the first 256 domains registered to your DNS by using the command: whois "server <server-handle>" where server-handle is the handle assigned to your DNS host by InterNIC.
I agree that the Internic should check nameservers before putting up a domain, even though it's more resource intensive. In addition to controlling speculators, it might just prevent or at least detect honest mistakes.
In the latest domain-dispute policy to go into effect on the 25th, the document states in part: 8.The requirement for operational service from two DNS servers has been deleted.
The CA-Domain registration authority used to do this but I don't think they do it anymore.
While they're at it, I should be able to NAK a registration or domain modification so that it is cancelled if I don't want it on my nameservers.
Except for the fact that BEFORE-USE still hasn't been implemented :( In fact, I've often received the "Please ACK/NAK this request" letter *AFTER* receiving a message saying "Registration for the domain name shown below has been completed." NAK's rarely work in these cases unless I make a phone call to stop it.
-Phil
-- John-David Childs (JC612) Enterprise Internet Solutions System Administration 8707 E Florida Ave Suite 814 & Network Engineering Denver, CO 80231 http://www.nterprise.net As of this^H^H^H^H next week, passwords will be entered in Morse code.
At 11:48 12/02/98 -0500, you wrote:
I agree that the Internic should check nameservers before putting up a domain, even though it's more resource intensive. In addition to controlling speculators, it might just prevent or at least detect honest mistakes.
The CA-Domain registration authority used to do this but I don't think they do it anymore.
While they're at it, I should be able to NAK a registration or domain modification so that it is cancelled if I don't want it on my nameservers.
-Phil
Registration/delegation of domains here (i.e. com.au, net.au) requires the servers to be functional beforehand ... they are checked at form submission ... of course you can always pull it down later ... Damien
Additionally there is a problem where the InterNIC refuses to remove DNS entries even when the DNS site requests it. For example: # whois cyberpromo.com Cyber Promotions, Inc (CYBERPROMO-DOM) 8001 Castor Avenue Suite #127 Philadelphia, PA 19152 US Domain Name: CYBERPROMO.COM Administrative Contact, Technical Contact, Zone Contact: Wallace, Sanford (SW1708) domreg@CYBERPROMO.COM 215-628-9780 Billing Contact: Wallace, Sanford (SW1708) domreg@CYBERPROMO.COM 215-628-9780 Record last updated on 24-Jan-97. Record created on 26-Apr-96. Database last updated on 12-Feb-98 04:16:59 EDT. Domain servers in listed order: NS7.CYBERPROMO.COM 205.199.2.250 NS5.CYBERPROMO.COM 205.199.212.50 NS8.CYBERPROMO.COM 207.120.46.30 NS9.CYBERPROMO.COM 209.40.15.21 # whois 205.199.2.0 AGIS/Net99 (NETBLK-NET99-BLK4) 3601 Pelham Dearborn, MI 48124 Netname: NET99-BLK4 Netblock: 205.198.0.0 - 205.199.255.0 Maintainer: AGIS # whois 207.120.46.0 New Mellenium Cafe (NETBLK-NEW-MEL) NEW-MEL 207.120.46.0 - 207.120.46.63 Both AGIS and New Millenium would glady have the entries dropped if they could. Particularly since New Millenium is not able to use the 207.120.46.30 to date due to anti-spam attacks :-( On Thu, 12 Feb 1998, Phillip Vandry wrote:
From: Phillip Vandry <vandry@Mlink.NET> To: nanog@merit.edu
I agree that the Internic should check nameservers before putting up a domain, even though it's more resource intensive. In addition to controlling speculators, it might just prevent or at least detect honest mistakes.
The CA-Domain registration authority used to do this but I don't think they do it anymore.
While they're at it, I should be able to NAK a registration or domain modification so that it is cancelled if I don't want it on my nameservers.
-Phil
- James D. Wilson netsurf@sersol.com
On Thu, 12 Feb 1998, NetSurfer wrote:
NS7.CYBERPROMO.COM 205.199.2.250 NS5.CYBERPROMO.COM 205.199.212.50 NS8.CYBERPROMO.COM 207.120.46.30 NS9.CYBERPROMO.COM 209.40.15.21
Both AGIS and New Millenium would glady have the entries dropped if they could. Particularly since New Millenium is not able to use the 207.120.46.30 to date due to anti-spam attacks :-(
That brings up another question. What can/should be done with such "tainted" IP space? Who in their right mind would want space previously used by Cyberpromo? It's almost as if they destroyed the IP space they used. ------------------------------------------------------------------ Jon Lewis <jlewis@fdt.net> | Unsolicited commercial e-mail will Network Administrator | be proof-read for $199/message. Florida Digital Turnpike | ______http://inorganic5.fdt.net/~jlewis/pgp for PGP public key____
That brings up another question. What can/should be done with such "tainted" IP space? Who in their right mind would want space previously used by Cyberpromo? It's almost as if they destroyed the IP space they used.
Take well known "tainted" space and add it onto RFC1918 for the next revision ;-) -- Peter Galbavy @ Home in Wonderland http://www.wonderland.org/ http://www.whirl-y-gig.org.uk/ http://www.demon.net Be remembered not for your final destination, but for your journey.
Phillip Vandry <vandry@Mlink.NET> writes:
How do you find them all? You could check your DNS logs for lame delegations and collect a list, but that's not all that great.
NSI used to make the root zone files freely available via FTP; they are still up for FTP, but not without restriction. You can apply to NSI to get a login and password to FTP into the restricted zone host; if you have a decent justification for why you need the data, you can obtain one without much trouble. Once you have the root zone files, you can list all of the domains registered on your nameservers; I have a small set of perl scripts that massage the data into a more usable format. I've been meaning to tar them up and make them available for a while now. Or roll your own, it's not particularly difficult. [ ... ]
While they're at it, I should be able to NAK a registration or domain modification so that it is cancelled if I don't want it on my nameservers.
According to the original Guardian paper, setting the BEFORE-USE attribute on a host record would require the nameserver admin to ACK every domain registration before their nameserver could be listed for that domain. However, the BEFORE-USE attribute has never been implemented for contacts or host records. When the issue was raised on guard-talk@internic long long ago, an NSI rep explained that BEFORE-USE was never implemented because ``there was no consensus from the community that we should implement BEFORE-USE'', and because they were ``afraid that people would erroneously set BEFORE-USE on their nameservers or contacts and be deluged with mail requesting ACKs on new domain registrations, and new domain registrations would get slowed down, and the queues would back up forever'' (paraphrased). I may actually still have that thread in an old guard-talk mailbox, I should dig it up. -- Michael Handler <handler@sub-rosa.com> you might surprise yourself
An alternate interpretation might be that 17% (or some portion thereof) of the domains in .COM are held by domain speculators who have no intention of paying for them and can't be bothered to provide DNS for them.
And a third alternative would be domainholders who have registered and paid for domain names which they intend to use in the future but have not gotten around to setting up websites on them yet.
I'm posting here because I feel it is an operational issue; that, plus I feel there're more folks here who can and will hammer at InterNIC to start doing something to enforce their policies that require real, authoritative nameservers.
Internic's Domain Dispute Resolution Policy, Rev 2 [1], required two operational nameservers; in Rev 3 [2], they've dropped that requirement. They're still listing it in the FAQ [3], tho. It seems unclear whether they intend it to be a policy or not... Lisa 1: http://rs.internic.net/domain-info/nic-rev02.html 2: http://rs.internic.net/domain-info/nic-rev03.html 3: http://rs.internic.net/faq/name_servers.html Lisa Lorenzin Webmaster / System Administrator TriNet Services, Inc. lisa@trinet.com
participants (14)
-
Dalvenjah FoxFire -
Damien O'Rourke -
Dan Haskovec -
dirk＠power.net -
J.D. Falk -
John-David Childs -
Jon Lewis -
Lisa Lorenzin -
Michael Handler -
Michael K. Sanders -
NetSurfer -
Peter Galbavy -
Phillip Vandry -
Randy Bush