Is there a list of NIC (and other popular whois server) features (what can be searched on) and what data they provide (and what title they give it)? A quick search yields: http://www.ripe.net/ripe/docs/ripe-358 https://www.arin.net/resources/whoisrws/whois_diff.html https://www.apnic.net/apnic-info/whois_search/using-whois/searching/query-op... (In declining order of information - I also couldn't find the info for AFRINIC queries) I also couldn't find information on what fields they have (and obviously how they map to one another). There are also a few other whois servers around that I have no idea about: https://www.opensource.apple.com/source/adv_cmds/adv_cmds-149/whois/whois.c
On Jan 7, 2015, at 10:38 AM, shawn wilson <ag4ve.us@gmail.com> wrote:
Is there a list of NIC (and other popular whois server) features (what can be searched on) and what data they provide (and what title they give it)?
Heh, heh, heh. There are just about as many whois output formats as there are back-end data-stores. Note that I say “data-stores” rather than databases. Some of them aren’t. So when you say “title” I assume you’re referring to half of a key-value pair. A concept some large whois sources don’t have. So, you’re not running into a poorly-documented mystery, you’ve run afoul of one of the rotten armpits of the shub-Internet. -Bill
On Wed, Jan 7, 2015 at 1:53 PM, Bill Woodcock <woody@pch.net> wrote:
On Jan 7, 2015, at 10:38 AM, shawn wilson <ag4ve.us@gmail.com> wrote:
Is there a list of NIC (and other popular whois server) features (what can be searched on) and what data they provide (and what title they give it)?
Heh, heh, heh. There are just about as many whois output formats as there are back-end data-stores. Note that I say “data-stores” rather than databases. Some of them aren’t. So when you say “title” I assume you’re referring to half of a key-value pair. A concept some large whois sources don’t have.
Yes, I'm referring to mapping between key names.
So, you’re not running into a poorly-documented mystery, you’ve run afoul of one of the rotten armpits of the shub-Internet.
So there's no consensus between NICs for the information they should have in whois and what search mechanisms they should provide? I guess what you're saying is that whois is just a protocol definition and nothing else?
So, you’re not running into a poorly-documented mystery, you’ve run afoul of one of the rotten armpits of the shub-Internet.
So there's no consensus between NICs for the information they should have in whois and what search mechanisms they should provide? I guess what you're saying is that whois is just a protocol definition and nothing else?
Might not even qualify for "protocol definition", and RFC 3912 ACKs this: https://tools.ietf.org/html/rfc3912 Rubens
So, you’re not running into a poorly-documented mystery, you’ve run afoul of one of the rotten armpits of the shub-Internet.
So there's no consensus between NICs for the information they should have in whois and what search mechanisms they should provide? I guess what you're saying is that whois is just a protocol definition and nothing else?
Correct. It gets you a blob of text. Sometimes, a blob is just a blob. Other times, it contains what _appear_ to be key-value pairs, but are instead loosely-formatted text. Other times, it contains textually-represented key-value pairs that are programmatically generated from an actual database, and can thus be re-imported into another database. Depends what’s on the back end. -Bill
On Wed, Jan 7, 2015 at 3:07 PM, Bill Woodcock <woody@pch.net> wrote:
So, you’re not running into a poorly-documented mystery, you’ve run afoul of one of the rotten armpits of the shub-Internet.
So there's no consensus between NICs for the information they should have in whois and what search mechanisms they should provide? I guess what you're saying is that whois is just a protocol definition and nothing else?
Correct. It gets you a blob of text. Sometimes, a blob is just a blob. Other times, it contains what _appear_ to be key-value pairs, but are instead loosely-formatted text. Other times, it contains textually-represented key-value pairs that are programmatically generated from an actual database, and can thus be re-imported into another database. Depends what’s on the back end.
This is not the response I was looking for (and reading the RFC makes me feel even worse). Is there a better mechanism for querying NICs for host/owner information?
This is not the response I was looking for (and reading the RFC makes me feel even worse).
Is there a better mechanism for querying NICs for host/owner information?
There will be, one day. And the start (although not the whole journey) will be when this I-D follows the standard path all the way to STD: http://tools.ietf.org/html/draft-ietf-weirds-rdap-query-18.html Rubens
Scripting languages have modules that can parse many registrar whois formats. However, most are incomplete due to the plurality of output formats as stated above. I, and i suspect many others, wouls *love* to see a more concrete key value format drafted and enforced by ICANN. -AK On Jan 7, 2015 12:26 PM, "Rubens Kuhl" <rubensk@gmail.com> wrote:
This is not the response I was looking for (and reading the RFC makes me feel even worse).
Is there a better mechanism for querying NICs for host/owner information?
There will be, one day. And the start (although not the whole journey) will be when this I-D follows the standard path all the way to STD: http://tools.ietf.org/html/draft-ietf-weirds-rdap-query-18.html
Rubens
On Wed, Jan 7, 2015 at 3:32 PM, anthony kasza <anthony.kasza@gmail.com> wrote:
Scripting languages have modules that can parse many registrar whois formats. However, most are incomplete due to the plurality of output formats as stated above. I, and i suspect many others, wouls *love* to see a more concrete key value format drafted and enforced by ICANN.
Yes, that's what I was looking at. And that REST API looks nice... Though from what I read (admittedly not the whole doc yet) I didn't see it defined what type of data is returned, nor what data should be expected, which would leave me in the same place. If I'm only getting a blob back (that would be, I guess, internationalized at this point) I've still got to loop through with a regex expecting some type of key/value thing and concatenate data after a line break to the last value (probably removing spaces since they try to format it pretty).
Hi,
Scripting languages have modules that can parse many registrar whois formats. However, most are incomplete due to the plurality of output formats as stated above. I, and i suspect many others, wouls *love* to see a more concrete key value format drafted and enforced by ICANN.
ICANN can only 'enforce' stuff on the contracted parties, namely the gTLDs. And, in fact, they do so in contractual agreements (if you want the gory details, see Specification 4 of http://newgtlds.icann.org/sites/default/files/agreements/agreement-approved-... for registries and https://www.icann.org/resources/pages/approved-with-specs-2013-09-17-en#whoi... for registrars). ICANN can't 'enforce' anything on the ccTLDs or RIRs. Regards, -drc
CRISP is dead. RDAP is real. If people need to script, then RDAP is workable JSON and for once, has converged on sensible stuff in both names and numbers. the whois "problem" is a formalism owned by ICANN, but as DRC pointed out the WHOIS solution is dispersed. RPSL lies to one side btw. I wish it was a defined RDAP type. It may be work-in-progress but thats not clear. On Thu, Jan 8, 2015 at 11:38 AM, David Conrad <drc@virtualized.org> wrote:
Hi,
Scripting languages have modules that can parse many registrar whois formats. However, most are incomplete due to the plurality of output formats as stated above. I, and i suspect many others, wouls *love* to see a more concrete key value format drafted and enforced by ICANN.
ICANN can only 'enforce' stuff on the contracted parties, namely the gTLDs. And, in fact, they do so in contractual agreements (if you want the gory details, see Specification 4 of http://newgtlds.icann.org/sites/default/files/agreements/agreement-approved-... for registries and https://www.icann.org/resources/pages/approved-with-specs-2013-09-17-en#whoi... for registrars).
ICANN can't 'enforce' anything on the ccTLDs or RIRs.
Regards, -drc
On 07/01/2015 20:07, Bill Woodcock wrote:
Correct. It gets you a blob of text. Sometimes, a blob is just a blob. Other times, it contains what _appear_ to be key-value pairs, but are instead loosely-formatted text. Other times, it contains textually-represented key-value pairs that are programmatically generated from an actual database, and can thus be re-imported into another database. Depends what’s on the back end.
If only we could create a committee to fix whois... Nick
On 1/7/15 12:48 PM, Nick Hilliard wrote:
On 07/01/2015 20:07, Bill Woodcock wrote:
Correct. It gets you a blob of text. Sometimes, a blob is just a blob. Other times, it contains what _appear_ to be key-value pairs, but are instead loosely-formatted text. Other times, it contains textually-represented key-value pairs that are programmatically generated from an actual database, and can thus be re-imported into another database. Depends what’s on the back end. If only we could create a committee to fix whois... wierd, I've heard that before someplace. Nick
If only we could create a committee to fix whois...
Quite astonishingly, the IETF WEIRDS working group finished successfully, and its documents will be published as RFCs when they get through the editing queue in a month or two. The protocol is called RDAP, the queries are http, the results are json. ARIN, APNIC, and RIPE have prototypes already that are a lot easier to script than the text WHOIS. RDAP is also supposed to work for domain name WHOIS, but the timeline there is less clear. ICANN has hired cnnic to do a freeware version which we hope a lot of smaller registries will use. R's, John
On Wed, Jan 7, 2015 at 10:22 PM, John Levine <johnl@iecc.com> wrote:
ARIN, APNIC, and RIPE have prototypes already that are a lot easier to script than the text WHOIS.
Meaning the data structure is in place or they have a RDAP service up? If so, is it publicly accessible?
http://rdap.apnic.net/ redirects to a web page documenting service http://rdap.apnic.net/ip shows a json error response http://rdap.apnic.net/ip/203.119.0.0/24 shows the /24 record for 203.119.0.0/24 -G On Thu, Jan 8, 2015 at 1:59 PM, shawn wilson <ag4ve.us@gmail.com> wrote:
On Wed, Jan 7, 2015 at 10:22 PM, John Levine <johnl@iecc.com> wrote:
ARIN, APNIC, and RIPE have prototypes already that are a lot easier to script than the text WHOIS.
Meaning the data structure is in place or they have a RDAP service up? If so, is it publicly accessible?
ARIN, APNIC, and RIPE have prototypes already that are a lot easier to script than the text WHOIS.
Meaning the data structure is in place or they have a RDAP service up?
Both. ARIN's and RIPE's are based on early versions so the URLs and JSON aren't quite what RDAP says they should be yet.
If so, is it publicly accessible?
Google is your friend. R's, John
On Jan 7, 2015, at 10:38 AM, shawn wilson <ag4ve.us@gmail.com> wrote:
Is there a list of NIC (and other popular whois server) features (what can be searched on) and what data they provide (and what title they give it)?
Your best bet today is http://sourceforge.net/projects/phpwhois/ and from http://phpwhois.cvs.sourceforge.net/viewvc/phpwhois/phpwhois/ You will see there are nearly as many data mappers as there are TLDs… WEIRDS is supposed to fix the protocol, data presentation and the field names, but not what fields will be present (as I understand it).
On Jan 8, 2015 4:23 AM, "Franck Martin" <fmartin@linkedin.com> wrote:
On Jan 7, 2015, at 10:38 AM, shawn wilson <ag4ve.us@gmail.com> wrote:
Is there a list of NIC (and other popular whois server) features (what can be searched on) and what data they provide (and what title they give it)?
Your best bet today is http://sourceforge.net/projects/phpwhois/
and from http://phpwhois.cvs.sourceforge.net/viewvc/phpwhois/phpwhois/
Awesome thanks. That answers half of my original question (though this route was much more insightful than I thought). I can run with that (php isn't my language but the etl is pretty clear).
On 1/8/2015 09:02, shawn wilson wrote:
Awesome thanks. That answers half of my original question (though this route was much more insightful than I thought). I can run with that (php isn't my language but the etl is pretty clear).
There is a Python module if that is more your thing: https://pypi.python.org/pypi/pythonwhois -- Sadiq Saif https://staticsafe.ca
participants (12)
-
anthony kasza -
Bill Woodcock -
David Conrad -
Franck Martin -
George Michaelson -
joel jaeggli -
John Levine -
John R. Levine
-
Nick Hilliard -
Rubens Kuhl -
Sadiq Saif -
shawn wilson