GeekTools Whois Proxy and RIPE/RIPE-NCC
NANOG and ARIN Friends, 14 Years ago, at the suggestion of Jon Postel and some of the early participants in NANOG, we developed the GeekTools Whois proxy to make it easier for *us* - network security and abuse techs - to deal with the expanding number of gtlds and registrars and the varied whois servers that were appearing. The service had both a CLI and web interface. The service also led directly to the creation of whois-servers.net, which now seems to be part of a number of *nix distributions. The service has been up for 14 years, and over that time we have fulfilled the requirements of all of the whois server operators in regards to minimizing and stopping abuse of the GT whois proxy by domain scrapers, spammers, etc, while enabling the security folks to do their jobs. In some cases we have even written code to pass the ip address of the requestor to the whois server registry operator when they wanted to manage quota's directly. We think we have a really good relationship with all of the whois server operators, and I think we provide a useful service to the community, and is widely used. And in 14 years we have never been tarred as an enabler of abuse of "the whois" system. There has obviously never been any kind of charge or fee for using the proxy, or any of the other tools on GeekTools. In about 2002 we started placing a banner ad on the web interface page to offset some of the costs for the bandwidth that the proxy consumes. An average of about $70 a month for over the last 10 years. Actual bandwidth costs are higher than that of course, but it was a thought in 2002 that we had frankly forgotten about until recently. Two weeks ago RIPE-NCC, who provide the whois data for IP addresses in the RIPE region, informed us that based on decisions by their members, as of January 1st 2013, tomorrow, they would no longer provide whois proxy query response services to GeekTools unless we ponied up $1,800 a year for RIPE membership. I don't work very well above layer 7. It is what it is. So I wanted to let you know that as of midnight tonight, apparently, you won't be able to use GeekTools for RIPE related queries. If you have automated scripts, and you are one of the users who has expanded access to GeekTools, you'll need to find an alternative for RIPE queries *today*. My guess is that you will be able to query RIPE directly, once you have worked out that the address space is within RIPE's assignments. I think its wrong to have to pay for whois data that is part of a community resource . So I won't do it.
Rodney, On Dec 31, 2012, at 7:41 AM, Rodney Joffe <rjoffe@centergate.com> wrote:
Two weeks ago RIPE-NCC, who provide the whois data for IP addresses in the RIPE region, informed us that based on decisions by their members, as of January 1st 2013, tomorrow, they would no longer provide whois proxy query response services to GeekTools unless we ponied up $1,800 a year for RIPE membership. ... I think its wrong to have to pay for whois data that is part of a community resource . So I won't do it.
I have to assume there is some sort of misunderstanding here as the actions on behalf of RIPE you describe are ... surprising. However, if there isn't a misunderstanding then I strongly agree with you. I'll be interested in seeing RIPE's side of the story... Regards, -drc
Hi David, On Dec 31, 2012, at 10:55 AM, David Conrad <drc@virtualized.org> wrote:
Rodney,
On Dec 31, 2012, at 7:41 AM, Rodney Joffe <rjoffe@centergate.com> wrote:
Two weeks ago RIPE-NCC, who provide the whois data for IP addresses in the RIPE region, informed us that based on decisions by their members, as of January 1st 2013, tomorrow, they would no longer provide whois proxy query response services to GeekTools unless we ponied up $1,800 a year for RIPE membership. ... I think its wrong to have to pay for whois data that is part of a community resource . So I won't do it.
I have to assume there is some sort of misunderstanding here as the actions on behalf of RIPE you describe are ... surprising. However, if there isn't a misunderstanding then I strongly agree with you.
I'll be interested in seeing RIPE's side of the story…
I am absolutely open to believing that I have misunderstood. The older I've gotten, the dumber I've realized I am ;-) The references I can provide (besides the notice from RIPE which you already have) appear to be: http://www.ripe.net/ripe/docs/ripe-558 , specifically 2.4.7 RIPE Database Proxy Service /rlj
Hi Rodney, From the looks of it, this decision was made by the RIPE NCC Executive Board rather than at the General Meeting. Inqueries will have to be made why this was decided, and what the consequences are. But, I don't expect a resolution to be reached in the next 6 hours. In the meantime you could consider setting up an irrd[1], redirect queries to that instance instead of whois.ripe.net, and keep it kind of fresh by feeding it ftp://ftp.ripe.net/ripe/dbase/ripe.db.gz on a daily basis. Kind regards, Job [1] http://www.irrd.net/ On Dec 31, 2012, at 4:41 PM, Rodney Joffe <rjoffe@centergate.com> wrote:
NANOG and ARIN Friends,
14 Years ago, at the suggestion of Jon Postel and some of the early participants in NANOG, we developed the GeekTools Whois proxy to make it easier for *us* - network security and abuse techs - to deal with the expanding number of gtlds and registrars and the varied whois servers that were appearing. The service had both a CLI and web interface.
The service also led directly to the creation of whois-servers.net, which now seems to be part of a number of *nix distributions.
The service has been up for 14 years, and over that time we have fulfilled the requirements of all of the whois server operators in regards to minimizing and stopping abuse of the GT whois proxy by domain scrapers, spammers, etc, while enabling the security folks to do their jobs. In some cases we have even written code to pass the ip address of the requestor to the whois server registry operator when they wanted to manage quota's directly. We think we have a really good relationship with all of the whois server operators, and I think we provide a useful service to the community, and is widely used. And in 14 years we have never been tarred as an enabler of abuse of "the whois" system.
There has obviously never been any kind of charge or fee for using the proxy, or any of the other tools on GeekTools. In about 2002 we started placing a banner ad on the web interface page to offset some of the costs for the bandwidth that the proxy consumes. An average of about $70 a month for over the last 10 years. Actual bandwidth costs are higher than that of course, but it was a thought in 2002 that we had frankly forgotten about until recently.
Two weeks ago RIPE-NCC, who provide the whois data for IP addresses in the RIPE region, informed us that based on decisions by their members, as of January 1st 2013, tomorrow, they would no longer provide whois proxy query response services to GeekTools unless we ponied up $1,800 a year for RIPE membership.
I don't work very well above layer 7. It is what it is. So I wanted to let you know that as of midnight tonight, apparently, you won't be able to use GeekTools for RIPE related queries. If you have automated scripts, and you are one of the users who has expanded access to GeekTools, you'll need to find an alternative for RIPE queries *today*. My guess is that you will be able to query RIPE directly, once you have worked out that the address space is within RIPE's assignments.
I think its wrong to have to pay for whois data that is part of a community resource . So I won't do it.
-- AS5580 - Atrato IP Networks
Hi Job, On Dec 31, 2012, at 11:46 AM, Job Snijders <job.snijders@atrato-ip.com> wrote:
Hi Rodney,
From the looks of it, this decision was made by the RIPE NCC Executive Board rather than at the General Meeting. Inqueries will have to be made why this was decided, and what the consequences are. But, I don't expect a resolution to be reached in the next 6 hours.
I don't expect it to be resolved in any different way at all, based on my experience over the last 20 years. We're not a RIPE member, so we have *zero* influence, and relevance for the RIP-NCC board.
In the meantime you could consider setting up an irrd[1], redirect queries to that instance instead of whois.ripe.net, and keep it kind of fresh by feeding it ftp://ftp.ripe.net/ripe/dbase/ripe.db.gz on a daily basis.
As far as bulk data, one *really* important aspect of GeekTools from day 1, is that we do not provide any actual data, we *only* proxy data. So there is no possibility that at any time we have stale data. We are a proxy, not a provider of data. Its what Jon told me to do 14 years ago, and its what we have stuck to (I think we're the only whois proxy that has done this). If we give you an answer today, you can count on it being the authoritative answer as of this second. If we can't reach a whois server when you query us, we do *not* give you a cached answer. We store nothing. Important when chasing miscreants or problems. I don't want to change this.
Kind regards,
Job
On Dec 31, 2012, at 4:41 PM, Rodney Joffe <rjoffe@centergate.com> wrote:
NANOG and ARIN Friends,
14 Years ago, at the suggestion of Jon Postel and some of the early participants in NANOG, we developed the GeekTools Whois proxy to make it easier for *us* - network security and abuse techs - to deal with the expanding number of gtlds and registrars and the varied whois servers that were appearing. The service had both a CLI and web interface.
The service also led directly to the creation of whois-servers.net, which now seems to be part of a number of *nix distributions.
The service has been up for 14 years, and over that time we have fulfilled the requirements of all of the whois server operators in regards to minimizing and stopping abuse of the GT whois proxy by domain scrapers, spammers, etc, while enabling the security folks to do their jobs. In some cases we have even written code to pass the ip address of the requestor to the whois server registry operator when they wanted to manage quota's directly. We think we have a really good relationship with all of the whois server operators, and I think we provide a useful service to the community, and is widely used. And in 14 years we have never been tarred as an enabler of abuse of "the whois" system.
There has obviously never been any kind of charge or fee for using the proxy, or any of the other tools on GeekTools. In about 2002 we started placing a banner ad on the web interface page to offset some of the costs for the bandwidth that the proxy consumes. An average of about $70 a month for over the last 10 years. Actual bandwidth costs are higher than that of course, but it was a thought in 2002 that we had frankly forgotten about until recently.
Two weeks ago RIPE-NCC, who provide the whois data for IP addresses in the RIPE region, informed us that based on decisions by their members, as of January 1st 2013, tomorrow, they would no longer provide whois proxy query response services to GeekTools unless we ponied up $1,800 a year for RIPE membership.
I don't work very well above layer 7. It is what it is. So I wanted to let you know that as of midnight tonight, apparently, you won't be able to use GeekTools for RIPE related queries. If you have automated scripts, and you are one of the users who has expanded access to GeekTools, you'll need to find an alternative for RIPE queries *today*. My guess is that you will be able to query RIPE directly, once you have worked out that the address space is within RIPE's assignments.
I think its wrong to have to pay for whois data that is part of a community resource . So I won't do it.
-- AS5580 - Atrato IP Networks
So we think we're working out the impact, and have a work-around for users. There seem to be more than a few hundred network operations groups (thats many of you on NANOG) that use GeekTools (we can tell by the NAT IP addresses, and the rate of queries) that will be affected. It seems that what RIPE is doing is removing the ability for us to query their whois server using the special format that passes "your" ip address to RIPE in our queries that go to them. This was how they satisfied themselves that if *you* were abusing the query limit, and we had not caught it, and were not already preemptively blocking you or rate limiting you, they could do it. I guess its their version of "trust, but verify". No argument from us. They are not alone. We do the same thing with AFRINIC and APNIC amongst RIRs, nic.br as a TLD operator, and Network Solutions as a registrar. DENIC and a few others have asked us to provide queries in special formats, and we happily comply with all of these. We appreciate their efforts to enable us to help the community. And I think they've mostly been happy with us for the last 14 years or whatever. (BTW there are about 310 of them total at the moment that we're able to parse and identify and query for, as well as many more specially requested cases, like uk.com, au.com, etc. RIPE-NCC has decided to limit this to their members only. Not us. So they are now removing that from us. We will now be subject to their normal limits (whatever that is). When we reach our daily limit, we will be blocked. When we do that a few times, we will be permanently blacklisted. The good news is that if you query them yourselves, you'll be able to query them up to your daily individual limit before being blocked. So if you have been using us, and have never been blocked with RIPE queries, you will likely not be blocked when you query then direct (we have already been passing them your IP address so they can count and rate limit). The only difference is that now you you can make a single query for every TLD, every RWHOIS delegated server via the TLD whois server, and every RIR, and get a answer in one. Except if it ends up in RIPE land. Then you're on your own, walking their tree, etc. But you can do it manually. Later today, when we see how RIPE handles rejecting us, we'll write a script, and <sarcasm> without asking you all to become members and pay us $1,800 a year </sarcasm>, we'll post here, identifying the text we'll pass so that you can configure scripts to recognize the rejection, and handle the query in an exception routine. Also, more than 10 years ago, we created a windows program that loaded in the systray, and provided desktop capabilities. And we also made available the gpl'd unix source for people who wanted to run it locally. We haven't updated it for years, but many of you have it and did update, and that will not be affected, beyond the existing limitation you would be seeing - the app queries from your own IP address already. If any of you has been maintaining and upgrading/updating the app, and feels like sharing it, please do ;-). If you want, send it to us and we'll audit it (I know you won't mind in today's environment) and then add it to the geektools website. I guess I should also put together a smartphone app that uses the proxy as well… Anyway, enough noise for now. Apologies. And thanks to all of you who responded privately, with offers etc. Fortunately we don't need finance, or resources or support. I'm just happy it has helped for so long. Wishing you everything you want for yourselves in 2013 - the year of IPv6 and hundreds of new TLDs. Rodney and the CenterGate/GeekTools crew (yes, we're still around ;-)). . . . - . - On Dec 31, 2012, at 11:46 AM, Job Snijders <job.snijders@atrato-ip.com> wrote:
Hi Rodney,
From the looks of it, this decision was made by the RIPE NCC Executive Board rather than at the General Meeting. Inqueries will have to be made why this was decided, and what the consequences are. But, I don't expect a resolution to be reached in the next 6 hours.
In the meantime you could consider setting up an irrd[1], redirect queries to that instance instead of whois.ripe.net, and keep it kind of fresh by feeding it ftp://ftp.ripe.net/ripe/dbase/ripe.db.gz on a daily basis.
Kind regards,
Job
On Dec 31, 2012, at 4:41 PM, Rodney Joffe <rjoffe@centergate.com> wrote:
NANOG and ARIN Friends,
14 Years ago, at the suggestion of Jon Postel and some of the early participants in NANOG, we developed the GeekTools Whois proxy to make it easier for *us* - network security and abuse techs - to deal with the expanding number of gtlds and registrars and the varied whois servers that were appearing. The service had both a CLI and web interface.
The service also led directly to the creation of whois-servers.net, which now seems to be part of a number of *nix distributions.
The service has been up for 14 years, and over that time we have fulfilled the requirements of all of the whois server operators in regards to minimizing and stopping abuse of the GT whois proxy by domain scrapers, spammers, etc, while enabling the security folks to do their jobs. In some cases we have even written code to pass the ip address of the requestor to the whois server registry operator when they wanted to manage quota's directly. We think we have a really good relationship with all of the whois server operators, and I think we provide a useful service to the community, and is widely used. And in 14 years we have never been tarred as an enabler of abuse of "the whois" system.
There has obviously never been any kind of charge or fee for using the proxy, or any of the other tools on GeekTools. In about 2002 we started placing a banner ad on the web interface page to offset some of the costs for the bandwidth that the proxy consumes. An average of about $70 a month for over the last 10 years. Actual bandwidth costs are higher than that of course, but it was a thought in 2002 that we had frankly forgotten about until recently.
Two weeks ago RIPE-NCC, who provide the whois data for IP addresses in the RIPE region, informed us that based on decisions by their members, as of January 1st 2013, tomorrow, they would no longer provide whois proxy query response services to GeekTools unless we ponied up $1,800 a year for RIPE membership.
I don't work very well above layer 7. It is what it is. So I wanted to let you know that as of midnight tonight, apparently, you won't be able to use GeekTools for RIPE related queries. If you have automated scripts, and you are one of the users who has expanded access to GeekTools, you'll need to find an alternative for RIPE queries *today*. My guess is that you will be able to query RIPE directly, once you have worked out that the address space is within RIPE's assignments.
I think its wrong to have to pay for whois data that is part of a community resource . So I won't do it.
-- AS5580 - Atrato IP Networks
* Job Snijders:
In the meantime you could consider setting up an irrd[1], redirect queries to that instance instead of whois.ripe.net, and keep it kind of fresh by feeding it ftp://ftp.ripe.net/ripe/dbase/ripe.db.gz on a daily basis.
RIPE NCC strips all contact information from the bulk exports, so this isn't a replacement. But RIPE NCC promised that the bulk exports will include data related to the (new, mandatory) abuse-c field, so that data might be more useful again in the future.
Hi Rodney, Would support from a RIPE LIR be sufficient to keep the service up ? I'm pretty sure there isn't a requirement to register for a LIR membership if this is the only usage. As a RIPE LIR, we can have a look at what the options are if that would help. Have a good new year, Regards, Erik Bais A2B Internet Verstuurd vanaf mijn iPad Op 31 dec. 2012 om 16:41 heeft Rodney Joffe <rjoffe@centergate.com> het volgende geschreven:
NANOG and ARIN Friends,
14 Years ago, at the suggestion of Jon Postel and some of the early participants in NANOG, we developed the GeekTools Whois proxy to make it easier for *us* - network security and abuse techs - to deal with the expanding number of gtlds and registrars and the varied whois servers that were appearing. The service had both a CLI and web interface.
The service also led directly to the creation of whois-servers.net, which now seems to be part of a number of *nix distributions.
The service has been up for 14 years, and over that time we have fulfilled the requirements of all of the whois server operators in regards to minimizing and stopping abuse of the GT whois proxy by domain scrapers, spammers, etc, while enabling the security folks to do their jobs. In some cases we have even written code to pass the ip address of the requestor to the whois server registry operator when they wanted to manage quota's directly. We think we have a really good relationship with all of the whois server operators, and I think we provide a useful service to the community, and is widely used. And in 14 years we have never been tarred as an enabler of abuse of "the whois" system.
There has obviously never been any kind of charge or fee for using the proxy, or any of the other tools on GeekTools. In about 2002 we started placing a banner ad on the web interface page to offset some of the costs for the bandwidth that the proxy consumes. An average of about $70 a month for over the last 10 years. Actual bandwidth costs are higher than that of course, but it was a thought in 2002 that we had frankly forgotten about until recently.
Two weeks ago RIPE-NCC, who provide the whois data for IP addresses in the RIPE region, informed us that based on decisions by their members, as of January 1st 2013, tomorrow, they would no longer provide whois proxy query response services to GeekTools unless we ponied up $1,800 a year for RIPE membership.
I don't work very well above layer 7. It is what it is. So I wanted to let you know that as of midnight tonight, apparently, you won't be able to use GeekTools for RIPE related queries. If you have automated scripts, and you are one of the users who has expanded access to GeekTools, you'll need to find an alternative for RIPE queries *today*. My guess is that you will be able to query RIPE directly, once you have worked out that the address space is within RIPE's assignments.
I think its wrong to have to pay for whois data that is part of a community resource . So I won't do it.
Hi Erik, I appreciate the offer (a number of RIPE members have stepped forward). However I would not a) want this to in any way threaten your membership status - its possible I guess that this might violate the RIPE contract because it is a circumvention, and b) would not want special status - its important that the problem should be resolved for all the parties who are being affected and don't have a voice. GeekTools isn't special. I can easily afford RIPE membership. However its the principle, and the small folks that matter. I'm hoping that the good folks on the RIPE board think about the unintended detrimental consequences of their decision. I'm sure they didn't mean this to happen... Thanks again. Rodney On Dec 31, 2012, at 2:48 PM, Erik Bais <ebais@a2b-internet.com> wrote:
Hi Rodney,
Would support from a RIPE LIR be sufficient to keep the service up ?
I'm pretty sure there isn't a requirement to register for a LIR membership if this is the only usage.
As a RIPE LIR, we can have a look at what the options are if that would help.
Have a good new year,
Regards, Erik Bais A2B Internet
Verstuurd vanaf mijn iPad
Op 31 dec. 2012 om 16:41 heeft Rodney Joffe <rjoffe@centergate.com> het volgende geschreven:
NANOG and ARIN Friends,
14 Years ago, at the suggestion of Jon Postel and some of the early participants in NANOG, we developed the GeekTools Whois proxy to make it easier for *us* - network security and abuse techs - to deal with the expanding number of gtlds and registrars and the varied whois servers that were appearing. The service had both a CLI and web interface.
The service also led directly to the creation of whois-servers.net, which now seems to be part of a number of *nix distributions.
The service has been up for 14 years, and over that time we have fulfilled the requirements of all of the whois server operators in regards to minimizing and stopping abuse of the GT whois proxy by domain scrapers, spammers, etc, while enabling the security folks to do their jobs. In some cases we have even written code to pass the ip address of the requestor to the whois server registry operator when they wanted to manage quota's directly. We think we have a really good relationship with all of the whois server operators, and I think we provide a useful service to the community, and is widely used. And in 14 years we have never been tarred as an enabler of abuse of "the whois" system.
There has obviously never been any kind of charge or fee for using the proxy, or any of the other tools on GeekTools. In about 2002 we started placing a banner ad on the web interface page to offset some of the costs for the bandwidth that the proxy consumes. An average of about $70 a month for over the last 10 years. Actual bandwidth costs are higher than that of course, but it was a thought in 2002 that we had frankly forgotten about until recently.
Two weeks ago RIPE-NCC, who provide the whois data for IP addresses in the RIPE region, informed us that based on decisions by their members, as of January 1st 2013, tomorrow, they would no longer provide whois proxy query response services to GeekTools unless we ponied up $1,800 a year for RIPE membership.
I don't work very well above layer 7. It is what it is. So I wanted to let you know that as of midnight tonight, apparently, you won't be able to use GeekTools for RIPE related queries. If you have automated scripts, and you are one of the users who has expanded access to GeekTools, you'll need to find an alternative for RIPE queries *today*. My guess is that you will be able to query RIPE directly, once you have worked out that the address space is within RIPE's assignments.
I think its wrong to have to pay for whois data that is part of a community resource . So I won't do it.
participants (5)
-
David Conrad -
Erik Bais -
Florian Weimer -
Job Snijders -
Rodney Joffe