Upcoming Improvements to ARIN's Directory Service
Hi This was posted on arin-announce this morning as many of you may be interested: ARIN is pleased to announce that it plans to deploy an improved Whois service called Whois-RWS on 26 June 2010. Included in the deployment are the following services that provide the general public with access to ARIN's registration data. * a RESTful Web Service (RWS) * a NICNAME/WHOIS port 43 service * a user-friendly web site (http://whois.arin.net) A demo of this service has been available since October 2009. The demonstration service will be available at http://whoisrws-demo.arin.net until the production service is deployed on 26 June 2010. When using Whois-RWS you will notice some differences in behavior for certain queries and corresponding result sets on the NICNAME/WHOIS port 43 service. ARIN will make a separate announcement on 11 June when it publishes detailed documentation on these differences along with the demonstration service update. ARIN continues to welcome community participation on the Whois-RWS mailing list, and we invite you to subscribe and share your thoughts and suggestions at: http://lists.arin.net/mailman/listinfo/arin-whoisrws More detailed information on these changes and other future features that may impact the community at ARIN is available at: https://www.arin.net/features/ Regards, Mark Kosters Chief Technical Officer American Registry for Internet Numbers (ARIN)
I just found out that with the move to this new service that the bulk access FTP is going to be phased out. By design, there will be no way to automate the bulk download of this data. Is anyone else using the data in an environment that will be seriously impacted by this change? On Fri, Jun 4, 2010 at 3:05 PM, Mark Kosters <markk@arin.net> wrote:
Hi
This was posted on arin-announce this morning as many of you may be interested:
ARIN is pleased to announce that it plans to deploy an improved Whois service called Whois-RWS on 26 June 2010. Included in the deployment are the following services that provide the general public with access to ARIN's registration data.
* a RESTful Web Service (RWS) * a NICNAME/WHOIS port 43 service * a user-friendly web site (http://whois.arin.net)
A demo of this service has been available since October 2009. The demonstration service will be available at http://whoisrws-demo.arin.net until the production service is deployed on 26 June 2010.
When using Whois-RWS you will notice some differences in behavior for certain queries and corresponding result sets on the NICNAME/WHOIS port 43 service. ARIN will make a separate announcement on 11 June when it publishes detailed documentation on these differences along with the demonstration service update.
ARIN continues to welcome community participation on the Whois-RWS mailing list, and we invite you to subscribe and share your thoughts and suggestions at: http://lists.arin.net/mailman/listinfo/arin-whoisrws
More detailed information on these changes and other future features that may impact the community at ARIN is available at: https://www.arin.net/features/
Regards,
Mark Kosters Chief Technical Officer American Registry for Internet Numbers (ARIN)
On 6/10/2010 11:46, Jason Lewis wrote:
I just found out that with the move to this new service that the bulk access FTP is going to be phased out. By design, there will be no way to automate the bulk download of this data.
Is anyone else using the data in an environment that will be seriously impacted by this change?
Apparently we're supposed to be going all Web 2.0 now. ~Seth
On Thu, Jun 10, 2010 at 2:23 PM, Seth Mattinen <sethm@rollernet.us> wrote:
On 6/10/2010 11:46, Jason Lewis wrote:
I just found out that with the move to this new service that the bulk access FTP is going to be phased out. By design, there will be no way to automate the bulk download of this data.
Is anyone else using the data in an environment that will be seriously impacted by this change?
Apparently we're supposed to be going all Web 2.0 now.
~Seth
Nothing wrong with having a nicer interface, but hopefully not at the expense of bulk data. If it's a huge issue to support FTP data transfers, they could at least provide a means through the web service to get bulk data intelligently. -- Brandon Galbraith Voice: 630.492.0464
Hi ARIN is making significant improvements to our systems and services. ARIN encourages the community to look for upcoming features as details are available at: https://www.arin.net/features. I would like to clear up the confusion about the changes to access to Bulk Whois that have been discussed in this thread. The next release of ARIN Online on 26 June 2010 will include an easy way of automating bulk Whois reports. ARIN sent an announcement about this change to all current Bulk Whois recipients on 1 June 2010. The current legacy ftp service for Bulk Whois recipients will continue to operate until 31 August 2010. This should allow enough time to make the changes required to your scripts to migrate to the new solution. Regards, Mark Kosters ARIN CTO
Apparently we're supposed to be going all Web 2.0 now.
Web 2.0 can handle bulk transfers of data just fine. I wonder if this is somehow related to privacy and data protection laws. Just recently, RIPE announced that they were going to block bulk transfers as a result of data protection laws, presumably because some law has just changed. Obviously ARIN is under a different legal regime than RIPE, however data protection has recently been a hot button issue in the USA and it is possible that something similar will happen. Given the importance of case law in the USA, as opposed to legislation, I wouldn't be surprised if there was some sort of legal review going on. But again, as far as technology goes, HTTP is a superior file transfer protocol to FTP, so the move to Web 2.0 RESTful transactions over HTTP does not give any technical reason to stop bulk transfers. In fact, it may just be an oversight so you should really ask them Clearly, if nobody bothers to ask about bulk transfers, then nobody uses them and nobody cares, so shutting them down is the right thing to do. --Michael Dillon
It's very clear. I went back and forth with support, asking how to automate my bulk transfer with the new system. Me: Is the bulk data download going to be available for automated download. I can currently download the data daily from the ftp via a script. The new web page doesn't seem to support that. Support: No, there is no automation by design. I'm ok with whatever system they provide if the functionality stays the same. I don't understand what they gain by making a human login and download the file. On Thu, Jun 10, 2010 at 5:26 PM, Michael Dillon <wavetossed@googlemail.com> wrote:
Apparently we're supposed to be going all Web 2.0 now.
Web 2.0 can handle bulk transfers of data just fine.
I wonder if this is somehow related to privacy and data protection laws.
Just recently, RIPE announced that they were going to block bulk transfers as a result of data protection laws, presumably because some law has just changed. Obviously ARIN is under a different legal regime than RIPE, however data protection has recently been a hot button issue in the USA and it is possible that something similar will happen. Given the importance of case law in the USA, as opposed to legislation, I wouldn't be surprised if there was some sort of legal review going on.
But again, as far as technology goes, HTTP is a superior file transfer protocol to FTP, so the move to Web 2.0 RESTful transactions over HTTP does not give any technical reason to stop bulk transfers. In fact, it may just be an oversight so you should really ask them Clearly, if nobody bothers to ask about bulk transfers, then nobody uses them and nobody cares, so shutting them down is the right thing to do.
--Michael Dillon
I'm ok with whatever system they provide if the functionality stays the same. I don't understand what they gain by making a human login and download the file.
Accountability. If versions X and Y of database got abused (breach of ToS), and only user U has downloaded such versions, gotcha. Using honeytokens on the downloaded file can be interesting to quickly connect the dots: if one of the handles on the list is comeonspammer32767@wannahaveapieceofme.com, dynamically generated to match a download session, and suddenly this account starts to get spam... Rubens
On Thu, Jun 10, 2010 at 9:56 PM, Rubens Kuhl <rubensk@gmail.com> wrote:
comeonspammer32767@wannahaveapieceofme.com, dynamically generated to match a download session, and suddenly this account starts to get spam... well... yes.. doesn't help much if the token being abused is the admin POC's phone number, however. A session-based generated token alone would not be a very robust form of accountability; it is only as good as the strength of the verification required to get an account (and the confidence that multiple accounts do not collude).
A user might simply sign up twice or more using fake signup details, they can compare their different downloads, and screen out any records that changed between the several sessions. e.g. grab 3 copies of thesame file (that were obtained using 3 different logins, from 3 different countries), run a 3-way diff, strip out any lines that changed. Any session-specific token would be excluded... That is, if obtaining such a listing of e-mail addresses is even is worth it to them. Maybe it is not. Maybe the more common abuse is manual solicitation by a human being, trying to sell some high-margin product targeted at enterprises in the directory, who can easily recognize "comeonspammer" and stay away. I doubt the average POC is going to be duped by the pill salesmen, latest money making scam, too-good-to-be-true offer, go phish attempt, or other standardized junk mail. -- -J
On Jun 10, 2010, at 6:16 PM, Jason Lewis wrote:
It's very clear. I went back and forth with support, asking how to automate my bulk transfer with the new system.
Me: Is the bulk data download going to be available for automated download. I can currently download the data daily from the ftp via a script. The new web page doesn't seem to support that. Support: No, there is no automation by design.
I'm ok with whatever system they provide if the functionality stays the same. I don't understand what they gain by making a human login and download the file.
Jason - My apologies for the confusion over this when you called in; while we had briefed the support team on RESTful WHOIS, we hadn't covered the updated Bulk Whois interface as it is a bit of a specialized item and coming out on the next release of ARIN Online due to its need for "API key" support. The 26 June release of ARIN Online will allow you to create and manage these keys, which in turn may be used in RESTful calls (and email templates!) for authentication. A brief overview of this feature was provided at the ARIN Toronto meeting and is available here: https://www.arin.net/participate/meetings/reports/ARIN_XXV/PDF/Tuesday/Newto... We will rollout the API key functionality and Bulk Whois via the RESTful interface with this next release of ARIN Online on 26 June, and this will allow the Bulk Whois data to be downloaded directly without logging into ARIN Online by using a RESTful HTTP request containing your "API key". As Mark Kosters noted in his message, we did contact current Bulk Whois users ahead of time about these changes, but if you were missed or have any questions about the change, please don't hesitate to contact myself or Mark directly. Thanks! /John John Curran President and CEO ARIN Begin forwarded message:
From: Mark Kosters <markk@arin.net> Date: June 11, 2010 3:17:49 PM EDT To: nanog@nanog.org Subject: Re: Upcoming Improvements to ARIN's Directory Service
Hi
ARIN is making significant improvements to our systems and services. ARIN encourages the community to look for upcoming features as details are available at: https://www.arin.net/features.
I would like to clear up the confusion about the changes to access to Bulk Whois that have been discussed in this thread. The next release of ARIN Online on 26 June 2010 will include an easy way of automating bulk Whois reports. ARIN sent an announcement about this change to all current Bulk Whois recipients on 1 June 2010. The current legacy ftp service for Bulk Whois recipients will continue to operate until 31 August 2010. This should allow enough time to make the changes required to your scripts to migrate to the new solution.
Regards, Mark Kosters ARIN CTO
My apologies for the confusion over this when you called in; while we had briefed the support team on RESTful WHOIS, we hadn't covered the updated Bulk Whois interface as it is a bit of a specialized item and coming out on the next release of ARIN Online due to its need for "API key" support. The 26 June release of ARIN Online will allow you to create and manage these keys, which in turn may be used in RESTful calls (and email templates!) for authentication. A brief overview of this feature was provided at the ARIN Toronto meeting and is available here:
https://www.arin.net/participate/meetings/reports/ARIN_XXV/PDF/Tuesday/Newto...
We will rollout the API key functionality and Bulk Whois via the RESTful interface with this next release of ARIN Online on 26 June, and this will allow the Bulk Whois data to be downloaded directly without logging into ARIN Online by using a RESTful HTTP request containing your "API key". As Mark Kosters noted in his message, we did contact current Bulk Whois users ahead of time about these changes, but if you were missed or have any questions about the change, please don't hesitate to contact myself or Mark directly.
john, today, a research batch script running periodic bulk whois work has a line something like ncftpget ftp://user:pass@ftp.arin.net/arin_db.txt.gz well, it can actually be simpler. for the web 9.3 impaired of us, could you describe the simple batch script line under the new improved system? thanks! randy
john,
today, a research batch script running periodic bulk whois work has a line something like
ncftpget ftp://user:pass@ftp.arin.net/arin_db.txt.gz
well, it can actually be simpler.
for the web 9.3 impaired of us, could you describe the simple batch script line under the new improved system?
Randy - You're going to have to get on ARIN Online at least once to generate an key (this means after June 26), but then accessing the data should be just as simple for a batch script (i.e. use curl or wget for this purpose). I've extracted the relevant draft info from the June 26 release documents and attached below. This is obviously subject to change until the release actually comes out... /John ---- DOWNLOADING USING AN API KEY ---- The report can be downloaded directly without logging into ARIN Online using a RESTful HTTP request containing your API key. The URL must look like: https://www.arin.net/public/rest/downloads/nvpr?apikey=YOUR-API-KEY There are a variety of ways to automate the retrieval of this report. For example, on a Linux system, where your API key is API-1111-2222-3333-4444, you can use the following 'curl' command to download the report file: curl https://www.arin.net/public/rest/downloads/nvpr?apikey=API-1111-2222-3333-44... > arin_nvpr.zip You can manage your API keys on the when logged into your ARIN Online account.
You're going to have to get on ARIN Online at least once to generate an key
i can probably survive this experience. is there a tee shirt? :)
The report can be downloaded directly without logging into ARIN Online using a RESTful HTTP request containing your API key. The URL must look like:
https://www.arin.net/public/rest/downloads/nvpr?apikey=YOUR-API-KEY
this looks quite doable. thank you! randy
On Jun 12, 2010, at 11:14 AM, Randy Bush wrote:
You're going to have to get on ARIN Online at least once to generate an key
i can probably survive this experience. is there a tee shirt? :)
Your request has been noted... ;-)
The report can be downloaded directly without logging into ARIN Online using a RESTful HTTP request containing your API key. The URL must look like:
https://www.arin.net/public/rest/downloads/nvpr?apikey=YOUR-API-KEY
this looks quite doable.
thank you!
Thank you (and Jason Lewis!) for pointing out the lack of actionable information regarding this announcement and its impact on Bulk Whois. I've chatted with Mark and Nate on the timing of this service change, and going forward we'll make sure to have replacement services fully deployed and verifiable by the community before announcing an end date for the current service. Thanks again! /John John Curran President and CEO ARIN
participants (9)
-
Brandon Galbraith -
James Hess -
Jason Lewis -
John Curran -
Mark Kosters -
Michael Dillon -
Randy Bush -
Rubens Kuhl -
Seth Mattinen