RIPE Database Proxy Service Issues
[Apologies for duplicate emails] Dear colleagues, There has been discussion on various mailing lists regarding the status of the RIPE Database Proxy Service. Before I address the issues that arose, I'd like to give you some background information on the service itself that may help with the discussions. Technical Background -------------------- To prevent the automatic harvesting of personal information (real names, email addresses, phone numbers) from the RIPE Database, there are PERSON and ROLE object query limits defined in the RIPE Database Acceptable Use Policy. This is set at 1,000 PERSON or ROLE objects per IP address per day. Queries that result in more than 1,000 objects with personal data being returned result in that IP address being blocked from carrying out queries for that day. Users of the RIPE Database have unlimited access to Network Information Centre (NIC)-related objects. They can use the -r flag in order to filter out personal objects and query NIC objects without any limitations. The RIPE Database Proxy Service allows websites to provide a third party interface to the RIPE Database. Without the proxy service, the third parties would quickly run into the limits set on RIPE Database queries. With the proxy service, we whitelist the third party IP address and ask them to pass their user's IP address to us, so limits are only set on the user's IP address, not the third party's. There is no technical way to ensure that the user IP addresses passed to us by the third party are valid. Potentially, third party users of the proxy service could harvest all personal data in the RIPE Database (approximately 2 million objects) in a matter of hours. To ensure that the RIPE NCC's Terms and Conditions are followed, we require a contract between the third party and the RIPE NCC. Users of the Proxy Service -------------------------- In the past ten years, the RIPE NCC has had 31 requests for the proxy service and over the past year, there have been only four active users of the service. Of these four, one is already a RIPE NCC member. NIC Information --------------- All NIC information is still available without access to the proxy service. In the normal presentation of whois data, there is a redirect system that allows users with a normal whois client to deal directly with the RIPE Database whois service. There is no need for a proxy service in this scenario. The proxy service is only necessary if the data needs to be presented in alternative forms, such as on a third party's website. The limits imposed on RIPE Database queries only apply to personal data. Users can always access NIC data in any form they like if they are happy not to receive personal data. On 6 March 2012, the RIPE NCC proposed to change the default behaviour of the query system to instead return only "ALLOWED" results if a user had reached their daily personal data query limit, but there was disagreement over this on the mailing list so the change was not implemented. The proposal is available at: http://www.ripe.net/ripe/mail/archives/db-wg/2012-March/003885.html Legal Considerations -------------------- The RIPE NCC operates under European Data Protection laws, so to avoid risk in this area we insist on having a contract with third parties who wish to use the proxy service. The RIPE NCC and its Executive Board believes that the proxy service should become a member service because it tightens the contractual relationship between the RIPE NCC and third parties. Currently, no such agreement that meets the EU Data Protection legislation is in place between the RIPE NCC and the proxy service users. In order to tighten the contractual relationship between the RIPE NCC and the Proxy service users, taking into account the recent approval of the Charging Scheme 2013 that caused a simplification of the contractual agreements between the RIPE NCC and its service users, the RIPE NCC offered to conclude the membership agreement for continuation of the service. Next Steps? ------------ The Executive Board approved changes to the draft version of the Activity Plan and Budget 2013, and the RIPE NCC published the final version on 13 December 2012: http://www.ripe.net/internet-coordination/news/announcements/ripe-ncc-activi... We do apologise, however, that the changes regarding the proxy service were not more explicitly communicated to the members and the RIPE community in advance of the final publication of the Activity Plan. The RIPE NCC asks that non-RIPE NCC member proxy service users become members but we propose to waive their membership fee until the discussion of the RIPE NCC Charging Scheme 2014 takes place. This will give the membership and community the opportunity to discuss the best way forward for the proxy service in the coming months while ensuring a strong contractual bond between the RIPE NCC and users of this service. In the meantime, there will be no changes to the proxy service and no loss of functionality for the community. The RIPE NCC and its Executive Board will return to its members with proposals for ways to ensure that their wishes are met with regard to service developments while allowing the RIPE NCC to be operate efficiently and responsively. If you have any comments on this issue, please direct them to the RIPE NCC Services Working Group mailing list <ncc-services-wg@ripe.net>. Best regards, Axel Pawlik Managing Director RIPE NCC
Hell Axel, On Jan 2, 2013, at 11:00 AM, Axel Pawlik <ripencc-management@ripe.net> wrote:
[Apologies for duplicate emails]
Dear colleagues,
There has been discussion on various mailing lists regarding the status of the RIPE Database Proxy Service.
We do apologise, however, that the changes regarding the proxy service were not more explicitly communicated to the members and the RIPE community in advance of the final publication of the Activity Plan.
Not being members, we obviously were not privy to these discussions or decisions. Not your fault, of course, just a reality.
The RIPE NCC asks that non-RIPE NCC member proxy service users become members but we propose to waive their membership fee until the discussion of the RIPE NCC Charging Scheme 2014 takes place. This will give the membership and community the opportunity to discuss the best way forward for the proxy service in the coming months while ensuring a strong contractual bond between the RIPE NCC and users of this service.
In the meantime, there will be no changes to the proxy service and no loss of functionality for the community.
I appreciate the decision and accommodation… And I am sure the community appreciates it. As users have no doubt realized, the proxy data continued to be available after Dec 31. We were waiting to see what the "DENIED" output looked like before we implemented our changes, so there was no impact. This too is appreciated. And thank you to the many community and RIPE members who offered and provided assistance and support. Thank you. Rodney Joffe CenterGate Research/GeekTools
This looks to be a happy ending. I thought we were going to get to see a fight. ;)
From my Galaxy Note II, please excuse any mistakes.
-------- Original message -------- From: Rodney Joffe <rjoffe@centergate.com> Date: 01/02/2013 2:51 PM (GMT-08:00) To: ripencc-management@ripe.net Cc: nanog@nanog.org Subject: Re: RIPE Database Proxy Service Issues Hell Axel, On Jan 2, 2013, at 11:00 AM, Axel Pawlik <ripencc-management@ripe.net> wrote:
[Apologies for duplicate emails]
Dear colleagues,
There has been discussion on various mailing lists regarding the status of the RIPE Database Proxy Service.
We do apologise, however, that the changes regarding the proxy service were not more explicitly communicated to the members and the RIPE community in advance of the final publication of the Activity Plan.
Not being members, we obviously were not privy to these discussions or decisions. Not your fault, of course, just a reality.
The RIPE NCC asks that non-RIPE NCC member proxy service users become members but we propose to waive their membership fee until the discussion of the RIPE NCC Charging Scheme 2014 takes place. This will give the membership and community the opportunity to discuss the best way forward for the proxy service in the coming months while ensuring a strong contractual bond between the RIPE NCC and users of this service.
In the meantime, there will be no changes to the proxy service and no loss of functionality for the community.
I appreciate the decision and accommodation… And I am sure the community appreciates it. As users have no doubt realized, the proxy data continued to be available after Dec 31. We were waiting to see what the "DENIED" output looked like before we implemented our changes, so there was no impact. This too is appreciated. And thank you to the many community and RIPE members who offered and provided assistance and support. Thank you. Rodney Joffe CenterGate Research/GeekTools
On Wed, Jan 02, 2013 at 05:00:14PM +0100, Axel Pawlik wrote:
To prevent the automatic harvesting of personal information (real names, email addresses, phone numbers) from the RIPE Database, there are PERSON and ROLE object query limits defined in the RIPE Database Acceptable Use Policy. This is set at 1,000 PERSON or ROLE objects per IP address per day. Queries that result in more than 1,000 objects with personal data being returned result in that IP address being blocked from carrying out queries for that day.
1. The technical measures you've outlined will not prevent, and have not prevented, anyone from automatically harvesting the entire thing. Anyone who owns or rents, for example, a 2M-member botnet, could easily retrieve the entire database using 1 query per IP address, spread out over a day/week/month/whatever. (Obviously more sophisticated approaches immediately suggest themselves.) Of course a simpler approach might be to buy a copy from someone who already has. I'm not picking on you, particularly: all WHOIS operators need to stop pretending that they can protect their public databases via rate-limiting. They can't. The only thing that they're doing is preventing NON-abusers from acquiring and using bulk data. 2. This presumes that the database is actually a target for abusers. I'm sure for some it is. But as a source, for example, of email addresses, it's a poor one: the number of addresses per thousand records is relatively small and those addresses tend to belong to people with clue, making them rather suboptimal choices for spamming/phishing/etc. Far richer targets are available on a daily basis simply by following the dataloss mailing list et.al. and observing what's been posted on pastebin or equivalent. These not only include many more email addresses, but often names, passwords (encrypted or not), and other personal details. And once again, the simpler approach of purchasing data is available. 3. Of course answering all those queries no doubt imposes significant load. Happily, one of the problems that we seem to have pretty much figured out how to solve is "serving up many copies of static content" because we have tools like web servers and rsync. So let me suggest that one way to make this much easier on yourselves is to export a (timestamped) static snapshot of the entire database once a day, and let the rest of the Internet mirror the hell out of it. Spreads out the load, drops the pretense that rate-limiting accomplishes anything useful, makes all the data available to everyone equally, and as long as everyone is aware that it's a snapshot and not a real-time answer, would probably suffice for most uses. (It would also come in handy during network events which render your service unreachable/unusable in whole or part, e.g., from certain parts of the world. Slightly-stale data is way better than no data.) The same thing should be done with all domain WHOIS data, too, BTW. The spammers/phishers/etc. have been getting copies of it for a very long time, whether by mass harvesting, exploiting security holes, paying off insiders, or other means, so it's security theater to pretend that limiting query rates has any value. ---rsk
participants (4)
-
Axel Pawlik
-
Rich Kulawiec
-
Rodney Joffe
-
Warren Bailey