How do service providers get all the GeoIP companies to have correct information for their address ranges? Do they just pay them to update it? At first I thought it had to do with whois data, but my home Verizon IP whois lists Ashburn, VA, yet the GeoIP data shows my local city. We're trying to find a way to correct our GeoIP data for a specific IP range, but aren't sure what the best practices are for doing so. Any advice would be awesome! -- Ian Clark Network Engineer DreamHost
Did that Wiki page ever come back up? Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sep 24, 2015 7:01 PM, "Ian Clark" <ian.clark@dreamhost.com> wrote:
How do service providers get all the GeoIP companies to have correct information for their address ranges? Do they just pay them to update it? At first I thought it had to do with whois data, but my home Verizon IP whois lists Ashburn, VA, yet the GeoIP data shows my local city.
We're trying to find a way to correct our GeoIP data for a specific IP range, but aren't sure what the best practices are for doing so. Any advice would be awesome!
-- Ian Clark Network Engineer DreamHost
On Thu, Sep 24, 2015 at 7:29 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 25 Sep 2015, at 5:58, Ian Clark wrote:
Any advice would be awesome! There is no inherent correlation between IP addressing and geopolitical boundaries.
Maxmind does not concur. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Thu, Sep 24, 2015 at 08:47:56PM -0400, William Herrin wrote:
On Thu, Sep 24, 2015 at 7:29 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 25 Sep 2015, at 5:58, Ian Clark wrote:
Any advice would be awesome! There is no inherent correlation between IP addressing and geopolitical boundaries.
Maxmind does not concur.
Regards, Bill Herrin
I've recently SWIP'd some IP space to see if Maxmind would pick up the new location. 48 hours later it hasn't (just via their free, web-based query tool). Perhaps I need to be more patient. Ray
On Sep 24, 2015 5:54 PM, "Ray Van Dolson" <rvandolson@esri.com> wrote:
On Thu, Sep 24, 2015 at 08:47:56PM -0400, William Herrin wrote:
On Thu, Sep 24, 2015 at 7:29 PM, Roland Dobbins <rdobbins@arbor.net>
wrote:
On 25 Sep 2015, at 5:58, Ian Clark wrote:
Any advice would be awesome! There is no inherent correlation between IP addressing and geopolitical boundaries.
Maxmind does not concur.
Regards, Bill Herrin
I've recently SWIP'd some IP space to see if Maxmind would pick up the new location. 48 hours later it hasn't (just via their free, web-based query tool). Perhaps I need to be more patient.
Ray
We did the same thing about a week ago, including a correction request with maxmind to no avail, hence this thread. I have a feeling our patience won't be rewarded...
On Thu, Sep 24, 2015 at 9:12 PM, Ian Clark <ian.clark@dreamhost.com> wrote:
On Sep 24, 2015 5:54 PM, "Ray Van Dolson" <rvandolson@esri.com> wrote:
On Thu, Sep 24, 2015 at 08:47:56PM -0400, William Herrin wrote:
On Thu, Sep 24, 2015 at 7:29 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 25 Sep 2015, at 5:58, Ian Clark wrote:
Any advice would be awesome! There is no inherent correlation between IP addressing and geopolitical boundaries.
Maxmind does not concur.
Regards, Bill Herrin
I've recently SWIP'd some IP space to see if Maxmind would pick up the new location. 48 hours later it hasn't (just via their free, web-based query tool). Perhaps I need to be more patient.
We did the same thing about a week ago, including a correction request with maxmind to no avail, hence this thread. I have a feeling our patience won't be rewarded...
Maxmind would have to violate ARIN's rules to collect your geoip information through a feed of whois data. Those rules forbid republication of the data. Try contacting them directly. Then download the files they publish and check for yourself. https://support.maxmind.com/geoip-data-correction-request/ http://dev.maxmind.com/geoip/legacy/geolite/ When you've done these things and still haven't gotten satisfaction, perhaps then it's time to return here for a session of Name and Shame. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
Maxmind would have to violate ARIN's rules to collect your geoip information through a feed of whois data. Those rules forbid republication of the data.
Try contacting them directly. Then download the files they publish and check for yourself.
https://support.maxmind.com/geoip-data-correction-request/ http://dev.maxmind.com/geoip/legacy/geolite/
When you've done these things and still haven't gotten satisfaction, perhaps then it's time to return here for a session of Name and Shame.
Regards, Bill Herrin
Well, I have already submitted a correction request and waited for their stated update cycle to happen. Where do GeoIP companies get their data, if not whois records?
On Thu, Sep 24, 2015 at 9:33 PM, Ian Clark <ian.clark@dreamhost.com> wrote:
Where do GeoIP companies get their data, if not whois records?
I would assume that they query whois for one of their sources. They don't have to enter any contract with ARIN to do so but they also can't promptly collect any sizable portion database that way. That isn't the same as signing up for bulk whois access (https://www.arin.net/resources/request/bulkwhois.html). I imagine they also do traceroutes to identify the last known location in the route. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Thu, Sep 24, 2015 at 09:41:42PM -0400, William Herrin wrote:
On Thu, Sep 24, 2015 at 9:33 PM, Ian Clark <ian.clark@dreamhost.com> wrote:
Where do GeoIP companies get their data, if not whois records?
I would assume that they query whois for one of their sources. They don't have to enter any contract with ARIN to do so but they also can't promptly collect any sizable portion database that way. That isn't the same as signing up for bulk whois access (https://www.arin.net/resources/request/bulkwhois.html).
I imagine they also do traceroutes to identify the last known location in the route.
Regards, Bill Herrin
I assumed it must be based off of WHOIS. The IP space I'm working with is in the midwest (US). The address associated with it is from our primary IP block out here in California, which it would have only been able to gather from WHOIS. If it had gone off the last hop, presumably it would have seen that as something a little closer to the real location rather than *exactly* where our primary environment is. :) Ray
On Thu, Sep 24, 2015 at 9:45 PM, Ray Van Dolson <rvandolson@esri.com> wrote:
I assumed it must be based off of WHOIS. The IP space I'm working with is in the midwest (US). The address associated with it is from our primary IP block out here in California, which it would have only been able to gather from WHOIS. If it had gone off the last hop, presumably it would have seen that as something a little closer to the real location rather than *exactly* where our primary environment is. :)
They could also do RDNS lookups and then see what rwhois says about the domain. They could purchase sales records from online retailers. Hey guys, give us the IP address, city, state and zip code for each sale; we'll pay you a nickle each. Then correlate that with BGP announcements that show the range of impacted addresses. They could convince folks to install web browser plugins which give the users rewards in exchange for ceding personal information. Or buy data from a company which does. -Bill -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
They could purchase sales records from online retailers. Hey guys, give us the IP address, city, state and zip code for each sale; we'll pay you a nickle each. Then correlate that with BGP announcements that show the range of impacted addresses.
After looking more into the geo ip topic, I totally noticed, geo data should NOT correlate with BGP data at all! There are a couple of geo ip services that are doing it like you described, but IMO it's wrong. See big telco's announcing /12's and having these IPs spread all over the country. On 25.09.2015 at 03:51 William Herrin wrote:
On Thu, Sep 24, 2015 at 9:45 PM, Ray Van Dolson <rvandolson@esri.com> wrote:
I assumed it must be based off of WHOIS. The IP space I'm working with is in the midwest (US). The address associated with it is from our primary IP block out here in California, which it would have only been able to gather from WHOIS. If it had gone off the last hop, presumably it would have seen that as something a little closer to the real location rather than *exactly* where our primary environment is. :)
They could also do RDNS lookups and then see what rwhois says about the domain.
They could purchase sales records from online retailers. Hey guys, give us the IP address, city, state and zip code for each sale; we'll pay you a nickle each. Then correlate that with BGP announcements that show the range of impacted addresses.
They could convince folks to install web browser plugins which give the users rewards in exchange for ceding personal information. Or buy data from a company which does.
-Bill
https://tools.ietf.org/html/rfc1876 (EXPERIMENTAL) There appears to be a way of associating a subnet in the IN-ADDR.ARPA domain to a FQDN, which could then be queries for LOC data. For single addresses, the domain owner could opt to include location data for their domain. For subnets, the operator can include location data at their option. Also, I would add one more field to the LOC RR: country code. This would be a two-byte value that is the standard two character ASCII country code. When missing, a value of binary zero would be returned on query.
I don't believe anyone is actually using the LOC RR, but maybe I'm wrong. This seems like the best way to store this type of data. I could see CDNs being able to leverage this along with edns-client-subnet to decrease page load times significantly. How is this still an issue? I mean, we have the means to fix this. Whoever a reverse zone is delegated to could easily update the LOC record to provide this info. They can make the LOC record as "fuzzy" as they feel comfortable by zeroing out the minutes and seconds, as the LOC record is just a set of GPS coordinates. Who better to report the physical location of a network than that network's operators. I think country code would be a nice addition to the LOC record though. Sincerely, Clay Curtis On Fri, Sep 25, 2015 at 6:36 AM, Stephen Satchell <list@satchell.net> wrote:
https://tools.ietf.org/html/rfc1876 (EXPERIMENTAL)
There appears to be a way of associating a subnet in the IN-ADDR.ARPA domain to a FQDN, which could then be queries for LOC data. For single addresses, the domain owner could opt to include location data for their domain. For subnets, the operator can include location data at their option.
Also, I would add one more field to the LOC RR: country code. This would be a two-byte value that is the standard two character ASCII country code. When missing, a value of binary zero would be returned on query.
I don't believe anyone is either. We looked at it as well and after reviewing logs from our authoritative DNS server responsible for our in-addr.arpa zones, we saw zero queries for LOC records. Ray On Fri, Sep 25, 2015 at 10:43:13AM -0400, Clay Curtis wrote:
I don't believe anyone is actually using the LOC RR, but maybe I'm wrong. This seems like the best way to store this type of data. I could see CDNs being able to leverage this along with edns-client-subnet to decrease page load times significantly. How is this still an issue? I mean, we have the means to fix this. Whoever a reverse zone is delegated to could easily update the LOC record to provide this info. They can make the LOC record as "fuzzy" as they feel comfortable by zeroing out the minutes and seconds, as the LOC record is just a set of GPS coordinates. Who better to report the physical location of a network than that network's operators. I think country code would be a nice addition to the LOC record though.
Sincerely,
Clay Curtis
On Fri, Sep 25, 2015 at 6:36 AM, Stephen Satchell <list@satchell.net> wrote:
https://tools.ietf.org/html/rfc1876 (EXPERIMENTAL)
There appears to be a way of associating a subnet in the IN-ADDR.ARPA domain to a FQDN, which could then be queries for LOC data. For single addresses, the domain owner could opt to include location data for their domain. For subnets, the operator can include location data at their option.
Also, I would add one more field to the LOC RR: country code. This would be a two-byte value that is the standard two character ASCII country code. When missing, a value of binary zero would be returned on query.
On Fri, 25 Sep 2015 10:43:13 -0400, Clay Curtis said:
I don't believe anyone is actually using the LOC RR, but maybe I'm wrong. This seems like the best way to store this type of data. I could see CDNs being able to leverage this along with edns-client-subnet to decrease page load times significantly.
How is knowing the physical location going to decrease page load times? Hint: I live all of 3 miles from my office. But home is deep in Comcast cable territory, while office is hanging off connections to Ashburn and Atlanta. So from home to office, packets have to go 400 miles north or south, then back. If Akamai decided to serve me a page at home out of the Akamai servers at work because it's only 3 miles away, it just added 25ms to each RTT it has to take. Which is why Akamai (and any other *sane* CDN) make their decisions based on network topology, not physical location....
You're example is one specific case. I'm not advocating that it works in every case, or that geo-location should be used in routing decisions exclusively. I have dealt with cases in which a CDN responded to a client request with a resource on another continent, thus having to cross an ocean and adding considerable latency, when there was a POP on that continent. If the CDN could have LOC information that is more accurate/updated, it could allow them to make a better decision and direct a client to a resource that is physically closer. It's another piece of information. Clearly Maxmind or whatever other current means there are to determine physical location are not ideal. Clay On Fri, Sep 25, 2015 at 12:44 PM, <Valdis.Kletnieks@vt.edu> wrote:
I don't believe anyone is actually using the LOC RR, but maybe I'm wrong. This seems like the best way to store this type of data. I could see CDNs being able to leverage this along with edns-client-subnet to decrease
On Fri, 25 Sep 2015 10:43:13 -0400, Clay Curtis said: page
load times significantly.
How is knowing the physical location going to decrease page load times?
Hint: I live all of 3 miles from my office. But home is deep in Comcast cable territory, while office is hanging off connections to Ashburn and Atlanta. So from home to office, packets have to go 400 miles north or south, then back. If Akamai decided to serve me a page at home out of the Akamai servers at work because it's only 3 miles away, it just added 25ms to each RTT it has to take.
Which is why Akamai (and any other *sane* CDN) make their decisions based on network topology, not physical location....
On Fri, 25 Sep 2015 13:39:22 -0400, Clay Curtis said:
exclusively. I have dealt with cases in which a CDN responded to a client request with a resource on another continent, thus having to cross an ocean and adding considerable latency, when there was a POP on that continent.
And what was the root cause for the CDN to misfire that badly? I hereby submit the hypothesis that if a CDN's data tables are so dorked up that they're serving the data from the wrong continent, adding physical location probably won't help, and may make things even worse. (For instance, consider a location in Alaska, where the *closest* CDN may be one in the northwest part of Canada - specifically put there because the network is at the wrong end of a satellite link. You almost certainly want to skip that one and hit one in Seattle or someplace similar....)
If the CDN could have LOC information that is more accurate/updated, it could allow them to make a better decision and direct a client to a resource that is physically closer.
You don't want the one that's physically closest. You want the one that's netwise closest.
You will find that it takes years before every site out there updated their copy of whatever geo database they are using. Regards Baldur
With a new block it took less than a week. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sep 25, 2015 11:36 AM, "Baldur Norddahl" <baldur.norddahl@gmail.com> wrote:
You will find that it takes years before every site out there updated their copy of whatever geo database they are using.
Regards
Baldur
On 25 September 2015 at 17:52, Josh Luthman <josh@imaginenetworksllc.com> wrote:
With a new block it took less than a week.
On Sep 25, 2015 11:36 AM, "Baldur Norddahl" <baldur.norddahl@gmail.com> wrote:
You will find that it takes years before every site out there updated their copy of whatever geo database they are using.
Our original /22 block from RIPE has never had any geo IP issues. Everyone seems to recognize the country correctly from day 1. However our transferred blocks from Romania are still have issues a year later. Every geo IP database knows the correct location. But nothing forces customers of the database to actually keep their copy updated. So we keep getting complaints that this or that site is reporting the wrong country. Some of those sites are big sites that should know better. It is impossible to hunt down every single e-commerce site out there and force them to update their geo IP database. A lot of them have no clue what you are talking about. They either bought a hosted solution, and the tech guy you really want to talk to is somewhere else. Or they used contractors to build the system and now it is on auto pilot. Or you simply can't get through customer support to the tech people that know what geo IP actually is. At the end of the day, you are not going to waste too much time trying to beat clue into the heads of people running some niche site specialising in selling pink shoes, that refused to take the order from one customer because their brain dead e-commerce solution believes the customer is located in the wrong country. Regards, Baldur
On 25 Sep 2015, at 7:47, William Herrin wrote:
Maxmind does not concur.
<https://news.ycombinator.com/item?id=7888280> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
I love OVH where they ask where you want your IP space to be geolocated, but it’s still France/Canada… Why ask, I guess it worked in the past? Sincerely, Eric Tykwinski TrueNet, Inc. P: 610-429-8300 F: 610-429-3222
On Sep 24, 2015, at 8:55 PM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 25 Sep 2015, at 7:47, William Herrin wrote:
Maxmind does not concur.
<https://news.ycombinator.com/item?id=7888280>
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Is there anyone here who has successfully changed their GeoIP data for a subset of their ARIN allocation? How do service providers get all the GeoIP companies to have correct information for their address ranges? Do they just pay them to update it? At first I thought it had to do with whois data, but my home Verizon IP whois lists Ashburn, VA, yet the GeoIP data shows my local city. We're trying to find a way to correct our GeoIP data for a specific IP range, but aren't sure what the best practices are for doing so. Any advice would be awesome! -- Ian Clark Network Engineer DreamHost
It is a big pain to do so. We did a couple of times in the past and always took us many months. On 25.09.2015 at 03:48 Ian Clark wrote:
Is there anyone here who has successfully changed their GeoIP data for a subset of their ARIN allocation? How do service providers get all the GeoIP companies to have correct information for their address ranges? Do they just pay them to update it? At first I thought it had to do with whois data, but my home Verizon IP whois lists Ashburn, VA, yet the GeoIP data shows my local city.
We're trying to find a way to correct our GeoIP data for a specific IP range, but aren't sure what the best practices are for doing so. Any advice would be awesome!
It'd really help if some larger content providers would give LIR's some tools to effectively manage GeoTargeting within IP allocations and the subnets therein that they own. In my experience it takes anywhere between 2 weeks and 6 month to get IP blocks effectively Geo-targeted. Google is one of the harder but most visible ones, their online form to change it doesn't do anything. Going through their NOC usually fixes an issue within 2-3 weeks. (Google engineers, idea to add tools for that at https://isp.google.com on a webmaster tool-ish management interface?) Akamai is usually pretty fast and changing maxmind will eventually follow up a lot of the remaining sites. But it's a slow process. Our strategy is generally to change the RIR DB entry to include the correct country and geoloc fields. Followed by a maxmind update request and then some direct strings pulled from friendly operator colleagues and a mail to the Google NOC. On 25/09/15 09:19, Fred Hollis wrote:
It is a big pain to do so. We did a couple of times in the past and always took us many months.
On 25.09.2015 at 03:48 Ian Clark wrote:
Is there anyone here who has successfully changed their GeoIP data for a subset of their ARIN allocation? How do service providers get all the GeoIP companies to have correct information for their address ranges? Do they just pay them to update it? At first I thought it had to do with whois data, but my home Verizon IP whois lists Ashburn, VA, yet the GeoIP data shows my local city.
We're trying to find a way to correct our GeoIP data for a specific IP range, but aren't sure what the best practices are for doing so. Any advice would be awesome!
-- Jeroen Wunnink IP NOC Manager - Hibernia Networks Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300 Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623 jeroen.wunnink@hibernianetworks.com www.hibernianetworks.com This e-mail and any attachments thereto is intended only for use by the addressee(s) named herein and may be proprietary and/or legally privileged. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, without the prior written permission of the sender is strictly prohibited. If you receive this e-mail in error, please immediately telephone or e-mail the sender and permanently delete the original copy and any copy of this e-mail, and any printout thereof. All documents, contracts or agreements referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may contain software viruses that could damage your own computer system. While Hibernia Networks has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage that you sustain as a result of software viruses. You should carry out your own virus checks before opening any attachment.
We have been using Maxmind's open source data set [0] on RIPEstat [1] now for quite a while and there have been quite many user complaints about the correctness of the location or the recency of the data. One reason can be that [0] is updated/released only once a month, usually beginning of the month. In some cases the Maxmind's online preview [2] - assuming the underlying data is the paid version - seemed to be more up-to-date. Getting BGP topology to geographical location right is not a trivial task for various reasons of which some were already stated here. From my experience with Maxmind I think that the RIR's Whois information is a starting point to bootstrap their database and from there Maxmind uses ping/traceroute triangulation, change requests... and keep in mind that Maxmind's main business field is e-commerce which allows them to correlate requests (IPs from customers) from e-shop site (with known geographical affiliation). Since many ISPs seem to suffer from misguided geo-ip information provided by different providers it would be a huge improvement to have at least one data set that allows the ISPs to provide location information to the IP space they own. Few years ago I heard of a project called OpenGeoFeed [3] but I don't know about its status. Christian [0] http://dev.maxmind.com/geoip/legacy/geolite/ [1] e.g. https://stat.ripe.net/widget/geoloc#w.resource=2001%3A67c%3A2e8%3A%3A%2F48 [2] https://www.maxmind.com/en/geoip-demo [3] https://www.opengeofeed.org/ On 25/09/15 03:48, Ian Clark wrote:
Is there anyone here who has successfully changed their GeoIP data for a subset of their ARIN allocation? How do service providers get all the GeoIP companies to have correct information for their address ranges? Do they just pay them to update it? At first I thought it had to do with whois data, but my home Verizon IP whois lists Ashburn, VA, yet the GeoIP data shows my local city.
We're trying to find a way to correct our GeoIP data for a specific IP range, but aren't sure what the best practices are for doing so. Any advice would be awesome!
participants (13)
-
Baldur Norddahl -
Christian Teuschel -
Clay Curtis -
Eric Tykwinski -
Fred Hollis -
Ian Clark -
Jeroen Wunnink -
Josh Luthman -
Ray Van Dolson -
Roland Dobbins -
Stephen Satchell -
Valdis.Kletnieks＠vt.edu -
William Herrin