Does anybody know what the threshold for google searches is before you get the captcha?I am trying to decide if I need to break up the overload NAT to a pool. -thx
On Fri, Feb 26, 2016 at 3:01 PM, Philip Lavine via NANOG <nanog@nanog.org> wrote:
Does anybody know what the threshold for google searches is before you get the captcha?I am trying to decide if I need to break up the overload NAT to a pool.
There isn't a threshold -- if you send automated searches from an IP, then it gets blocked (for a while). So... this comes down to how much you trust your machines/users. If you're a company with managed systems, then you can have thousands of users share the same IP without problems. But if you're an ISP, you'll likely run into problems much earlier (since users like their malware). Some tips: - if you do NAT: try to partition users into pools so one abusive user can't get all your external IPs blocked - if you have a proxy: make sure it inserts the X-Forwarded-For header, and is restricted to your own users - if you're an ISP: IPv6 will allow each user to have their own /64, which avoids shared-fate from abusive ones Damian (responsible for DDoS defense) -- Damian Menscher :: Security Reliability Engineer :: Google :: AS15169
I have about 2000 users behind a single NAT. I have been looking at netflow, URL filter logs, IDS logs, etc. The traffic seems to be legit. I am going to move more users to IPv6 and divide some of the subnets into different NATS and see if that alleviates the traffic load. Thanks for the advice. -Philip From: Damian Menscher <damian@google.com> To: Philip Lavine <source_route@yahoo.com> Cc: "nanog@nanog.org" <nanog@nanog.org> Sent: Friday, February 26, 2016 6:05 PM Subject: Re: google search threshold On Fri, Feb 26, 2016 at 3:01 PM, Philip Lavine via NANOG <nanog@nanog.org> wrote: Does anybody know what the threshold for google searches is before you get the captcha?I am trying to decide if I need to break up the overload NAT to a pool. There isn't a threshold -- if you send automated searches from an IP, then it gets blocked (for a while). So... this comes down to how much you trust your machines/users. If you're a company with managed systems, then you can have thousands of users share the same IP without problems. But if you're an ISP, you'll likely run into problems much earlier (since users like their malware). Some tips: - if you do NAT: try to partition users into pools so one abusive user can't get all your external IPs blocked - if you have a proxy: make sure it inserts the X-Forwarded-For header, and is restricted to your own users - if you're an ISP: IPv6 will allow each user to have their own /64, which avoids shared-fate from abusive ones Damian (responsible for DDoS defense)-- Damian Menscher :: Security Reliability Engineer :: Google :: AS15169
FWIW I have seen the captchas more often on IPv6 both from home and the office than when both networks were using a single shared IPv4; not sure if this is just related to chronology or a real effect. Once a month or so I seem to get them for a couple of days, then they go away. No idea what's triggering it. It would be *really* helpful if Google could provide some useful technical details beyond a generic FAQ page. As it is I just get annoyed by it and have no way to troubleshoot or correct the constant false positives. How is Google detecting "robots"? My sense is that I tend to trigger the captcha thing when iterating similar search terms (particularly due to removal of the + operator and extremely poor "change my search terms because you think you know better than I do what I want to search for" behaviour. My search patterns haven't really changed since turning up IPv6 everywhere, so I have to think either the captcha trigger has gotten more aggressive, or somehow prefers to blacklist IPv6 users. In any case, just going to IPv6 is definitely not a complete fix for this. It seems to be related to search behaviour and $blackbox_magic. Keenan Tims Stargate Connections ________________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Philip Lavine via NANOG <nanog@nanog.org> Sent: February 29, 2016 7:53 AM To: Damian Menscher Cc: nanog@nanog.org Subject: Re: google search threshold I have about 2000 users behind a single NAT. I have been looking at netflow, URL filter logs, IDS logs, etc. The traffic seems to be legit. I am going to move more users to IPv6 and divide some of the subnets into different NATS and see if that alleviates the traffic load. Thanks for the advice. -Philip From: Damian Menscher <damian@google.com> To: Philip Lavine <source_route@yahoo.com> Cc: "nanog@nanog.org" <nanog@nanog.org> Sent: Friday, February 26, 2016 6:05 PM Subject: Re: google search threshold On Fri, Feb 26, 2016 at 3:01 PM, Philip Lavine via NANOG <nanog@nanog.org> wrote: Does anybody know what the threshold for google searches is before you get the captcha?I am trying to decide if I need to break up the overload NAT to a pool. There isn't a threshold -- if you send automated searches from an IP, then it gets blocked (for a while). So... this comes down to how much you trust your machines/users. If you're a company with managed systems, then you can have thousands of users share the same IP without problems. But if you're an ISP, you'll likely run into problems much earlier (since users like their malware). Some tips: - if you do NAT: try to partition users into pools so one abusive user can't get all your external IPs blocked - if you have a proxy: make sure it inserts the X-Forwarded-For header, and is restricted to your own users - if you're an ISP: IPv6 will allow each user to have their own /64, which avoids shared-fate from abusive ones Damian (responsible for DDoS defense)-- Damian Menscher :: Security Reliability Engineer :: Google :: AS15169
DO's SG range is allocated out of a single /64 (I think?) and Google basically asks for captcha on every single request over IPv6. :( We're using it as a corporate vpn. On 3/1/2016 01:49 AM, Keenan Tims wrote:
FWIW I have seen the captchas more often on IPv6 both from home and the office than when both networks were using a single shared IPv4; not sure if this is just related to chronology or a real effect. Once a month or so I seem to get them for a couple of days, then they go away.
No idea what's triggering it. It would be *really* helpful if Google could provide some useful technical details beyond a generic FAQ page. As it is I just get annoyed by it and have no way to troubleshoot or correct the constant false positives. How is Google detecting "robots"? My sense is that I tend to trigger the captcha thing when iterating similar search terms (particularly due to removal of the + operator and extremely poor "change my search terms because you think you know better than I do what I want to search for" behaviour. My search patterns haven't really changed since turning up IPv6 everywhere, so I have to think either the captcha trigger has gotten more aggressive, or somehow prefers to blacklist IPv6 users.
In any case, just going to IPv6 is definitely not a complete fix for this. It seems to be related to search behaviour and $blackbox_magic.
Keenan Tims Stargate Connections ________________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Philip Lavine via NANOG <nanog@nanog.org> Sent: February 29, 2016 7:53 AM To: Damian Menscher Cc: nanog@nanog.org Subject: Re: google search threshold
I have about 2000 users behind a single NAT. I have been looking at netflow, URL filter logs, IDS logs, etc. The traffic seems to be legit.
I am going to move more users to IPv6 and divide some of the subnets into different NATS and see if that alleviates the traffic load. Thanks for the advice. -Philip
From: Damian Menscher <damian@google.com> To: Philip Lavine <source_route@yahoo.com> Cc: "nanog@nanog.org" <nanog@nanog.org> Sent: Friday, February 26, 2016 6:05 PM Subject: Re: google search threshold
On Fri, Feb 26, 2016 at 3:01 PM, Philip Lavine via NANOG <nanog@nanog.org> wrote:
Does anybody know what the threshold for google searches is before you get the captcha?I am trying to decide if I need to break up the overload NAT to a pool.
There isn't a threshold -- if you send automated searches from an IP, then it gets blocked (for a while).
So... this comes down to how much you trust your machines/users. If you're a company with managed systems, then you can have thousands of users share the same IP without problems. But if you're an ISP, you'll likely run into problems much earlier (since users like their malware). Some tips: - if you do NAT: try to partition users into pools so one abusive user can't get all your external IPs blocked - if you have a proxy: make sure it inserts the X-Forwarded-For header, and is restricted to your own users - if you're an ISP: IPv6 will allow each user to have their own /64, which avoids shared-fate from abusive ones Damian (responsible for DDoS defense)-- Damian Menscher :: Security Reliability Engineer :: Google :: AS15169
On 29 February 2016 at 08:53, Paul S. <contact@winterei.se> wrote:
DO's SG range is allocated out of a single /64 (I think?) and Google basically asks for captcha on every single request over IPv6. :(
The solution is to not signup with providers that have no respect for RFCs and BCPs. Proper VPS providers have no issue giving out a /64 to each customer, and many will even give out a /56 or /48 upon request as well. C.
Hello, Something similar to this topic. The other day working with Google APIs (geolocation [1] ) I thought that in order to promote a little bit IPv6, Google (and others) might do something like: Google Maps Geocoding API Usage Limits With IPv4: 2,500 free requests per day (from IPv4 clients) 10 requests per second (from IPv4 clients) With IPv6 5,000 free requests per day (from ipv6 clients) 20 requests per second (from ipv6 clients) Summary: increase rate limit to v6 clients Regards, Alejandro, [1] El 2/29/2016 a las 11:23 AM, Philip Lavine via NANOG escribió:
I have about 2000 users behind a single NAT. I have been looking at netflow, URL filter logs, IDS logs, etc. The traffic seems to be legit.
I am going to move more users to IPv6 and divide some of the subnets into different NATS and see if that alleviates the traffic load. Thanks for the advice. -Philip
From: Damian Menscher <damian@google.com> To: Philip Lavine <source_route@yahoo.com> Cc: "nanog@nanog.org" <nanog@nanog.org> Sent: Friday, February 26, 2016 6:05 PM Subject: Re: google search threshold
On Fri, Feb 26, 2016 at 3:01 PM, Philip Lavine via NANOG <nanog@nanog.org> wrote:
Does anybody know what the threshold for google searches is before you get the captcha?I am trying to decide if I need to break up the overload NAT to a pool.
There isn't a threshold -- if you send automated searches from an IP, then it gets blocked (for a while).
So... this comes down to how much you trust your machines/users. If you're a company with managed systems, then you can have thousands of users share the same IP without problems. But if you're an ISP, you'll likely run into problems much earlier (since users like their malware). Some tips: - if you do NAT: try to partition users into pools so one abusive user can't get all your external IPs blocked - if you have a proxy: make sure it inserts the X-Forwarded-For header, and is restricted to your own users - if you're an ISP: IPv6 will allow each user to have their own /64, which avoids shared-fate from abusive ones Damian (responsible for DDoS defense)-- Damian Menscher :: Security Reliability Engineer :: Google :: AS15169
participants (6)
-
Alejandro Acosta -
Constantine A. Murenin -
Damian Menscher -
Keenan Tims -
Paul S. -
Philip Lavine