On Wed, Sep 08, 2010 at 12:04:07AM -0700, Matthew Petach said:
I *am* curious--what makes it any worse for a search engine like Google to fetch the file than any other random user on the Internet? In either case, the machine doing the fetch isn't going to rate-limit the fetch, so you're likely to see the same impact on the machine, and on the bandwidth.
I think that the difference is that there's a way to get to Yahoo and ask them WTF. Whereas the guy who mass downloads your site with a script in 2 hrs you have no recourse to (modulo well funded banks dispatching squads with baseball bats to resolve hacking incidents). I also expect that Yahoo's behaviour is driven by policy, not random assholishness (I hope :), and therefore I should expect such incidents often. I also expect whinging on nanog might get me some visiblity into said policy and leverage to change it! </dream> /kc -- Ken Chase - ken@heavycomputing.ca - +1 416 897 6284 - Toronto CANADA Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.