On Tue, Sep 07, 2010 at 04:19:58PM -0400, Ken Chase wrote:
This makes it look like Yahoo is actually trafficking in pirated software, but that's kinda too funny to expect to be true, unless some yahoo tech decided to use that IP/server @yahoo for his nefarious activity, but there are better sites than my customer's box to get his 'juarez'.
It's not uncommon at all for a web-spider to find large files and download them. I don't think there's some conspiracy at Yahoo to find warez; they are just opperating as a normal spider, indexing the Internet.
~500K/s (4Mbps+) for a 3 gig file is kinda... a bit harsh.
What speed would you like a spider to download at? You could configure the speeds to Yahoo's blocks server-side if you care enough. Ideally, request your customer doesn't throw large programs on there if you're concerned about bandwidth. 4 Mb/s isn't abnormal at all for a spider, and especially on a larger file.
Is this expected/my own fault or what?
A little bit of both :)