:-) Let me add something before everyone on NANOG reminds me that gigablast is a search engine..... I know what they do, but what I don't understand is why are they searching my systems for URLs that haven't ever existed there before. It's as though they are doing random word searches in hopes of striking lucky. They are "crawling" for URLs like this: (unfortunately most people won't see these because their spam blockers will block all the exclamation points)
[list of random path names snipped] This seems to be a very wrong and bad thing to do. Google searches URLs because a human gives it permission to do so, for example by linking to that URL. (What purpose does a link have other than to be something to click on.) What gigablast seems to be doing, on the other hand, is trying to open every window in a house in the hopes that it will find one that's open. It has no invitation or permission to do this, and I would consider such behavior inappropriate. You do not have the right to make requests of other people's computers without their permission. You can certainly argue implied permission in many cases -- for example, if Ford registers the domain ford.com, and assigns an IP address to 'www.ford.com', you can certainly argue that they have invited the public to access that URL because that's the normal reason people create such things. However, you have no implied permission to try numerous combinations of random paths on the end of that in the hopes that you'll find something Ford did not invite you into. DS