Thats exactly it... they are doing site indexing .. if you like google... you'll need to like them! =P I personally wouldnt worry about anything in the logs unless you start seeing attempts to search and exploit .cgi and executable files... -Payam
Google is your friend? They're a search engine. robots.txt and forget it.
Malcolm
Jim Popovitch wrote:
Feel free to clue me in on this please... ;-)
What is www.gigablast.com? And why is it constantly performing "questionable" queries (mostly http) across every IP that I have access to check.
I get a could of thousand hits (mostly questionable non-existing URL requests) from that ip (66.154.103.75). Anyone else seeing/questioning this?
Completewhois shows some listings in some RBLs, but not the more popular ones.
-Jim P.
-- -- Payam Tarverdyan Chychi Network Analyst