Inktomi (now Yahoo!) sends it's spiders all over the Internet. Lately some of our systems are reporting that they open many HTTP connections to our web sites, without ever sending any data and immediately disconnecting. This is getting to a level where it disturbs us. Is something broke over there? I can't seem to be able to reach them and this is becoming a real annoyance. Anyone else observing this? -- Gadi Evron, Information Security Manager, Project Tehila - Israeli Government Internet Security. Ministry of Finance, Israel. gadi@tehila.gov.il gadi@CERT.gov.il Office: +972-2-5317890 Fax: +972-2-5317801 http://www.tehila.gov.il
On Thu, 20 Jan 2005 14:30:04 +0200, Gadi Evron <gadi@tehila.gov.il> wrote:
Inktomi (now Yahoo!) sends it's spiders all over the Internet. Lately some of our systems are reporting that they open many HTTP connections to our web sites, without ever sending any data and immediately disconnecting. This is getting to a level where it disturbs us.
I have heard previous stories of inktomi ignoring robots.txt (not seen this for myself though). And there are threads like this - Quoting from http://www.webmasterworld.com/forum11/1968-1-15.htm
I've got Scooter allowed in, but I've also got it lumped int with a number of agents that are not allowed to get non-HTML files. This is especially important at my site as it includes a number of very large binary datasets in numerous locations and the robots have proven too stupid to understand that downloading them is a waste of bandwidth.
RewriteCond %{HTTP_USER_AGENT} .*Ask.Jeeves.* [OR] RewriteCond %{HTTP_USER_AGENT} .*FAST.WebCrawl.* [OR] RewriteCond %{HTTP_USER_AGENT} .*ia_archiver.* [OR] RewriteCond %{HTTP_USER_AGENT} .*InfoSeek.* [OR] RewriteCond %{HTTP_USER_AGENT} .*inktomi.* [OR] RewriteCond %{HTTP_USER_AGENT} .*Scooter.* [OR] RewriteCond %{HTTP_USER_AGENT} .*Slurp.* [OR] RewriteCond %{HTTP_USER_AGENT} .*Teoma.* [OR] RewriteCond %{HTTP_USER_AGENT} .*VoilaBot.* [OR] RewriteCond %{HTTP_USER_AGENT} .*Google.* RewriteRule!.*(html¦htm¦txt¦/)$ /www/msgs/badagent.html [F]
On Thu, 20 Jan 2005, Suresh Ramasubramanian wrote:
Inktomi (now Yahoo!) sends it's spiders all over the Internet. Lately some of our systems are reporting that they open many HTTP connections to our web sites, without ever sending any data and immediately disconnecting. This is getting to a level where it disturbs us. I have heard previous stories of inktomi ignoring robots.txt (not seen
On Thu, 20 Jan 2005 14:30:04 +0200, Gadi Evron <gadi@tehila.gov.il> wrote: this for myself though). And there are threads like this - Quoting from http://www.webmasterworld.com/forum11/1968-1-15.htm
back in 1999 inktomi hammered our nameserver (which never has, and never will run http. ever.) After _weeks_ of complaining to them and to their upstream exodus (hah!) I finally got them to stop. Only to have them start up again a month later. not suprising to see them up to their old antics again. time to nullroute i guess? -Dan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 not sure if spiders falls under spam or ddos bracket when they repeatedly start hammering one's network. you could possible report to spamcop (*grin*) to get a quicker response. spamcom hasn't been accurate in some instances :-) do you remember this incident, http://www.cs.wisc.edu/~plonka/netgear-sntp/ regards, /vicky Dan Hollis wrote: | On Thu, 20 Jan 2005, Suresh Ramasubramanian wrote: | |>On Thu, 20 Jan 2005 14:30:04 +0200, Gadi Evron <gadi@tehila.gov.il> wrote: |> |>>Inktomi (now Yahoo!) sends it's spiders all over the Internet. Lately |>>some of our systems are reporting that they open many HTTP connections |>>to our web sites, without ever sending any data and immediately |>>disconnecting. This is getting to a level where it disturbs us. |> |>I have heard previous stories of inktomi ignoring robots.txt (not seen |>this for myself though). And there are threads like this - |>Quoting from http://www.webmasterworld.com/forum11/1968-1-15.htm | | | back in 1999 inktomi hammered our nameserver (which never has, and never | will run http. ever.) After _weeks_ of complaining to them and to their | upstream exodus (hah!) I finally got them to stop. Only to have them | start up again a month later. | | not suprising to see them up to their old antics again. | | time to nullroute i guess? | | -Dan | | | -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFB8DFOpbZvCIJx1bcRAu2FAJ4+a2SHF7XxWgaHKFZzi7hf46tJFwCfcU12 fbIMwtwkPhI33onPawlBKYE= =P+y0 -----END PGP SIGNATURE-----
Vicky Rode <vickyr@socal.rr.com> wrote:
not sure if spiders falls under spam or ddos bracket when they repeatedly start hammering one's network. you could possible report to spamcop (*grin*) to get a quicker response. spamcom hasn't been accurate in some instances :-)
Er.. just what would you report to spamcop, and what would spamcop do with your reports?
do you remember this incident, http://www.cs.wisc.edu/~plonka/netgear-sntp/
Not very new .. broken apps which keep hammering on a resource for some reason are a fairly regular "feature" of the internet. srs
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 in-line: Suresh Ramasubramanian wrote: | Vicky Rode <vickyr@socal.rr.com> wrote: | | |>not sure if spiders falls under spam or ddos bracket when they |>repeatedly start hammering one's network. you could possible report to |>spamcop (*grin*) to get a quicker response. spamcom hasn't been accurate |>in some instances :-) | | | Er.. just what would you report to spamcop, and what would spamcop do with your | reports? - ------------------ that's why i asked, this type of behavior falls under what abuse terms? | | |>do you remember this incident, http://www.cs.wisc.edu/~plonka/netgear-sntp/ | | | Not very new .. broken apps which keep hammering on a resource for some reason | are a fairly regular "feature" of the internet. - ----------------- doesn't mean that it shouldn't be blocked/reported. regards, /vicky | | srs | -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFB8a1ipbZvCIJx1bcRAmOrAKCnCHmj72VsJIec/CsA0JBjkbGdCACgi9BB N20N5nuLVPFN5+bYVF3k7pY= =BwbD -----END PGP SIGNATURE-----
On Fri, 21 Jan 2005 17:33:22 -0800, Vicky Rode <vickyr@socal.rr.com> wrote:
that's why i asked, this type of behavior falls under what abuse terms?
doesn't mean that it shouldn't be blocked/reported.
Block - you have enable on your routers and can do everything from access list 101 deny to something fancier like NBAR to block just these queries. -- Suresh Ramasubramanian (ops.lists@gmail.com)
participants (5)
-
Dan Hollis -
Gadi Evron -
ops.lists＠gmail.com -
Suresh Ramasubramanian
-
Vicky Rode