Re: Correctly dealing with bots and scrapers.

Am 16.07.2025 um 10:48:39 Uhr schrieb Andrew Latham via NANOG:

1. What tools or setups have people found most successful for dealing with bots/scrapers that do not respect robots.txt for example?

Place a link to a file that is hidden to normal people. Exclude the directory via robots.txt. Then use fail2ban to block all IP addresses that poll the file. -- Gruß Marco Send unsolicited bulk mail to 1752655719muell@cartoonies.org