Yes Marco that is one of the tactics I see to deal with this. Fail2ban is a great piece of software that I use. This reminded me of another related topic. Security Scans. Any requests for wordpress could be an easy way to flag and block with fail2ban when wordpress is not in use. On Wed, Jul 16, 2025 at 12:36 PM Marco Moock via NANOG <nanog@lists.nanog.org> wrote:
Am 16.07.2025 um 10:48:39 Uhr schrieb Andrew Latham via NANOG:
1. What tools or setups have people found most successful for dealing with bots/scrapers that do not respect robots.txt for example?
Place a link to a file that is hidden to normal people. Exclude the directory via robots.txt.
Then use fail2ban to block all IP addresses that poll the file.
-- Gruß Marco
Send unsolicited bulk mail to 1752655719muell@cartoonies.org _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/WX65VXNA...
-- - Andrew "lathama" Latham -