16 Jul
2025
16 Jul
'25
10:27 p.m.
On Wed, Jul 16, 2025 at 11:57 AM Chris Adams via NANOG <nanog@lists.nanog.org> wrote:
The problem with a lot of the "AI" scrapers is that they're apparently using botnets and will often only make a single request from a given IP address, so reactive blocking doesn't work
So do reactive un-blocking instead. The first request from any IP address gets an empty web page with, "<meta http-equiv=refresh content=1>," and triggers a permit rule which allows subsequent requests to reach the actual web server and content. That way the one-shot sources get an empty page no matter what URL they request. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/