2 Jul
2025
2 Jul
'25
10:50 a.m.
* Constantine A. Murenin [Wed 02 Jul 2025, 05:23 CEST]:
But the bots are not a problem if you're doing proper caching and throttling.
Have you been following the news at all lately? Website operators are complaining left and right about the load from scrapers related to AI companies. They're seeing 10x, 100x the normal visitor load, with not just User-Agents but also source IP addresses masked to present as regular visitors. Captchas is unfortunately one of the more visible ways to address this, even if not perfect. For example, https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-for... -- Niels.