There is what we could call a "growth market" of grey market organizations and individuals selling residential proxies that route traffic through actual residential cablemodem, DSL, FTTH connections at peoples' houses. Usually this is implemented one of two ways, a router/home gateway device that's been pwned and is part of a botnet, or by tricking people into installing some form of proxy software on one of their persistently connected home computers. Used for both bot scraping and also more mundane credit card fraud and various scams, providing people access to georestricted online casinos, weird hawala-adjacent online money/cryptocurrency exchange and transfer platforms. Just a vast array of shady stuff. Google "residential proxies for sale" to see the tip of the iceberg. On Wed, Jul 16, 2025 at 4:38 PM Andrew Latham via NANOG < nanog@lists.nanog.org> wrote:
Chris
Spot on, and I am getting the feeling this is where the value to a geo-ip service comes to play that offers defined "eyeball networks" to allow.
On Wed, Jul 16, 2025 at 12:57 PM Chris Adams via NANOG <nanog@lists.nanog.org> wrote:
Once upon a time, Marco Moock <mm@dorfdsl.de> said:
Place a link to a file that is hidden to normal people. Exclude the directory via robots.txt.
Then use fail2ban to block all IP addresses that poll the file.
The problem with a lot of the "AI" scrapers is that they're apparently using botnets and will often only make a single request from a given IP address, so reactive blocking doesn't work (and can cause other issues, like trying to block 100,000 IPs, which fail2ban for example doesn't really handle well). -- Chris Adams <cma@cmadams.net> _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/AFJF4UQJ...
-- - Andrew "lathama" Latham - _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/DHUYTBIX...