I’ve been collecting lots of bot traffic and other naughty things and have some automated stuff that reads and updates my nginx files. Feel free to check out my GitHub repository for IoC, signatures, and mitigations. https://github.com/trumb/nginx-hardening I’m also experimenting with a Claude Code plugin that does something similar but it’s a work in progress. https://github.com/trumb/claude-nginx-hardening On Wed, Mar 25, 2026 at 2:44 PM Andrew Latham via NANOG < nanog@lists.nanog.org> wrote:
Update
Many software projects have solutions for this, I was not searching for the right thing. The term to search for is "expensive" like in
https://github.com/go-gitea/gitea/blob/main/custom/conf/app.example.ini#L786... from https://github.com/go-gitea/gitea/issues/33966
I am also finding solutions for other software I use along the lines of requiring sign-in for paths that have computational costs. It was a show thought to require sign-in to read my gitea instance and when I found the setting I was very happy.
My sites use TLS(80 to 443 redirect) so I also banned user-agents of Windows 95(4k hits an hour), Windows 98(9k hits per hour), Windows CE(400 hits per hour), Windows NT 4(4k hits per hour), Windows NT 5(5k hits per hour) to name a few.
Next up: Make sense of the Apple OS versions and wich would most likely be able to reach TLS endpoints
P.S. I know people mean well, but no off list emails please.
On Sat, Mar 21, 2026 at 9:59 AM Andrew Latham <lathama@gmail.com> wrote:
Andrew
The issue is that it is hitting a gitea instance and a mediawiki
getting lost in the commit/change diff system. I should just just figue out how to dissable the difference tools on gitea and mediawiki to keep the bots from going down a rabbit hole on a decade or more of commits and page edits.
These are CPU intensive which is my issue at the moment.
On Sat, Mar 21, 2026 at 9:53 AM Andrew Kirch via NANOG <nanog@lists.nanog.org> wrote:
Get a small version of a very old very fast very inaccurate LLM. Have
it
generate a couple terabytes of endless nonsense.
Redirect scrapers to it, and poison whatever LLM they are trying to
instance, then train.
Andrew
On Wed, Jul 16, 2025 at 12:49 PM Andrew Latham via NANOG < nanog@lists.nanog.org> wrote:
I just had an issue with a web-server where I had to block a /18 of a large scraper. I have some topics I could use some input on.
1. What tools or setups have people found most successful for dealing with bots/scrapers that do not respect robots.txt for example?
2. What tools for response rate limiting deal with bots/scrapers that cycle over a large variety of IPs with the exact same user agent?
3. Has anyone written or found a tool to concentrate IP addresses into networks for IPTABLES or NFT? (60% of IPs for network X in list so add network X and remove individual IP entries.)
-- - Andrew "lathama" Latham - _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/Z2J6CFBK...
_______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/7N5RWJOU...
-- - Andrew "lathama" Latham -
-- - Andrew "lathama" Latham - _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/W6YNLWVR...