Correctly dealing with bots and scrapers.

I just had an issue with a web-server where I had to block a /18 of a large scraper. I have some topics I could use some input on. 1. What tools or setups have people found most successful for dealing with bots/scrapers that do not respect robots.txt for example? 2. What tools for response rate limiting deal with bots/scrapers that cycle over a large variety of IPs with the exact same user agent? 3. Has anyone written or found a tool to concentrate IP addresses into networks for IPTABLES or NFT? (60% of IPs for network X in list so add network X and remove individual IP entries.) -- - Andrew "lathama" Latham -