On 14 Jan 2007, at 13:27, Tony Finch wrote: [Blog spammers]

Most of the IP addresss you listed are are already on various DNS blacklists.

Ooh, now that is interesting. I had assumed that the DNSBLs only covered SMTP spam sources, but on reflection I suppose SMTP is a dead protocol these days in the wider Internet. For the benefit of those of us who have been lucky to Recover from ISP work and now herd blogs[0], would you be so kind as to share which blacklists are worthwhile and worth consulting on this front? [0] Before you ask, no, it's no easier, in fact arguably harder work, although the pay and hours are better. But yes, we're hiring.

Gadi, if your HTTP spam DNSBL gets working, we would certainly be interested in feeding our spam filter from it. It is my experience so far that comments spam is not very "botnetty" but more "boxy" - the proportion of the total we get from any single IP address is relatively high. Actually, to put that better, rather than being evenly distributed over many IPs, a core-group of the IPs spamming us at any one time account for the bulk of it. 80/20 rule again On 1/14/07, Gadi Evron <ge@linuxbox.org> wrote:

On Sun, 14 Jan 2007, Peter Corlett wrote:

...

On 14 Jan 2007, at 13:27, Tony Finch wrote: [Blog spammers]

...

Most of the IP addresss you listed are are already on various DNS blacklists.

Ooh, now that is interesting. I had assumed that the DNSBLs only covered SMTP spam sources, but on reflection I suppose SMTP is a dead protocol these days in the wider Internet.

For the benefit of those of us who have been lucky to Recover from ISP work and now herd blogs[0], would you be so kind as to share which blacklists are worthwhile and worth consulting on this front?

[0] Before you ask, no, it's no easier, in fact arguably harder work, although the pay and hours are better. But yes, we're hiring.

Your assumption is incorrect. These DNSBLs cover spam sent in email, indeed. Thing is, spam is spam and spammers are spammers. Meaning, they spam in every way they can.

In my experience 20-70 per cent would be flagged by email DNSBLs. Not accurate to filter out blog spam.

As in, bots will be bots.

I've been working on a new DNSBL for comment/etc. spam for a while, which will be reliable, generally, it doesn't exist yet for public consumption.

There is such a black listing service already, but again, reliability is an issue.

Gadi.

On Sun, 14 Jan 2007, William Warren wrote:

Heck feed it from spamkarma 2 or askimet. I use spamkarma 2 and it routinely nails tons of blog spammers..:)

SK2 and Akismet indeed do good work on WordPress, but are far from the solution to the problem. Things just get out of hand in the realm of comment spam as more and more spammers invest resources there and overload web pages and services. http://blogs.securiteam.com/index.php/archives/285 http://blogs.securiteam.com/index.php/archives/290 http://blogs.securiteam.com/index.php/archives/296 http://blogs.securiteam.com/index.php/archives/401 http://blogs.securiteam.com/index.php/archives/470 http://blogs.securiteam.com/index.php/archives/471 http://blogs.securiteam.com/index.php/archives/502 http://blogs.securiteam.com/index.php/archives/180

Alexander Harrowell wrote:

...

Gadi, if your HTTP spam DNSBL gets working, we would certainly be interested in feeding our spam filter from it. It is my experience so far that comments spam is not very "botnetty" but more "boxy" - the proportion of the total we get from any single IP address is relatively high.

Actually, to put that better, rather than being evenly distributed over many IPs, a core-group of the IPs spamming us at any one time account for the bulk of it. 80/20 rule again

On 1/14/07, *Gadi Evron* <ge@linuxbox.org <mailto:ge@linuxbox.org>> wrote:

On Sun, 14 Jan 2007, Peter Corlett wrote: > > On 14 Jan 2007, at 13:27, Tony Finch wrote: > [Blog spammers] > > Most of the IP addresss you listed are are already on various DNS > > blacklists. > > Ooh, now that is interesting. I had assumed that the DNSBLs only > covered SMTP spam sources, but on reflection I suppose SMTP is a dead > protocol these days in the wider Internet. > > For the benefit of those of us who have been lucky to Recover from > ISP work and now herd blogs[0], would you be so kind as to share > which blacklists are worthwhile and worth consulting on this front? > > [0] Before you ask, no, it's no easier, in fact arguably harder work, > although the pay and hours are better. But yes, we're hiring. >

Your assumption is incorrect. These DNSBLs cover spam sent in email, indeed. Thing is, spam is spam and spammers are spammers. Meaning, they spam in every way they can.

In my experience 20-70 per cent would be flagged by email DNSBLs. Not accurate to filter out blog spam.

As in, bots will be bots.

I've been working on a new DNSBL for comment/etc. spam for a while, which will be reliable, generally, it doesn't exist yet for public consumption.

There is such a black listing service already, but again, reliability is an issue.

Gadi.

-- My "Foundation" verse: Isa 54:17 No weapon that is formed against thee shall prosper; and every tongue that shall rise against thee in judgment thou shalt condemn. This is the heritage of the servants of the LORD, and their righteousness is of me, saith the LORD.

-- carpe ductum -- "Grab the tape" CDTT (Certified Duct Tape Technician)

Linux user #322099 Machines: 206822 256638 276825 http://counter.li.org/

On Tuesday 16 January 2007 03:06, Jason Frisvold wrote:

The argument there is that those users don't deserve to comment if they can't keep their computers clean, but let's get real.. Some of this stuff is getting pretty advanced and it's getting tougher for general users to keep their computers clean.

I'd have said it was getting easier to keep computers clean. Back in the late 1980's I use to have my own DOS boot disk, with bootsector antivirus tools, so that any PC I used on my University I could be sure was clean. Doesn't mean there aren't more computers, with less clueful users, these days.

I think a far better system is something along the lines of a SURBL with word filtering. I believe that Akismet does something along these lines.

This is the same issue as the email spam issue. Identify by source, or content. Just as content filters are error prone with email spam, they will be error prone with other types of content. I think either approach is viable, as long as the poster has an immediate method of redress. ("My IP is clean" works, and scales, "this URL is safe" works but doesn't scale, "this post" is safe is viable). In each case you need to make sure the redress is protected from abuse, so some sort of CAPTCHA is inevitable.

...

There is such a black listing service already, but again, reliability is an issue.

Reliability is always an issue with blacklists as they are run as independent entities. There is always someone who has a problem with how an individual blacklist is run...

That is easily solved with one's feet. Not as if there is a shortage of blacklists for various purposes.

Frisvold: How does this make his assumption incorrect? Spam is spam and DNSBLs will likely be very effective when it comes to stopping comment spam. There are, of course, some severe problems with using a DNSBL as a blocklist for comments...

But there's a major problem here... A DNSBL is a source blocklist.

Since the current trend in spam (comment and smtp) is to use botnets, then by blocking the bots, you also block the users who would make meaningful comments.

Especially as bots are usually found in customer dynamic-IP pools. Assigning a value for relative spamminess by country would work up to a point (Italy, Ukraine, we mean you) but the false positive rate is unacceptable. Anyway, very anti-Internet and hardly appropriate for a blog whose declared mission is pan-European opinion..

The argument there is that those users don't deserve to comment if they can't keep their computers clean, but let's get real.. Some of this stuff is getting pretty advanced and it's getting tougher for general users to keep their computers clean.

I think a far better system is something along the lines of a SURBL with word filtering. I believe that Akismet does something along these lines.

We had a word filter plus lookups of bsb.spamlookup.net. Our experience in the last few months was not good - the rate of false positives was high (essentially all genuines had to be individually approved, and worse, rather than into a queue they usually went into the spamtrap) and the rate of false negatives was nontrivial. We have recently implemented Akismet. It's a major improvement - the false positives have been nearly eliminated and the false negatives down to a couple a week. Multi-layered defence is a "must" - for example, most comments spam is very self-similar, so you could run a Bayesian filter comparing the stuff rejected by the blocklist with the content of the trap in order to sort between "spam" and "hold for approval". Mind you, some of the Bayesian-beating techniques used for SMTP spam are now showing up in comments - for example, delivering the beneficiary link and a paragraph of news scraped from news.bbc.co.uk, which is a lot like a real (but dull:-)) comment. Perhaps a better filter might be on the links they contain (some domains come up again, and again, and again). Then again, once you're doing anything like that, it's already hit your server and is costing cycles if nothing else. In the future, someone will lose the vote through being mistaken for a spambot. Alex

On Sun, 14 Jan 2007, Peter Corlett wrote:

For the benefit of those of us who have been lucky to Recover from ISP work and now herd blogs, would you be so kind as to share which blacklists are worthwhile and worth consulting on this front?

I would expect the lists of compromised hosts to be fairly effective - open proxies of various kinds and perhaps botnet hosts. As for SMTP the blacklists would only be a starting point that either provide a cheap preliminary check or feed a more sophisticated filtering system. Tony. -- f.a.n.finch <dot@dotat.at> http://dotat.at/ SOUTH UTSIRE: NORTHWEST BACKING SOUTHWEST 6 TO GALE 8, OCCASIONALLY SEVERE GALE 9. VERY ROUGH OR HIGH. RAIN OR SQUALLY SHOWERS. MODERATE OR GOOD.

On Sun, 14 Jan 2007, Tony Finch wrote:

I would expect the lists of compromised hosts to be fairly effective - open proxies of various kinds and perhaps botnet hosts. As for SMTP the blacklists would only be a starting point that either provide a cheap preliminary check or feed a more sophisticated filtering system.

If you allow anonymous, unauthenticated access to any system it will be abused. Auctions, blogs, chat, mail, phone, etc. IP addresses have never been good authenticators for applications. Sending confirmation E-mail addresses aren't that much better. And blacklists will just continue to grow longer. How do you know your user?

...

If you allow anonymous, unauthenticated access to any system it will be abused. Auctions, blogs, chat, mail, phone, etc. IP addresses have never been good authenticators for applications.

This is not true if you control the IP address space and the routers around it. I mention this merely because "IP addresses have never been good authenticators" or the like is becoming a truism. For ISPs with good source filtering in place then IP addresses ARE good first level authenticators (e.g. filter lists on management ports). Note: I say FIRST level authenticators; IP addresses are obviously not suitable as the whole authentication process.

I don't know why, but I feel the need to clarify some semantics. I am sure everyone involved in this discussion already knows what I am about to say. I think the word "system" here is being abused and the context is changing. IPs are reasonable in the authentication process for network-centric items (like routers, things that make up the lowest levels of the OSI stack). Systems here means routers, or the networks they make. IPs are less reasonable the higher up the OSI stack you go. A web server may authenticate with IPs and find use in them. An application running on that web server is almost always going to find less value in that authentication since it is capable of more specific authentication (password, cookie, post rate limit, etc). This use approaches, but may not reach, the "zero" asymptote when you consider cases of applications running on private networks (VPNs, NAT networks, localhost, etc). System here means anything else, but almost never a router or the underlying network infrastructure. Yes, Geotrack has given us some more detail (of varying levels of precision/accuracy) of where IPs come from. But pretty much IP level controls (IMO) should stay at the lowest levels of the OSI stack. Ian looks to me like he was talking about routers & their neighbors. Which is a very NANOG charter way to look at things. Sean looks like he was talking about everything else (applications and things in user space). All things things NANOGers support that pays for the pretty blinky lights. I'm done. Hope that was mildly interesting or useful. Deepak