Has anyone had any experience with this device? Turntide.com. Looks like a traffic-shaping device designed specifically for cutting down spammers throughput to your inbound SMTP servers. My main concern is, how does it make the distinction between legitimate mass-mailings (e.g.: mailing lists such as this one), and spam? Interesting approach to killing spam though I must say. -Andy
Andy Johnson [05/04/04 10:49 -0400]:
Has anyone had any experience with this device? Turntide.com. Looks like a traffic-shaping device designed specifically for cutting down spammers throughput to your inbound SMTP servers. My main concern is, how does it make the distinction between legitimate mass-mailings (e.g.: mailing lists such as this one), and spam? Interesting approach to killing spam though I must say.
Sounds like is basically the same principle as that of an IDS. srs
On Apr 5, 2004, at 10:49 AM, Andy Johnson wrote:
Has anyone had any experience with this device? Turntide.com. Looks like a traffic-shaping device designed specifically for cutting down spammers throughput to your inbound SMTP servers. My main concern is, how does it make the distinction between legitimate mass-mailings (e.g.: mailing lists such as this one), and spam? Interesting approach to killing spam though I must say.
Sounds like YABA (Yet Another Band Aid) solution for spam. If rate-limiting the spam packets does an effective job at killing spam. It will only make the spammers switch to a distrubuted attack method using trojaned virus hosts sending 1 mail message at a time. They are already doing this in some cases. SPAM is a living breathing entity that can learn and adapt. The smarter the network gets at killing it off, the smarter it gets in attacking. The evolution of spam/viruses is astounding and getting quicker all the time. The turntide box may be a good solution but it is expensive, I'll wait for the SNORT add-on that does the same thing ;) -Matt
From: "Matthew Crocker" <matthew@crocker.com>
Sounds like YABA (Yet Another Band Aid) solution for spam. If rate-limiting the spam packets does an effective job at killing spam. It will only make the spammers switch to a distrubuted attack method using trojaned virus hosts sending 1 mail message at a time. They are already doing this in some cases. SPAM is a living breathing entity that can learn and adapt. The smarter the network gets at killing it off, the smarter it gets in attacking. The evolution of spam/viruses is astounding and getting quicker all the time. The turntide box may be a good solution but it is expensive, I'll wait for the SNORT add-on that does the same thing ;)
-Matt
That's exactly what I was thinking when reading their documentation. A large portion of the spam we receive on our servers comes from hijacked broadband customers. I don't see how effective it would be rate limiting these guys, because the source IP is almost always unique. Still, if it could cut spam traffic by even half of what they advertise (92% on a system similar to ours), that would help tremendously. -Andy
... will only make the spammers switch to a distrubuted attack method using trojaned virus hosts sending 1 mail message at a time.
"switch to"? i don't know where you're getting your spam from, but the spammers "switched to" that methodology a long long time ago.
The evolution of spam/viruses is astounding and getting quicker all the time.
which is why many of us have fought so hard against bad solutions. the baynesian filtering crowd is the worst. it's like treating every illness with antibiotics... what you end up with is a lot MORE illness in the medium to longer term, due to antibiotic-immune mutations.
... [OpenBSD] Spamd is really nice for hurting spammers and/or relays where it can... in their spool.
spammers don't have spools. they don't care if you get their stuff or not, and they don't retry. that's why greylisting has been so effective -- to combat it the spammers would have to add the one thing they cannot afford: "state." see http://www.rhyolite.com/dcc/ for how to get started. -- Paul Vixie
On 5 Apr 2004, Paul Vixie wrote:
that's why greylisting has been so effective -- to combat it the spammers would have to add the one thing they cannot afford: "state." see http://www.rhyolite.com/dcc/ for how to get started.
why is 'state' so hard to afford? they already have a list of email addresses to spam, and they already have compromised boxes -- those are the big costs for spammers. another byte of state per email address is cheap (or if you are clever, a single bit stored in the email address itself, which doesnt cost you anything). i see greylisting being effective only as long as it doesnt get widely deployed. as soon as greylisting starts having any impact on spammers, they'll start spooling -- and it is very cheap to do so. after all, just about everything on compromised boxes costs them nothing. and compromised are the source of 99.9999999% of all spam. -Dan
this is actually not so much about spam as it is about security models.
that's why greylisting has been so effective -- to combat it the spammers would have to add the one thing they cannot afford: "state." see http://www.rhyolite.com/dcc/ for how to get started.
why is 'state' so hard to afford? they already have a list of email addresses to spam, and they already have compromised boxes -- those are the big costs for spammers. another byte of state per email address is cheap (or if you are clever, a single bit stored in the email address itself, which doesnt cost you anything).
that presumes a definite system wherein a spammer knows who he has sent what to. as if they felt it was nec'y to send only one copy of a spam to each person, or indeed, as if they had any records of what addresses bounce, what addresses (or servers) lead to quicksand, or whatever. it is difficult for the average professional engineer to comprehend, down in their bones, how little attention a spammer can afford to pay to any one server or address.
i see greylisting being effective only as long as it doesnt get widely deployed. as soon as greylisting starts having any impact on spammers, they'll start spooling -- and it is very cheap to do so. after all, just about everything on compromised boxes costs them nothing. and compromised are the source of 99.9999999% of all spam.
i half agree. any technique that pinches a spammer's success rate (which means, the rate at which they hit blind trap addresses monitored by their customers) will be cause for attention. this is information warfare, and there's an effort budget on both sides, asymmetric though it damnably is. however, "they'll start spooling" is simplistic. the compromised middle- boxes don't have state -- nothing gets written to disk. these are not mail relays, but rather, deliberately open proxies. if state were kept it would be (a) evidence to be used against the spammer, and (b) cause for the box-owner to notice the activity and perhaps scrape their malware. the endgame for greylisting is the same as for every moderately successful "antispam" technique. there will be a three-way schism. (1) some spammers won't notice or won't care that their success rate drops, and they'll eventually upgrade their spamware when somebody else improves it. (2) some spammers will explore ways to keep state and do the retries necessary to get past the greylist filters. (3) some spammers will just send everything to everybody every 30 minutes, no matter whether the last response code was 4xx or 5xx. all three will make themselves easier to triangulate upon, and the conviction rate will edge upward slightly. (the things spammers do to avoid brightmail and DCC smell really strong -- there's no mistaking that kind of zwil for honest e-mail, even robotically.) -- Paul Vixie
which is why many of us have fought so hard against bad solutions. the baynesian filtering crowd is the worst. it's like treating every illness with antibiotics... what you end up with is a lot MORE illness in the medium to longer term, due to antibiotic-immune mutations.
Any content analysis that can be done to classify content can be defeated by building a content generator that builds content to meet the criteria of the analysis tool. To succeed against the spammers we need to IGNORE the content and target the behaviors. Why does your mail server accept incoming email from unknown and unauthenticated sources? Why does your mail server allow your customers to relay more than a few messages a day without special permission? If we change the email system architecture so that every node in the architecture only accepts mail from known sources and enforces a pre-arranged behavior pattern, then we can clean up this mess. It makes sense to accept large quantities of inbound email from aol.com, i.e. the large flows are acceptable behavior in that instance. But it does not make sense to accept large quantities of inbound email from a broadband customer. It also does not make sense for a site with a large inflow of mail to accept mail from all and sundry. It does make sense for them to prearrange mail exchange with their 100 or 200 or 1000 largest mail exchange partners and force everyone else to deal with one of those partners. This is called adding hierarchy to build a scalable solution. It's called spreading the load. It's called sharing the pain. And it is not something that can be avoided as we now see. Everyone is sharing the pain alright but it is the spammers who are choosing how much and when it hurts. I guess it's a case of pay me now, or pay me later... --Michael Dillon
On 6 Apr 2004, at 05:23, Michael.Dillon@radianz.com wrote:
To succeed against the spammers we need to IGNORE the content and target the behaviors. Why does your mail server accept incoming email from unknown and unauthenticated sources? Why does your mail server allow your customers to relay more than a few messages a day without special permission?
If the behaviours were easy to identify, there would be no spam. My mail server accepts incoming e-mail from unknown and unauthenticated sources (a) because there is no widely-deployed mechanism to recognise or authenticate sources such that good ones can be distinguished from bad ones and (b) because the same sources are frequently responsible for sending spam and non-spam. How do you distinguish between a home user sending twenty legitimate, real messages per day, and a home user whose PC has been 0wned, and which is sending twenty illegitimate messages per day? The behaviours will adapt to defeat any attempt at classification. The content is the only thing which reliably identifies messages as spam, and the only way to classify the content with high confidence is to have the recipient read it and decide whether she is glad she received it. I have now exceeded my self-imposed mailing list threshold of 0 messages about spam per month. Joe
On Tue, 06 Apr 2004 11:02:33 EDT, Joe Abley said:
How do you distinguish between a home user sending twenty legitimate, real messages per day, and a home user whose PC has been 0wned, and which is sending twenty illegitimate messages per day?
Back of the envelope handwaving calculation (we're not worrying about exact numbers, merely having somewhere near the right number of zeros): Media reported that Hotmail was rejecting 2 billion pieces of mail a day (and that's not including AOL, Yahoo, and every single smaller ISP - our site alone is seeing several million a day). Let's say it adds up to 10 billion across the board. Let's assume that 75% of spam is sent via hijacked zombie machines. This would mean that to get 7.5 billion spams/day at 20 msgs/day/zombie, you'd need several hundred million compromised machines. And even though the average machine is woefully insecure, there's not THAT many zombies. On the other hand, 20K msgs/day/zombie is only about 1 ever 4 seconds, not enough to make the average cablemodem user notice - and reduces the number of zombies down to several million - a much more plausible number. If you rate-limit 2 million compromised machines to 20 msgs/day each, there's only 400 million spams. Total.
If you rate-limit 2 million compromised machines to 20 msgs/day each, there's only 400 million spams. Total.
IF you can rate-limit them across the whole Internet, If you limit 2 million machines to 20 msgs/day per mail server you are back up to your 10 Billion msgs/day mark. This is where DCC or other distributed checksum systems come into play. -Matt
On Tue, 06 Apr 2004 13:14:31 EDT, Matthew Crocker said:
IF you can rate-limit them across the whole Internet, If you limit 2 million machines to 20 msgs/day per mail server you are back up to your 10 Billion msgs/day mark. This is where DCC or other distributed checksum systems come into play.
My point was that there's no real *need* to distinguish between a legitimate user sending 20 emails and an 0wned box sending 20 emails, as the distinction is "legitimate 20 emails" versus "0wned 20K emails". If I were to only give my users 20 outbound connections/day, there wouldn't be a per-mail-server issue. Whether I can make such a policy stick is another question entirely.
On Tue, 6 Apr 2004 Valdis.Kletnieks@vt.edu wrote:
On Tue, 06 Apr 2004 13:14:31 EDT, Matthew Crocker said:
IF you can rate-limit them across the whole Internet, If you limit 2 million machines to 20 msgs/day per mail server you are back up to your 10 Billion msgs/day mark. This is where DCC or other distributed checksum systems come into play.
My point was that there's no real *need* to distinguish between a legitimate user sending 20 emails and an 0wned box sending 20 emails, as the distinction is "legitimate 20 emails" versus "0wned 20K emails".
If I were to only give my users 20 outbound connections/day, there wouldn't be a per-mail-server issue. Whether I can make such a policy stick is another question entirely.
I sent more than 20 mails in the last hour. Given that I have a local mta each of those results in a seperate connection attempts to the machine I use as smart-host. I'm sure I could batch them all up and send them at once thereby returning to my uucp days, but bleh, that really breaks up the pattern of back-and-forth communication that we've gotten used to. There's a bunch of forces pushing in various directions that make email less usable for me and I assume everyone else... The big one is spam, restricitive mta behavior is another, and there are others. When my mta becomes more selective about what senders I choose to accept mail from or in this case when or how often, then eventually I lose mail from people I would otherwise have communicated with. That's frustrating becuase it's as disruptive, if not more so than having a mail box full of crap. Eventually I suspect I'll be forced to abandon the rfc2821 email system as a communications tool entirely, and brick myself up in the cellar, but I actuallly liked it as a tool when it worked. joelja
-- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja@darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2
On Tue, 06 Apr 2004 11:02:41 PDT, Joel Jaeggli said:
I sent more than 20 mails in the last hour. Given that I have a local mta each of those results in a seperate connection attempts to the machine I
OK. Make it 100, or make it "20 by default, user can ask for 100". Or anything else like that. The *POINT* was that too often, a compromised end-user machine can send *THOUSANDS* of messages. Not tens. Not hundreds. Thousands. Remember - if you're catching 1M spams/day, that means that the spammer has to have a machines*rate product over 1M spams/day. If there's 10 billion spams/day total, the total machines*rate product has to be over 10G. And if there's only several million source machines, that means the rate has to be in the thousands.
OK. Make it 100, or make it "20 by default, user can ask for 100". Or anything else like that. The *POINT* was that too often, a compromised end-user machine can send *THOUSANDS* of messages. Not tens. Not hundreds. Thousands.
Here's another way to structure this sort of policy using a "soft" limit which would also make it feasible to have a limit lower than 20. If any of your user connections is the origin of more than 5 SMTP sessions in a single day, send an email to the registered contact at that site with a little statistical summary of the activity. No blocking of sessions, just a note saying that we noticed you sent x number of emails today. Give the user some action such as a URL that they can do if they believe that this is abnormal. Then you could make the hard limit for blocking sessions into a larger number such as 50 which is extremely unlikely to block anyone's real email. Of course, anyone running a mailing list would still have to register that fact with you so that you can remove the hard limit on them. --Michael Dillon
On Wed, 2004-04-07 at 13:18, Michael.Dillon@radianz.com wrote:
If any of your user connections is the origin of more than 5 SMTP sessions in a single day, send an email to the registered contact at that site with a little statistical summary of the activity. No blocking of sessions, just a note saying that we noticed you sent x number of emails today. Give the user some action such as a URL that they can do if they believe that this is abnormal.
Why not use a more detailed time-interval based approach only blocking further SMTP connections for say an hour if a user made more than x connects in an y amount of time and automatically resetting the counters and block afterwards..? On top of the x/hour you could make the mechanism less of a burden by putting in an option that would allow connections to be "saved" for a maximum of two or three hours, so when someone comes into his office in the morning he can safely pour out his start-of-the-day e-mail flow without being bothered by the rigid 10 e-mails/hour since there wouldn't have been any connections in the few hours before coming into the office and he might be able to send 20 or 30 e-mails in the first hour before the counters are reset. Spammers can only work when making enormous amounts of connections each hour, so limiting a normal user to 10 connections per hour with some extra slack after two or three connectionless hours, with an hour blocking penalty if the user goes over shouldn't pose a problem to Joe Average and will definitely keep spammers at bay without the added administrative overhead of sending user's mail statistics. Ofcourse as you mentioned, mailinglists and certain users making extreme use of e-mail should always have the possibility of registering for more connections, but when done correctly this could be a more or less hassle free way of controlling mail connection rates without burdening 99% of all users. Regards, -- --- Erik Haagsman Network Architect We Dare BV tel: +31(0)10 7507008 fax:+31(0)10 7507005 http://www.we-dare.nl
Spammers can only work when making enormous amounts of connections each hour, so limiting a normal user to 10 connections per hour with some extra slack after two or three connectionless hours, with an hour blocking penalty if the user goes over shouldn't pose a problem to Joe Average and will definitely keep spammers at bay without the added administrative overhead of sending user's mail statistics. I think 10 is a bit low. I am not really an abnormal email user - but I tend to block answer a lot of emails, and send them as fast as I type them - so I can easily send 20-30 emails in the first hour, then maybe an hour slack, then another dozen or so - depending on inbound traffic and what arguments are ongoing on my mailing lists at the time. Ok, I could in theory use web forums, usenet (probably also subject to your rate limiting) or whatever for this, but tbh I don't think I can in
Erik Haagsman wrote: practice - if the discussion is on a mailing list, at best I would have to sign that list to a web mail account and reply that way, and as an average user I don't see why should I make life awkward for myself like that just to make life easier for admins (and I *am* an admin, so I have to look at both sides of the coin here) if you had (say) 30 emails per hour, accumulating unused emails until you have 200, then that might work - but again, I notice you are limiting by smtp session, and a spammer could easily send 100 emails each going to 100 recipients in a single session.
On Wed, 2004-04-07 at 14:25, Dave Howe wrote:
I think 10 is a bit low.
It is, although it's more of an example value than a practical one. You'd have to get some statistics on average e-mail use from your mail servers and tune the value accordingly.
I am not really an abnormal email user - but I tend to block answer a lot of emails, and send them as fast as I type them - so I can easily send 20-30 emails in the first hour, then maybe an hour slack, then another dozen or so - depending on inbound traffic and what arguments are ongoing on my mailing lists at the time.
Same here, but this pattern of e-mail burst - slack - burst etc. could be quite easily implemented in the way described, as long as you have some accurate statistics to use as baseline values and adjust the actual operational values accordingly.
Ok, I could in theory use web forums, usenet (probably also subject to your rate limiting) or whatever for this, but tbh I don't think I can in practice - if the discussion is on a mailing list, at best I would have to sign that list to a web mail account and reply that way, and as an average user I don't see why should I make life awkward for myself like that just to make life easier for admins (and I *am* an admin, so I have to look at both sides of the coin here)
Agree, it should be transparent to the user, but again that's where accurate figures come in, and ofcourse the whole system could be as fine-grained as you like, with further limits and slack on subnet level, or by dividing into departments/organisations each with their own limits on different levels (although keeping it as simple as possible would ofcourse be preferred).
I notice you are limiting by smtp session, and a spammer could easily send 100 emails each going to 100 recipients in a single session.
Yep, that's the main problem, limiting the amount of recipients as well as SMTP connections seems to be impractical although perhaps not impossible. An average user nor running a mailing-list will not realisticly send many e-mails to >100 recipients, and when they do it's often internal distribution lists within the same domain, so limiting recipients to a sensible value might not be as hard as it sounds. It also depends on where you want the limiter. When limiting connections between the user and his outgoing SMTP server you run into the recipient problem, so you might be better of limiting outgoing connections from your SMTP server, since multiple recipients will result in multiple outgoing connections from the sending server, althoug this does make coming up with accurate values for the actual base-line limits harder. It would probably require a pretty painful initial setup where the provider tracks e-mail statistics over a period of time and either bases a general limiting value on a good analysis or tweaks the limits on a per customer basis, making the initial setup very labour intensive, but perhaps better in the long term. Instead of automatic blocking you might put in a system where the admin gets alarmed by unusually high activity above the initial limit+slack and the mail is cached but not sent out before admin intervention, allowing the admin to decide whether it's malicious mail traffic or not without disrupting normal service for the user, apart from occasional delivery delay. Regards, -- --- Erik Haagsman Network Architect We Dare BV tel: +31(0)10 7507008 fax:+31(0)10 7507005 http://www.we-dare.nl
On Tue, 6 Apr 2004 Valdis.Kletnieks@vt.edu wrote:
If you rate-limit 2 million compromised machines to 20 msgs/day each, there's only 400 million spams. Total.
this implies network operators will suddenly find a clue, something which will never happen. ever. (well, they sometimes suddenly find clue when it is forced upon them, like say with a subpoena or search warrant.) -Dan
Dan Hollis wrote:
On Tue, 6 Apr 2004 Valdis.Kletnieks@vt.edu wrote:
If you rate-limit 2 million compromised machines to 20 msgs/day each, there's only 400 million spams. Total.
this implies network operators will suddenly find a clue, something which will never happen. ever.
Clue is generally available in exchange of money. However it requires a seed of foresight or clue to hire more in. Or a business neccessity and a strike of luck. Clue only pays in the long run, so todays quarter-capitalism does not promote clue. Pete
(well, they sometimes suddenly find clue when it is forced upon them, like say with a subpoena or search warrant.)
-Dan
On Tue, 6 Apr 2004, Petri Helenius wrote:
Dan Hollis wrote:
If you rate-limit 2 million compromised machines to 20 msgs/day each, there's only 400 million spams. Total.
On Tue, 6 Apr 2004 Valdis.Kletnieks@vt.edu wrote: this implies network operators will suddenly find a clue, something which will never happen. ever. Clue is generally available in exchange of money. However it requires a seed of foresight or clue to hire more in. Or a business neccessity and a strike of luck. Clue only pays in the long run, so todays quarter-capitalism does not promote clue.
So we need to make it expensive to avoid clue. -Dan
On Tue, 06 Apr 2004 13:17:53 PDT, Dan Hollis said:
this implies network operators will suddenly find a clue, something which will never happen. ever.
Death of the Internet Predicted. Film at 11. Note that *no* anti-spam solution will work unless the network operators have enough clue to deploy it, be it rate limiting, extensions to SMTP, a replacement for SMTP, or what have you. The only reason network operators don't have clues is because there's been little to no evolutionary pressure to acquire same. At some point, there *will* have to be a weeding out....
Michael.Dillon@radianz.com wrote:
If we change the email system architecture so that every node in the architecture only accepts mail from known sources and enforces a pre-arranged behavior pattern,
Although I fail not to sound like a broken record here; Say hello to the X.400 guys when you meet them... Pete
On Apr 5, 2004, at 10:49 AM, Andy Johnson wrote:
Has anyone had any experience with this device? Turntide.com. Looks like a traffic-shaping device designed specifically for cutting down spammers throughput to your inbound SMTP servers. My main concern is, how does it make the distinction between legitimate mass-mailings (e.g.: mailing lists such as this one), and spam? Interesting approach to killing spam though I must say.
You might want to consider an inexpensive mail relay running OpenBSD's spamd (not to be confused with SpamAssassin's spamd) in conjunction with PF. Spamd is really nice for hurting spammers and/or relays where it can... in their spool. Granted, it's based on address lists like spamhaus or spews, but it's better than content filtering in one very important way... bandwidth savings. With content filtering, the payload is already at your doorstep. http://www.openbsd.org/cgi-bin/man.cgi?query=spamd http://www.benzedrine.cx/relaydb.html -- Jason Dixon, RHCE DixonGroup Consulting http://www.dixongroup.net
participants (13)
-
Andy Johnson -
Dan Hollis -
Dave Howe -
Erik Haagsman -
Jason Dixon -
Joe Abley -
Joel Jaeggli -
Matthew Crocker -
Michael.Dillon＠radianz.com -
Paul Vixie -
Petri Helenius -
Suresh Ramasubramanian -
Valdis.Kletnieks＠vt.edu