Yes, there was lots of teeth gnashing and screams of agony allegedly because MS refused to license the technology on the terms that folks wanted. MS was more than willing to let folks have it at no cost, they just weren't willing to give the naysayer everything they wanted, so everyone went home.

(that is, of course, a biased assessment, but not an unfair one)

There were two problems with the patent license that the MS lawyers offered. The first was that it reserved to them the right to stop granting new licenses, thereby pulling the rug out from under anyone who'd used licensed technology in a product, particularly an open source product. The said they didn't plan to do that, but MS' lawyers adamantly refused to change that, so a lot of us concluded that if they thought the pull out the rug language was important, so did we. The second problem was that the license was for two unpublished patent applications that they described in general terms. When the applications were published (on a schedule known from the day they were filed), they turned out to cover vastly more than MS had ever said. That left a bad taste in a lot of people's mouths. See my blog entry at http://www.taugh.com/weblog/patapp.html

In the mean time, nothing stops MS (and everyone else) from building Sender-ID into their MTAs. SPF is a standard and Sender-ID utilized SPF records to perform inbound phishing control based on PRA.

(Danger: operational content ahead.) Sender-ID and SPF have serious practical problems. The first is that they don't match the way that a lot of mail is really sent. If all of your mail comes from a single set of servers, like if you're a big company or an ESP, then SPF or Sender-ID work reasonably well to tell people which mail is yours. On the other hand, if you're a university who lets its graduates keep using their whatever.edu addresses after they graduate and forwards their mail to them, they doen't work at all. (Other than a legalistic version of "work" in which you publish a useless SPF record saying that mail could come from anywhere.) This would not be a problem except that SPF has been greatly oversold, the SPF community has not been particularly diligent in disabusing people of their misconceptions, and I can promise that the moment there is a Sender-ID checkbox in Exchange, clueless MCSEs will set it to reject anything that fails Sender-ID, it'll reject buckets of normal valid mail, and when people complain, they will insist that the mail must have been sent wrong because Sender-ID said it was spam. Even for fixed senders, Sender-ID is useless against phishing, because it can only tell you that a message purporting to be from phoop.com came from a source that is OK for phoop.com, but it cannot tell you whether phoop.com is someone you want to get mail from. This is a real problem. For example, I got mail the other day from customercenter.net purporting to tell me about the status of my MBNA credit card, with a link to mbnanetaccess.com. Was that a phish? Or consider mbna-account.com and mbna-accounts.com. One is MBNA, one is not. Does your MUA know which one is which? Spammers and phishers can publish SPF records just like anyone else, and Ciphertrust has said that of the mail they see that passes SPF, there's more spam than legit mail. I am all in favor of sender authentication, if it's real sender authentication. But Sender-ID isn't. Domainkeys will be better since it matches mail sending patterns better, but it has the same problem that a sender that's been authenticated has nothing to do with whether its mail is desired. Shameless plug: over in the anti-spam research group at asrg.sp.am I sure would like it if people were working on reputation systems to plug the gaping hole left by all these authentication schemes. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor "More Wiener schnitzel, please", said Tom, revealingly.

On 08/06/05, J.D. Falk <jdfalk@cybernothing.org> wrote:

We can't have reliable reputation until we know who the mail is coming from -- so reliable identity is a necessary first step.

What the doctor ordered seems to be something like an API that'd let you plug dkim + csv into $mta Anyone? --srs -- Suresh Ramasubramanian (ops.lists@gmail.com)

On 08/06/05, Tony Finch <dot@dotat.at> wrote:

I don't think the lack of standard APIs is a problem: it seems to me that the various MTA authors have been reasonably keen to support the various nascent specifications, especially if they have a reasonably good reference implementation library.

Easier for those of us who run multiple MTAs, especially large farms of them But yes, you do have a point that it is not exactly a problem. Is there something in the works for postfix and qmail yet? regards srs -- Suresh Ramasubramanian (ops.lists@gmail.com)

On Thu, 9 Jun 2005, Barry Shein wrote:

We've already tackled reputation systems, they were called web site certificates. You have to submit to a few fairly stringent checks on who you are, typically provide a D&B id which isn't very expensive or difficult but not all that easily defrauded w/in some reasonable parameters (it ain't bank security but it's good enough to be pretty sure you're giving your credit card info to who you think you are, damn, you hand your card to strange bartenders right?)

This is accreditation, not reputation. When you contact somebody and ask to verify your credentials this is accreditation (and it says nothing about how you're going to act, just verifies your identity and provides additional info about you). When somebody else looks at your activity and makes "subjective" judgement (mosly based on multiple reports from users) and then lets this judgement about your activities be available to other users then its reputation. -- William Leibzon Elan Networks william@elan.net

Thus spake "Barry Shein" <bzs@world.std.com>

However, I'll add my voice that I believe "reputation" systems as an approach to spam-prevention are neither useful nor sufficient w/o repeating what others have said.

Agreed. If my grandmother has a "reputation" for sending legitimate email, and she inadvertently installs some spam zombie software, it is certainly feasible (and probably trivial) for the spammer to steal all her credentials and thus her "reputation". Spam will get out for a while, but once her "reputation" significantly degrades, it will be stopped -- as will any future legitimate email from her. This "solution" strikes me as worse than the problem it tries to address. S Stephen Sprunk "Those people who think they know everything CCIE #3723 are a great annoyance to those of us who do." K5SSS --Isaac Asimov

On Thu, 09 Jun 2005 17:40:55 PDT, Matt Ghali said:

So you see, the reputation has nothing to do with your mom, and everything to do with the controlling entity, her ISP. Which makes the whole address-based sender reputation scheme almost workable, if you ignore the scaling issues.

That's suspiciously close to "Ralph Nader or Ross Perot could have been elected President, if you ignore the scaling issues". :) Other than that, what Matt said is correct - the problem is that legitimate mail can come from literally millions of places whose reputation we have no clue on....

On 10/06/05, Andre Oppermann <nanog-list@nrg4u.com> wrote:

Everyone in the SPAMwar has to be aware that SPAM can't be stopped until its transaction costs approach that of the cheapest other advertising method. That can be snailmail spam, telephone terror^Wmarketing, whatever, you name it.

The issue of course is that by making it more expensive for senders of spam, you're making it just as expensive for senders of email

Each of them can contribute to a different part of the problem and none of them can fix the entire one. IETF MARID tried to stuff too many things into one of the above systems and failed.

Authentication without backing of a reputation is not too useful, as you say. The way AOL uses spf is to just use it to let people it wants to whitelist update their whitelist records with aol on the fly, so they dont have to open a ticket with AOL each time they add a new /24 worth of outbound servers for "high volume email deployment"

Each of them has its own unique advantages and disadvantages and tackles the problem on a different layer and is under different administrative control.

Nice. Only, all this falls totally in a technical space, where you need at least two other things (policy and user awareness) to flesh the picture out. I'll be teaching a short but quite general tutorial (~ 3 hours) on spam issues at apnic 20 in Hanoi this september, based mostly on a whole lot of conclusions I've drawn in my oecd paper http://www.oecd.org/dataoecd/5/47/34935342.pdf I've designed it just for this purpose - to let operators anywhere in the world use it, teach stuff based on it .. and I'd be obliged if people who do this stuff on the NOG circuit see fit to use it that way .. regards srs -- Suresh Ramasubramanian (ops.lists@gmail.com)

Barry Shein wrote:

One useful definition of (some sorts of) insanity is doing the same thing over and over but expecting different results.

I therefore assert there is no technical solution to spam.

The ultimate solution would have to be a combination of social, technical and probably legislative. -- JustThe.net - Steve Sobol / sjsobol@JustThe.net / PGP: 0xE3AE35ED Coming to you from Southern California's High Desert, where the temperatures are as high as the gas prices! / 888.480.4NET (4638) "Life's like an hourglass glued to the table" --Anna Nalick, "Breathe"

Valdis.Kletnieks@vt.edu wrote:

Amen (mostly).

I think we can write off "legislative" until we fix the root causes that gave us the CAN-SPAM act.

That's the precise reason I said "probably". "root causes" include congresscritters in the pockets of the DMA, and that's not likely to change soon. -- JustThe.net - Steve Sobol / sjsobol@JustThe.net / PGP: 0xE3AE35ED Coming to you from Southern California's High Desert, where the temperatures are as high as the gas prices! / 888.480.4NET (4638) "Life's like an hourglass glued to the table" --Anna Nalick, "Breathe"

I therefore assert there is no technical solution to spam.

I think you're preaching to the choir here.

What will stop it is some sort of new economic model, billing for e-mail (yeah yeah some reasonable amt "included"),

Unfortunately, that's a technical solution, because it requires that we invent some sort of technology that can track all the mail, assign responsibility for postage, and do the settlements. As I've been saying for quite a while, it doesn't exist and it's not likely to, ever, because mantaining large rapidly updated databases with authentication on the updates is a fundamentally hard problem.

along with vigorous enforcement of that model against theft of service etc. Miscreants of the sort we're dealing with only understand jail time.

If it's OK with you, I'd rather skip the epostage vaporware and move directly to the enforcement. Most spammers are breaking multiple laws, even the inane CAN-SPAM act, now. Where I think technology can help is to make it easier to build cases against spammers that will stand up in court. I was the Commonwealth's technical expert in the criminal case against Jeremy Jaynes, and it was clear that kind of prosecution is much too expensive to work against any but the very largest spammers who are targeting recipients that are motivated to spend their own money to help prepare the case. R's, John -- Regards, John R. Levine, IECC, POB 727, Trumansburg NY 14886 +1 607 330 5711 johnl@iecc.com, Mayor, http://johnlevine.com, Member, Provisional board, Coalition Against Unsolicited Commercial E-mail

We've been here before, but to recap.

1. If a particular billing/business model presents difficulties then we might have to consider a different model, others are possible (hence, straw man of e-postage etc.)

I look forward to hearing about a design for an email billing system that does not require technology that is two orders of magnitude beyond the state of the art and would be effective against spammers. I hope my inability to envision one is just due to lack of imagination, but I've been asking for years and I've never seen anything even close. It's not hard to sketch out a scheme that does statistical billing and works so long as everyone plays more or less by the rules. It is far harder to come up with one that will work against determined bad guys who would be delighted if 1% of their mail leaked through without paying (or paid by other people.)

2. It would seem to say, for example, the long distance voice billing system is impossible since it would seem to have many of the same qualities you delineate as insurmountable obstacles.

Telephony is heavily regulated, has high costs of entry, has nothing comparable to the spam problem (fraudulent callers at higher volumes than legit callers), and is still subject to gross frauds like MCI. I'm thinking both of MCI's accounting frauds, and of more technical stuff like routing calls through Canada and reoriginating them at small telcos in the upper midwest to make them look like local rather than long distance traffic. Is that really the business you want to be in? It is also my impression that the number of phone calls is a whole lot less than the number of e-mail messages, returning us to the previous problem. I can't help but note that telephony is rapidly moving in the other direction, away from itemized billing. My phone service is now flat rate for calls to anywhere from Honolulu to Helsinki. Perhaps they know something.

Most importantly the big difference between those billing systems and e-mail is that there are billions and billions of dollars in those voice billing systems and the systems they support. There is very little money in e-mail,

This part, I completely agree with. No matter what anti-spam technique you propose, someone will complain that it costs too much. (This is particularly true of bulk marketers who apparently think that if e-mail were merely 100 times cheaper than paper mail rather than 1000 times cheaper, mass bankruptcies would ensue.) Spamhaus says, on the record, that MCI and SBC will not disconnect spammers for anything other than non-payment. The money is talking there. One of the few hopeful signs on the horizon is that Verizon, for all its faults, has a lot better history of booting off spammers than MCI does. R's, John

You have to manage to lower the reputation of that host within a very short amount of time to increase the transaction costs sufficiently for the spammer to make the effort worthwhile.

Or you have to ensure that the sending ISP can react quickly and stop the flow of spam so that only a trickle of messages actually enters the system

In all likelyhood it can only be achieved by a combination of simple systems/mechanisms each tailored to deal with a specific part of the problem.

Here's a simple mechanism which has not yet been tried seriously. Email server peering. This means that an SMTP server operator only accepts incoming mail from operators with whom they have a bilateral email peering agreement. Bilateral agreements have been shown to scale quite well whether you look at BGP peering or the world of business contracts. In any case, the fundamental need here is that for somebody to notify the email administrator that is sending spam and for that administrator to act immediately to cut the flow. Today, notifications arrive in a flood that smells a lot like SPAM itself. They come from millions of unknown and unvalidated and often confused sources. No email admin can afford to act without careful investigation first. But if the notifications only come from email peers who only report problems that involve the bilateral relationship, i.e. source of spam flow to destination of spam flow, then the source email administrator can afford to act immediately without any investigation. After all, this is all covered in the peering agreement. The two parties have already agreed to do this, have already agreed on notification mechanisms and have already agreed on responsibility (and penalties) for bad notifications. Because all of this is codified in a set of bilateral email peering agreements, it becomes easy to incorporate any implications in the email service level agreements with customers. In other words, if an email operator agrees to shut down IP access to an 0wn3d workstation immediately upon request of their peer, then they can put that in the customer contract so that the customer knows that SMTP is strictly forbidden without prior arrangement with their operator. SPAM is not a technical problem and there are no technical solutions. It is a social problem and the solutions are to be found in social arrangements like contracts, laws, email services associations, etc. A lot of this dubious technical garbage can be swept aside if we simply recognize that the flat structure of a completely open SMTP service is not scalable. But if we manage the structure through agreements and contracts, we can organize the exact same technology and protocols into a relatively thin hierarchy, no more than 3 or 4 levels, with a global mesh of bilateral peering agreements forming the top level. This top level could easily scale to thousands of members if it is based on a standard bilateral email peering agreement that is administered by a consortium of the top level peers. In this way, a new operator can join the consortium and simultaneously sign the bilateral agreement with all current members. I envisage the top level of this hierarchy to actually be composed of several international consortia, probably 5 of them, building on the work that has been done by the Regional Internet Registries in building up international cooperation amongst ISPs. I'm not suggesting that the RIRs would become these email consortia, simply that the RIRs are proof that ISPs can cooperate on an international level to manage Internet resources in an orderly fashion. And associated with the 5 RIRs are 5 regional gatherings of ISPs where this type of cooperation could be hammered out. What we really need to get this off the ground is for some of the large email operators to get together in public at a NANOG meeting and discuss this openly in some type of round table discussion. Who will take the first step? Perhaps the NANOG program committee? --Michael Dillon

On Mon, 13 Jun 2005, Michael.Dillon@btradianz.com wrote:

Here's a simple mechanism which has not yet been tried seriously. Email server peering.

All this stuff you're describing [almost] exactly matches the former UUCP-based e-mail distribution mechanism.

A lot of this dubious technical garbage can be swept aside if we simply recognize that the flat structure of a completely open SMTP service is not scalable.

But we already found that the UUCP system didn't scale because its management load was way too high. That's why RFC821 exists! We aren't rolling back the clock now -- the e-mail infrastructure is far too large to do that. (Oh sh*t, did I just feed a troll?) -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>

On Fri, 10 Jun 2005 Valdis.Kletnieks@vt.edu wrote: On Thu, 09 Jun 2005 17:40:55 PDT, Matt Ghali said:

So you see, the reputation has nothing to do with your mom, and everything to do with the controlling entity, her ISP. Which makes the whole address-based sender reputation scheme almost workable, if you ignore the scaling issues.

That's suspiciously close to "Ralph Nader or Ross Perot could have been elected President, if you ignore the scaling issues". :) Yes. There's a reason I did not include a ringing endorsement of sender reputation schemes as the FUSSP; it has colossal inherent scaling issues; however I believe the 90/10 rule will make it at least somewhat effective. Other than that, what Matt said is correct - the problem is that legitimate mail can come from literally millions of places whose reputation we have no clue on.... Yes. Sender reputation on an per-ip level is a lot of state. However; I believe that sender reputation on a swip level may be attainable, and provide positive value. matto PS: Even though it's painfully obvious, I speak only for myself and no entity currently/previously employing me- Especially those kooks at UCB. --matt@snark.net------------------------------------------<darwin>< The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke

Reputation is a missing element in all sender authentications schemes and will (likely) be solved separately. No approach is perfect, but building closer to a solution is preferred over sitting on our hands and debating, which (historically) seems to be the IETF's approach. - Dan On 6/8/05 12:37 AM, "John Levine" <johnl@iecc.com> wrote:

...

Yes, there was lots of teeth gnashing and screams of agony allegedly because MS refused to license the technology on the terms that folks wanted. MS was more than willing to let folks have it at no cost, they just weren't willing to give the naysayer everything they wanted, so everyone went home.

(that is, of course, a biased assessment, but not an unfair one)

There were two problems with the patent license that the MS lawyers offered. The first was that it reserved to them the right to stop granting new licenses, thereby pulling the rug out from under anyone who'd used licensed technology in a product, particularly an open source product. The said they didn't plan to do that, but MS' lawyers adamantly refused to change that, so a lot of us concluded that if they thought the pull out the rug language was important, so did we.

The second problem was that the license was for two unpublished patent applications that they described in general terms. When the applications were published (on a schedule known from the day they were filed), they turned out to cover vastly more than MS had ever said. That left a bad taste in a lot of people's mouths. See my blog entry at http://www.taugh.com/weblog/patapp.html

...

In the mean time, nothing stops MS (and everyone else) from building Sender-ID into their MTAs. SPF is a standard and Sender-ID utilized SPF records to perform inbound phishing control based on PRA.

(Danger: operational content ahead.)

Sender-ID and SPF have serious practical problems.

The first is that they don't match the way that a lot of mail is really sent. If all of your mail comes from a single set of servers, like if you're a big company or an ESP, then SPF or Sender-ID work reasonably well to tell people which mail is yours. On the other hand, if you're a university who lets its graduates keep using their whatever.edu addresses after they graduate and forwards their mail to them, they doen't work at all. (Other than a legalistic version of "work" in which you publish a useless SPF record saying that mail could come from anywhere.)

This would not be a problem except that SPF has been greatly oversold, the SPF community has not been particularly diligent in disabusing people of their misconceptions, and I can promise that the moment there is a Sender-ID checkbox in Exchange, clueless MCSEs will set it to reject anything that fails Sender-ID, it'll reject buckets of normal valid mail, and when people complain, they will insist that the mail must have been sent wrong because Sender-ID said it was spam.

Even for fixed senders, Sender-ID is useless against phishing, because it can only tell you that a message purporting to be from phoop.com came from a source that is OK for phoop.com, but it cannot tell you whether phoop.com is someone you want to get mail from. This is a real problem. For example, I got mail the other day from customercenter.net purporting to tell me about the status of my MBNA credit card, with a link to mbnanetaccess.com. Was that a phish? Or consider mbna-account.com and mbna-accounts.com. One is MBNA, one is not. Does your MUA know which one is which? Spammers and phishers can publish SPF records just like anyone else, and Ciphertrust has said that of the mail they see that passes SPF, there's more spam than legit mail.

I am all in favor of sender authentication, if it's real sender authentication. But Sender-ID isn't. Domainkeys will be better since it matches mail sending patterns better, but it has the same problem that a sender that's been authenticated has nothing to do with whether its mail is desired.

Shameless plug: over in the anti-spam research group at asrg.sp.am I sure would like it if people were working on reputation systems to plug the gaping hole left by all these authentication schemes.

Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor "More Wiener schnitzel, please", said Tom, revealingly.

-- Daniel Golding Network and Telecommunications Strategies Burton Group

On Wed, 8 Jun 2005, william(at)elan.net wrote:

You miss the point. Reputation is already widely being used today - every blocklist is a reputation system

Strike that point. Not every blocklist is reputation system, though the most widely used ones like SBL, SORBS, SPAMCOP are all like that. But there others simply provide certain info about corresponding ip or domain which is derived from domain info itself (like what country its from, if its bogon/not-allocated space for ip, etc), those would be closer accreditation data. I seriously doubt this distinction is that important for those at nanog. But anyway, I had to correct myself, since my statement was not accurate. -- William Leibzon Elan Networks william@elan.net

On 6/8/2005 12:37 AM, John Levine wrote:

I am all in favor of sender authentication, if it's real sender authentication.

I don't disagree with anything you said, and I certainly agree with your closing sentiment. Having said that, SPF and Sender-ID are useful as another hammer in the bag-o-tricks, even if they aren't useful for everybody, or if they don't solve everything, or if their authors misrepresent the technology, or any of the other number of things that you didn't mention. At the least, being able to say that mail from my domain is really only from me if it came from my fixed server[s] is very useful, even if it does nothing else. They are useful as proof-of-concept works, too. There have been many demands to produce something like SPF/Sender-ID for many years (myself included), and just having them out there is useful for research and analysis purposes. Without them, folks would still be arguing to produce them, at the least. As you've argued on ASRG yourself, the research is worthwhile in and of itself, even if it produces something unexpected. -- Eric A. Hall http://www.ehsco.com/ Internet Core Protocols http://www.oreilly.com/catalog/coreprot/