I'm having a bit of an argument with a customer over the command syntax in RFC 2821 that shows command arguments for MAIL/RCPT commands in brackets, i.e.: Path = "<" [ A-d-l ":" ] Mailbox ">" Mailbox = Local-part "@" Domain Our mail servers reject connections that don't follow the RFC. Am I wrong to do this? This guy certainly thinks so, even after I've cited the RFC and appropriate section, his arguments is "even hotmail does not keep up the standards" and if we ignore them as well. Hotmail at least follows this rule... am I wrong to require the brackets? ~Seth
his arguments is "even hotmail does not keep up the standards" and if we ignore them as
It sounds like he's admitting that the RFC is correct, but since Hotmail will accept "malformed" messages, so should everyone else. He basically wants you to adjust your server to be more lax so he doesn't have to adjust his to get in line with the RFC. Depending on your business and how important mail is you may or may not be able to tell him to fix his MTA. Personally, if his mail isn't spam I'd lean more toward the "let it through" side of the argument for the sake of cooperation, though if he followed the standards there wouldn't be a problem to begin with. That may be worth pointing out. -Justin Scott
On Jan 4, 2008, at 10:51 AM, Seth Mattinen wrote:
I'm having a bit of an argument with a customer over the command syntax in RFC 2821 that shows command arguments for MAIL/RCPT commands in brackets, i.e.:
Path = "<" [ A-d-l ":" ] Mailbox ">" Mailbox = Local-part "@" Domain
Our mail servers reject connections that don't follow the RFC. Am I wrong to do this? This guy certainly thinks so, even after I've cited the RFC and appropriate section, his arguments is "even hotmail does not keep up the standards" and if we ignore them as well. Hotmail at least follows this rule... am I wrong to require the brackets?
No you are not wrong for following the RFC. Customer shouldn't be running broken software that doesn't follow the RFCs. Point out to customer that AOL requires the brackets as well. and that is probably a better example to use that Hotmail. 250 rly-dc04.mx.aol.com OK mail from: doon@inoc.net 501 SYNTAX ERROR IN PARAMETERS OR ARGUMENTS mail from: <doon@inoc.net> 250 OK -Patrick -- Patrick Muldoon Network/Software Engineer INOC (http://www.inoc.net) PGPKEY (http://www.inoc.net/~doon) Key ID: 0x370D752C "Wooh, All this computer hacking is making me thirsty, I think I'll order a Tab" --Homer Simpson
On Jan 4, 2008 10:51 AM, Seth Mattinen <sethm@rollernet.us> wrote:
Our mail servers reject connections that don't follow the RFC. Am I wrong to do this?
Seth, RFC 1122 (Requirements for Internet Hosts - Communication Layers) section 1.2.2 (Robustness Principle): "Be liberal in what you accept, and conservative in what you send" Regards. Bill Herrin -- William D. Herrin herrin@dirtside.com bill@herrin.us 3005 Crane Dr. Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
RFC 1122 (Requirements for Internet Hosts - Communication Layers) section 1.2.2 (Robustness Principle):
"Be liberal in what you accept, and conservative in what you send" Which has done nothing but allow countless Exchange server to HELO with exchange.local and other such idiocy, not to mention contribute to
the spam problem. Every time you try to configure your server for strict parsing to help block spam, you get an email from some admin who doesn't know the first thing about SMTP but wants to know why you're blocking their email. -Don
On Fri, 4 Jan 2008, Donald Stahl wrote:
RFC 1122 (Requirements for Internet Hosts - Communication Layers) section 1.2.2 (Robustness Principle):
"Be liberal in what you accept, and conservative in what you send" Which has done nothing but allow countless Exchange server to HELO with exchange.local and other such idiocy, not to mention contribute to the spam problem.
Yep, which is why I suggest the modification: You may be liberal in what you accept, and you should always be conservative in what you send.
i call it do-gooder software. when it fixes your bad send, you never say thanks. when it makes a mistake in doing so, i catch hell. no thanks randy, a naggumite
On Jan 4, 2008 10:51 AM, Seth Mattinen <sethm@rollernet.us> wrote:
Our mail servers reject connections that don't follow the RFC. Am I wrong to do this?
Seth,
RFC 1122 (Requirements for Internet Hosts - Communication Layers) section 1.2.2 (Robustness Principle):
"Be liberal in what you accept, and
That particular philosophy has done great wonders for e-mail and the spam problem, been a key issue on both the penetration and implementation sides of firewall design, etc. "Liberal," when defined as "accept anything you reasonably can, and try to deal with it," appears to be a policy that has had an overall negative impact on protocol design and interoperability on the Internet. "Liberal," if defined instead as "must accept anything in compliance with RFC sender-side MUST/SHOULD/MAY's, and should reject as much of anything else as you can figure out to," would be a better way to have defined "liberal." I think I would have preferred the word "robust" instead of "liberal." This would have spared us the agony of systems that are "smart" enough to go direct-to-MX, but not smart enough to send a valid FROM line.
conservative in what you send"
If only a more significant percentage of software was written in that manner... ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Jan 4, 2008 11:27 AM, Joe Greco <jgreco@ns.sol.net> wrote:
"Be liberal in what you accept, and
That particular philosophy has done great wonders for e-mail and the spam problem
Joe, I've heard similarly unsubstantiated versions of this claim over and over. The fact is I've done quite a bit of development on anti-spam systems and the only protocol violation that has been consistently valuable for rejecting spam is the fire-and-forget violation. That's the one where they pipeline the entire send-side of the conversation in the first data packet without waiting for the banner or checking each response as it comes back. Its a terribly tempting optimization to the spam-sending process and not enough servers detect or reject it. Anti-spam activity at the protocol level is looking for behavioral signatures unique to spammer software. Protocol-correct signatures are just as valuable as protocol-incorrect ones but its all a game of whac-a-mole. Once a signature is identified and promulgated, the software exhibiting it either versions or falls out of use. A few months later the folks still nailed are the false positives.
conservative in what you send"
If only a more significant percentage of software was written in that manner...
I'll second that sentiment. Seth's customer is unambiguously wrong. Unfortunately, that doesn't make Seth right. Missing brackets has been a common SMTP error since the inception of the protocol, second only to incorrect end-of-line (LF instead of CRLF). If you want your implementation to be robust, you have to silently allow those common mistakes. Regards, Bill Herrin -- William D. Herrin herrin@dirtside.com bill@herrin.us 3005 Crane Dr. Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Jan 4, 2008 11:27 AM, Joe Greco <jgreco@ns.sol.net> wrote:
"Be liberal in what you accept, and
That particular philosophy has done great wonders for e-mail and the spam problem
Joe,
I've heard similarly unsubstantiated versions of this claim over and over.
All right, here you go, then: You already cover pre-banner pipelining. Failure to reject HELO that is missing. This situation is generated often by direct-to-MX mail submission programs for webforms, etc. While most of these direct-to-the-local-MX, and are therefore of less concern from a global interoperability point of view, some of them are sufficiently smart to do the MX lookup for the destination, and connect to somewhere remote. Failure to reject HELO that is simply wrong ("exchange.local" or whatever it is). The guidelines for sending should require a host to determine its legitimate PTR, or, alternatively, at least a valid hostname (for those behind NAT). A mail server should refuse to run if it cannot even determine its own name can be resolved. Failure to include an anti-blowback mechanism within SMTP. Specifically, there's a wide-open default-valid sender of "<>", and some sites actually disallow this sender because spammers use it. This is horrifying, because it breaks mail failure notification, which further breaks the reliability of e-mail. One solution would have been for SMTP to include a consistent way for mail servers to be able to identify legitimate returns. The implementation of this that I've seen simply records Message-ID's on the way out, and should a NDN return which contains such a Message-ID, it is passed on. This requires database support at the sending site, which may be too high a bar. Had a method to sign a message, and guarantee that the signature be returned by any reporting MTA, been included in SMTP, it would have gone a long way to keep this working. There are still problems with it, of course, and the logical evolution moves on towards some of the solutions we've seen, but the failure to provide it out of the box and make sure every site supports it has created a mess. Failure to require a legitmate e-mail address in From:. Failure to require something like SPF from day 1, which might have made SPF useful. Do you want me to substantiate any more? I'm sure I can keep going.
The fact is I've done quite a bit of development on anti-spam systems and the only protocol violation that has been consistently valuable for rejecting spam is the fire-and-forget violation.
If you choose to count only "consistently valuable," then, you eliminate a bunch of things that are in fact pretty good spamsigns, but which are not 100% reliable.
That's the one where they pipeline the entire send-side of the conversation in the first data packet without waiting for the banner or checking each response as it comes back.
Except there's a bunch of old web form submission scripts that do that, so that's not 100% either.
Its a terribly tempting optimization to the spam-sending process and not enough servers detect or reject it.
Just as pretty much any other anti-spam optimization is terribly tempting.
Anti-spam activity at the protocol level is looking for behavioral signatures unique to spammer software. Protocol-correct signatures are just as valuable as protocol-incorrect ones but its all a game of whac-a-mole. Once a signature is identified and promulgated, the software exhibiting it either versions or falls out of use. A few months later the folks still nailed are the false positives.
The ideal thing is to key on things which legitimate mail servers should not be doing. Some of these are things which are difficult for a spammer to work around. Consider a policy that says: "Do not accept e-mail from an IP address on a dial-up users blocklist."
I'll second that sentiment. Seth's customer is unambiguously wrong. Unfortunately, that doesn't make Seth right. Missing brackets has been a common SMTP error since the inception of the protocol, second only to incorrect end-of-line (LF instead of CRLF). If you want your implementation to be robust, you have to silently allow those common mistakes.
Silently allowing those common mistakes is what allows users of those mail server products to think that their mail server is okee-dokee. If they were to find themselves unable to mail most places, the error (and whose error it was) would be more clear to them. Whether any particular error is worthy of being silently allowed is not a discussion for this forum, but I would say that a majority of them could have and should have been disallowed from Day 1. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Fri, Jan 04, 2008 at 10:27:47AM -0600, Joe Greco wrote:
That particular philosophy has done great wonders for e-mail and the spam problem,
I completely agree. If it weren't for that philosophy, we wouldn't have an email problem at all. A -- Andrew Sullivan 204-4141 Yonge Street Afilias Canada Toronto, Ontario Canada <andrew@ca.afilias.info> M2P 2A8 jabber: ajsaf@jabber.org +1 416 646 3304 x4110
On Jan 4, 2008 5:52 PM, Andrew Sullivan <andrew@ca.afilias.info> wrote:
I completely agree. If it weren't for that philosophy, we wouldn't have an email problem at all.
A
Because....we wouldn't have e-mail? Consider the pain of getting worldwide interoperability for a "notmail" system that insisted on strict validation...
Alexander Harrowell wrote:
Because....we wouldn't have e-mail? Consider the pain of getting worldwide interoperability for a "notmail" system that insisted on strict validation...
The SMTP ship has already sailed, so trying to change the behavior of email would be difficult. I do, however, reject the notion that strict validation make implementation of interoperability painful. If the specifications are clearly defined, rather than allowing interpretation by the implementer, then interoperability would be almost assured. The problem is that many specifications in RFCs are loose and left open to interpretation by the individual software programmers. But, to the original question... If the customer's email is important to the business, then you may want to accept the email that may not be complaint to a strict interpretation of the RFC. -Sean
On Fri, Jan 04, 2008 at 11:32:13AM -0700, Sean Figgins wrote:
Alexander Harrowell wrote:
Because....we wouldn't have e-mail? Consider the pain of getting worldwide interoperability for a "notmail" system that insisted on strict validation...
The SMTP ship has already sailed, so trying to change the behavior of email would be difficult.
I do, however, reject the notion that strict validation make implementation of interoperability painful. If the specifications are clearly defined, rather than allowing interpretation by the implementer, then interoperability would be almost assured. The problem is that many specifications in RFCs are loose and left open to interpretation by the individual software programmers.
There's another old saying: Quality, Schedule, Features: Pick 2. It applies to specifications as well as implementations. This is why the robustness principle is important (and IMO ought to be followed); it recognizes that there might be communication in the absence of perfect specification or implementation, and that it's valuable (in general) to let that communication proceed. (An argument to the contrary is that this principle was introduced at a time when there was a much lower incidence of "unwanted traffic", particularly that which strongly correlated to protocol violations.) --gregbo
Folks, Let's bring this one to closure. The authors question is answered and this is backing itself into an endless thread with arguments better suited for the IETF vs. NANOG. Best Regards, Martin Hannigan NANOG Mailing List Committee On Jan 4, 2008 1:02 PM, Alexander Harrowell <a.harrowell@gmail.com> wrote:
On Jan 4, 2008 5:52 PM, Andrew Sullivan <andrew@ca.afilias.info> wrote:
I completely agree. If it weren't for that philosophy, we wouldn't have an email problem at all.
A
Because....we wouldn't have e-mail? Consider the pain of getting worldwide interoperability for a "notmail" system that insisted on strict validation...
I completely agree. If it weren't for that philosophy, we wouldn't have an email problem at all. Because....we wouldn't have e-mail? Consider the pain of getting worldwide interoperability for a "notmail" system that insisted on strict validation...
consider the pain of getting worldwide interoperabilty for a "notmail" system where following the specs was thrown to the wind. randy
Our mail servers reject connections that don't follow the RFC. Am I wrong to do this? Seth, RFC 1122 (Requirements for Internet Hosts - Communication Layers) section 1.2.2 (Robustness Principle):
"Be liberal in what you accept, and conservative in what you send"
I would say this quotation of RC1122 1.2.2 seems a little bit out of context. "being liberal in what you accept" in terms of such egregious errors as a mailformed destination address does not mean you necessarily pass the message on. The robustness principal doesn't say you have to correct the error and accept the message, continuing normally, as if there was no error, it says you're prepared for it To be robust, your SMTP server implementation must not crash if a user omits the angle brackets, it must be prepared to accept the malformed text, and respond accordingly, I.E. Refuse the message, with an error response, if it doesn't recognize the proper destination format. The following paragraph of RFC1122 states: "Software should be written to deal with every conceivable error, no matter how unlikely; sooner or later a packet will come in with that particular combination of errors and attributes, and unless the software is prepared, chaos can ensue. In general, it is best to assume that the network is filled with malevolent entities that will send in packets designed to have the worst possible effect. This assumption will lead to suitable protective design, although the most serious problems in the Internet have been caused by unenvisaged mechanisms triggered by low-probability events; mere human malice would never have taken so devious a course! " -- -J
On Fri, Jan 04, 2008 at 07:51:15AM -0800, Seth Mattinen wrote: [snip]
Our mail servers reject connections that don't follow the RFC. Am I wrong to do this? This guy certainly thinks so, even after I've cited the RFC and appropriate section, his arguments is "even hotmail does not keep up the standards" and if we ignore them as well. Hotmail at least follows this rule... am I wrong to require the brackets?
"The" RFC is amusing WRT email. In this case, you are ignoring 3464, section 2 "Format of a Delivery Status Notification", viz: Whenever an SMTP transaction is used to send a DSN, the MAIL FROM command MUST use a NULL return address, i.e., "MAIL FROM:<>". Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Fri, Jan 04, 2008 at 07:51:15AM -0800, Seth Mattinen wrote:
Our mail servers reject connections that don't follow the RFC. Am I wrong to do this?
Nope. His software is either misconfigured or broken. I'm aware of the be-liberal-in-what-you-accept philosophy (and followed it for many years, still do in some cases), but no longer think it's applicable to inbound SMTP. The way I've put it is that the more someone's SMTP sender looks like a spam source -- say, missing rDNS or HELOing as something invalid or not sending valid SMTP commands -- the more likely it is that someone's mail system will reject, defer or quarantine the traffic. So it's in everyone's interest to make sure their outbound SMTP traffic is as compliant with de jure and de facto requirements as they possibly can -- doubly so given that this is quite often something as simple as fixing a HELO string or creating a single PTR record or something similar. ---Rsk
On Fri, 04 Jan 2008 07:51:15 PST, Seth Mattinen said:
I'm having a bit of an argument with a customer over the command syntax in RFC 2821 that shows command arguments for MAIL/RCPT commands in brackets, i.e.:
Path = "<" [ A-d-l ":" ] Mailbox ">" Mailbox = Local-part "@" Domain
Our mail servers reject connections that don't follow the RFC. Am I wrong to do this? This guy certainly thinks so, even after I've cited
The best reason I've come up with for rejecting mail from software so totally brain-dead that they can't get the < > around an address right is this: If they botch this, they've probably botched a bunch of other stuff, and accepting mail from them is almost certain to lead to grief once you send the '250 OK' after DATA. If they couldn't get < > *around* the address in the MAIL FROM:, what makes you think the address is valid? Where does any ensuing bounce end up? And more importantly, can you find out what crapware is that brain-dead and let us know, so that those of us who believe in enforcing standards can shun it? :)
participants (18)
-
Alexander Harrowell -
Andrew Sullivan -
Donald Stahl -
Greg Skinner -
James Hess -
Jeff Kell -
Joe Greco -
Joe Provo -
Justin Scott -
Martin Hannigan -
Patrick Muldoon -
Randy Bush -
Rich Kulawiec -
Sean Donelan -
Sean Figgins -
Seth Mattinen -
Valdis.Kletnieks＠vt.edu -
William Herrin