Hi, trying to pin down why so much email isn't making it recently. We see issues with various big ISPs. The most obvious is none of the three UK ISPs I have ready access to can connect to port 25 on relay.verizon.net. (MX for all the verizon.net email addresses). We can ping it (I'm sure it isn't singular?), but we have no more luck delivering email than contacting verizon technical staff, logs suggests we are in day 3 of this. I'm now listening to hold music at International rates - ouch. Have a support call with GXN open on connectivity to hotmail, which seems to vary with UK ISP, but is in general pretty poor from all of them. In general I'd say off topic in Nanog, but there comes a point where you start wondering if the routing is messed up. Am I alone before posting more details. Simon
On Fri, Dec 10, 2004 at 02:43:21PM +0000, Simon Waters wrote:
The most obvious is none of the three UK ISPs I have ready access to can connect to port 25 on relay.verizon.net. (MX for all the verizon.net email addresses). We can ping it (I'm sure it isn't singular?), but we have no more luck delivering email than contacting verizon technical staff, logs suggests we are in day 3 of this. I'm now listening to hold music at International rates - ouch.
I think I can shine a little bit of light on what might be your Verizon problem. Summary: Verizon has put in place an exceedingly stupid "anti-spam" system which does not work, which facilitates DoS attacks, and which provides active assistance to spammers. Verizon has been told all of this, and it's been discussed on Spam-L. If there's been a response from Verizon, I haven't seen it: and AFAIK the practice continues. Anyone trying to deliver mail there might want to at least skim this to get an idea of the issues they may bump into. Please note that in places this is sketchy because it seems impossible to get Verizon to provide the information necessary to make it otherwise (or correct any errors). Details: When an incoming SMTP connection is made to one of Verizon's MX's, they allow it to proceed until the putative sender is specified, i.e. they wait for this part of the SMTP transaction: MAIL From:<blah@example.com> Then they pause the incoming connection. And then they start up an outbound SMTP connection from somewhere else on Verizon's network, back to one of the MX's for example.com. They then attempt to verify that "blah" is a valid, deliverable address there. Since most people have long since disabled SMTP VRFY, they actually construct a fake message and attempt delivery with RCPT. If delivery looks like it's going to succeed, they hang up this connection (which is rude), and un-pause the incoming one, and allow it to proceed. If delivery looks like it's going to fail, then they also hang up their outbound connection (still rude), un-pause the incoming one, and reject the traffic. This also means that if the MX they try to connect to is (a) busy (b) down (c) unaware of all the deliverable addresses (d) something else, that they'll refuse the incoming message. It also means that if the address that's trying to send mail to Verizon is something like "support@thuleracks.com", which is the address that the people at Thule Racks emit support traffic from, but which doesn't accept traffic, that Verizon will deny the message. (Yeah, this isn't very bright on Thule's part, either.) Whoops. This is bad for a whole bunch of reasons: two of the more obvious ones are (a) it's a pathetic "anti-spam" measure because ANY forged address ANYWHERE will do, and (b) it doesn't scale. Add to that (c) it abuses RCPT because apparently Verizon is unwilling to use VRFY and to accept the decision of many mail server operators to disable it. Oh, and (d) the behavior of their probe systems is nearly indistinguishable from that of spam-spewing zombies, which don't obey the SMTP protocol either. [ (b) is also how it lends itself to DoS attacks. Sure, Verizon could rate-limit the rate at which they make outbound connections, but then attacker X could impose significant delay on mail from domain Y just by forging a boatload of messages purporting to be from addresses in Y to Verizon. If Verizon rate-limits their outbound connections, then any real messages from Y will be stuck in the verification queue along with a kazillion forgeries. And beyond that: other people are foolishly adopting this "callback" nonsense as well. Slashdot carried a note the other day about a program _designed_ to do this. This allows attacker X to forge messages from domain Y to idiots I1, I2... In, for a very large "n", and then stand back as all of them simultaneously try to connect to the MX's for domain Y. General principle: any "anti-spam" measure that generates more junk SMTP traffic at a time when we're drowning in it is probably a bad idea. ] One thing that's not clear is whether or not Verizon caches any of this information. Doing so might help cut down on DoS attack methods that involve them, but of course it doesn't do anything about those which leverage everyone else who's doing "callbacks". And this is unfortunately, not the end of it. A lot of people, including me, are blocking particularly problematic spammer-controlled networks at (a) our border routers (b) our firewalls or (c) our mail servers. In other words, we not only won't accept mail from them, we won't even allow them to connect: we're blocking all IP traffic from them. This prevents them from spamming (at least directly from their own network space); it also prevents them from using their resources to build lists of deliverable addresses to sell to other spammers by poking at our mail servers. Since Verizon is doing this probing *from their network*, spammers can easily get around all of our blocking by getting Verizon to do it for them. They can thus use Verizon to build/check their lists...and there's no way for us to find out who's on the other side of these probes. Which means that Verizon is running a free, anonymizing, spam support service. Note that refusing Verizon's probes, either accidentally or deliberately, will of course cause Verizon to deny incoming mail. Which I suppose might be okay for some folks, it isn't workable here. So for now all we can do is explain to our users that it's causing problems and deal with it. But if you look in your MTA logs and find things like: Jul 15 07:24:51 <XXXXXXXXXXXXXXXXX@gsp.org>... User unknown Jul 15 07:24:51 lost input channel from sc014pub.verizon.net [206.46.170.58] to MTA after rcpt Jul 15 07:24:51 from=<antispam332078@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc014pub.verizon.net [206.46.170.58] then you can see what they're doing. ---Rsk
While I can't speak to what Verizon is using, Both Exim and Postfix have the very same feature called "address verification". Its in use at a number of ISPs. My systems reject 1000's of messages every day because of verification failures. Roy Engehausen -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Rich Kulawiec Sent: Friday, December 10, 2004 9:27 AM To: nanog@merit.edu Subject: Re: verizon.net and other email grief On Fri, Dec 10, 2004 at 02:43:21PM +0000, Simon Waters wrote:
The most obvious is none of the three UK ISPs I have ready access to can connect to port 25 on relay.verizon.net. (MX for all the verizon.net email addresses). We can ping it (I'm sure it isn't singular?), but we have no more luck delivering email than contacting verizon technical staff, logs suggests we are in day 3 of this. I'm now listening to hold music at International rates - ouch.
I think I can shine a little bit of light on what might be your Verizon problem. Summary: Verizon has put in place an exceedingly stupid "anti-spam" system which does not work, which facilitates DoS attacks, and which provides active assistance to spammers. Verizon has been told all of this, and it's been discussed on Spam-L. If there's been a response from Verizon, I haven't seen it: and AFAIK the practice continues. Anyone trying to deliver mail there might want to at least skim this to get an idea of the issues they may bump into. Please note that in places this is sketchy because it seems impossible to get Verizon to provide the information necessary to make it otherwise (or correct any errors). Details: When an incoming SMTP connection is made to one of Verizon's MX's, they allow it to proceed until the putative sender is specified, i.e. they wait for this part of the SMTP transaction: MAIL From:<blah@example.com> Then they pause the incoming connection. And then they start up an outbound SMTP connection from somewhere else on Verizon's network, back to one of the MX's for example.com. They then attempt to verify that "blah" is a valid, deliverable address there. Since most people have long since disabled SMTP VRFY, they actually construct a fake message and attempt delivery with RCPT. If delivery looks like it's going to succeed, they hang up this connection (which is rude), and un-pause the incoming one, and allow it to proceed. If delivery looks like it's going to fail, then they also hang up their outbound connection (still rude), un-pause the incoming one, and reject the traffic. This also means that if the MX they try to connect to is (a) busy (b) down (c) unaware of all the deliverable addresses (d) something else, that they'll refuse the incoming message. It also means that if the address that's trying to send mail to Verizon is something like "support@thuleracks.com", which is the address that the people at Thule Racks emit support traffic from, but which doesn't accept traffic, that Verizon will deny the message. (Yeah, this isn't very bright on Thule's part, either.) Whoops. This is bad for a whole bunch of reasons: two of the more obvious ones are (a) it's a pathetic "anti-spam" measure because ANY forged address ANYWHERE will do, and (b) it doesn't scale. Add to that (c) it abuses RCPT because apparently Verizon is unwilling to use VRFY and to accept the decision of many mail server operators to disable it. Oh, and (d) the behavior of their probe systems is nearly indistinguishable from that of spam-spewing zombies, which don't obey the SMTP protocol either. [ (b) is also how it lends itself to DoS attacks. Sure, Verizon could rate-limit the rate at which they make outbound connections, but then attacker X could impose significant delay on mail from domain Y just by forging a boatload of messages purporting to be from addresses in Y to Verizon. If Verizon rate-limits their outbound connections, then any real messages from Y will be stuck in the verification queue along with a kazillion forgeries. And beyond that: other people are foolishly adopting this "callback" nonsense as well. Slashdot carried a note the other day about a program _designed_ to do this. This allows attacker X to forge messages from domain Y to idiots I1, I2... In, for a very large "n", and then stand back as all of them simultaneously try to connect to the MX's for domain Y. General principle: any "anti-spam" measure that generates more junk SMTP traffic at a time when we're drowning in it is probably a bad idea. ] One thing that's not clear is whether or not Verizon caches any of this information. Doing so might help cut down on DoS attack methods that involve them, but of course it doesn't do anything about those which leverage everyone else who's doing "callbacks". And this is unfortunately, not the end of it. A lot of people, including me, are blocking particularly problematic spammer-controlled networks at (a) our border routers (b) our firewalls or (c) our mail servers. In other words, we not only won't accept mail from them, we won't even allow them to connect: we're blocking all IP traffic from them. This prevents them from spamming (at least directly from their own network space); it also prevents them from using their resources to build lists of deliverable addresses to sell to other spammers by poking at our mail servers. Since Verizon is doing this probing *from their network*, spammers can easily get around all of our blocking by getting Verizon to do it for them. They can thus use Verizon to build/check their lists...and there's no way for us to find out who's on the other side of these probes. Which means that Verizon is running a free, anonymizing, spam support service. Note that refusing Verizon's probes, either accidentally or deliberately, will of course cause Verizon to deny incoming mail. Which I suppose might be okay for some folks, it isn't workable here. So for now all we can do is explain to our users that it's causing problems and deal with it. But if you look in your MTA logs and find things like: Jul 15 07:24:51 <XXXXXXXXXXXXXXXXX@gsp.org>... User unknown Jul 15 07:24:51 lost input channel from sc014pub.verizon.net [206.46.170.58] to MTA after rcpt Jul 15 07:24:51 from=<antispam332078@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc014pub.verizon.net [206.46.170.58] then you can see what they're doing. ---Rsk
On Fri, 10 Dec 2004, Roy wrote:
While I can't speak to what Verizon is using, Both Exim and Postfix have the very same feature called "address verification". Its in use at a number of ISPs. My systems reject 1000's of messages every day because of verification failures.
That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net. This doesn't scale. ========================================================== Chris Candreva -- chris@westnet.com -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/
Christopher X. Candreva wrote:
That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net.
This doesn't scale.
How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting. -- Best Regards, Paul D Trebilco Systems Administrator Server101.com ---------------------------------- Fast and Reliable Website Hosting! http://www.server101.com/
----- Original Message ----- From: "Paul Trebilco" <ptreb@server101.com> To: <nanog@merit.edu> Sent: Friday, December 10, 2004 3:30 PM Subject: Re: verizon.net and other email grief
How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting.
oh, so you would be ok with someone joe-jobbing you on their 1 million messages/day spam run and getting 1 million 'verification' connections to your mailserver farm? -p --- paul galynin
--On Friday, December 10, 2004 15:38 -0500 Paul G <paul@rusko.us> wrote:
----- Original Message ----- From: "Paul Trebilco" <ptreb@server101.com> To: <nanog@merit.edu> Sent: Friday, December 10, 2004 3:30 PM Subject: Re: verizon.net and other email grief
How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting.
oh, so you would be ok with someone joe-jobbing you on their 1 million messages/day spam run and getting 1 million 'verification' connections to your mailserver farm?
Far less traffic than the bounces would create at both ends. Yes this doesn't prevent it from happening if the address is real, but that's why I mentioned SPF in my previous email......That helps to verify the sender can send email for a given domain, and if that passes, then you want to see if the sender exists, if both pass then you can go on to other methods. OF course I'd first check blacklists before any of this, but that's my personal preference.
--On Friday, December 10, 2004 12:30 -0800 Paul Trebilco <ptreb@server101.com> wrote:
Christopher X. Candreva wrote:
That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net.
This doesn't scale.
How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting.
The other part is that you CACHE the answer you get (good, bad, or indifferent). I think that SPF+sender address verification is a GOOD thing when properly implemented. Yes it can be a bit of a hassle, but you shouldn't be sending mail you're not prepared to bounce. That said, none of my sites are running a current enough version of Postfix to do this.
On Fri, 10 Dec 2004, Christopher X. Candreva wrote:
That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net.
That is already what happens when spammer forged your address - you see 1000's people sending you bounces and nastygrams. The real solution is to use some form of protection for envelope mail-from address so that it could not be so easily spoofed and forged. There are currently several proposals on the table on how to do it and some of the proposals are already being used on the internet in experimental ways: SPF (dns records listing ips of mail systems that can send mail with given envelope mail-from domain). For more information see: http://spf.pobox.com http://www.openspf.org http://www.spfhelp.org CSV with MPR records (similar SPF but provides list of mail-server hostnames that can use MAIL-FROM domain, the spoofing of mail-server names is protected based on EHLO by CSV): For more information see: http://www.csvmail.com/draft-otis-marid-mpr-00.html (and for CSV see http://mipassoc.org/csv/index.html) BATV (replacement of your real mail-from address with special private connection-specific address - this allows to /dev/null bad bounces if they come back to you and you did not send the email). For more information see: http://mipassoc.org/batv/index.html SES (predates BATV and similar technique, except that a HMAC encrypted address can confirmed by means of public server which allows email to be dropped at recepient instead of dropped at the source as being bad bounce as with BATV). For more information see: http://ses.codeshare.ca/ -- William Leibzon Elan Networks william@elan.net
----- Original Message ----- From: "Roy" <garlic@garlic.com> To: "Rich Kulawiec" <rsk@gsp.org>; <nanog@merit.edu> Sent: Friday, December 10, 2004 2:23 PM Subject: RE: verizon.net and other email grief
While I can't speak to what Verizon is using, Both Exim and Postfix have
the
very same feature called "address verification". Its in use at a number of ISPs. My systems reject 1000's of messages every day because of verification failures.
i've never seen this done with postfix, but i know that exim's default 'address verification' for non-local addresses just checks that the domain in the from is valid and that an mx record exists for it. they also have what they call 'callout verification', which is equivalent to what is being discussed, but the documentation makes the drawbacks painfully clear and suggests that it only be used against hosts within the same organization. i'm not a fan of exim, but it appears that although they've given users the rope, they've been diligent enough to label it appropriately. -p --- paul galyinin
Paul G <paul@rusko.us> wrote:
[...] they also have what they call 'callout verification', which is equivalent to what is being discussed, but the documentation makes the drawbacks painfully clear and suggests that it only be used against hosts within the same organization.
No, that caveat is given for *recipient callforward verification* which is dangerous if turned on blindly. I know, I tried it for a very short while :)
i'm not a fan of exim, but it appears that although they've given users the rope, they've been diligent enough to label it appropriately.
Sender callback verification is a different beast and is highly effective against spam. It does of course not come without its price of false positives caused by misconfigured senders. Unlike other mechanisms, it at least doesn't inconvenience senders who haven't botched their mail system. The only false positives I see are things like web sites that mail from a webserver role account which doesn't have a mailbox. Even so, ecommerce sites are learning to not do this, and ordered goods usually turn up regardless of whether or not an automatically-generated confirmation email arrives. -- PGP key ID E85DC776 - finger abuse@mooli.org.uk for full key
On Fri, 10 Dec 2004, Jeffrey I. Schiller wrote:
On Fri, Dec 10, 2004 at 12:26:59PM -0500, Rich Kulawiec wrote:
One thing that's not clear is whether or not Verizon caches any of this information.
It appears that they do some amount of caching.
-Jeff
It does not appear that they are caching it, here is a sample from my log file: Dec 6 19:18:15 white sm-mta[25976]: iB70IF6n025976: <lopqp@abc.net>... User unknown Dec 6 19:18:15 white sm-mta[25977]: iB70IF6n025977: <lopqp@abc.net>... User unknown Dec 6 19:18:16 white sm-mta[25976]: iB70IF6n025976: from=<>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182] Dec 6 19:18:16 white sm-mta[25977]: iB70IF6n025977: from=<>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68] Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: <lopqp@abc.net>... User unknown Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: <lopqp@abc.net>... User unknown Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: lost input channel from sc006pub.verizon.net [206.46.170.182] to MTA after rcpt Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: from=<antispam245967@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182] Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: lost input channel from sc019pub.verizon.net [206.46.170.68] to MTA after rcpt Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: from=<antispam830748@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68] What happens when verizon tries to send email to somebody who does the same type of check, does this not create an infinite loop? K
Krzysztof Adamski wrote:
On Fri, 10 Dec 2004, Jeffrey I. Schiller wrote:
On Fri, Dec 10, 2004 at 12:26:59PM -0500, Rich Kulawiec wrote:
One thing that's not clear is whether or not Verizon caches any of this information.
It appears that they do some amount of caching.
-Jeff
It does not appear that they are caching it, here is a sample from my log file:
Dec 6 19:18:15 white sm-mta[25976]: iB70IF6n025976: <lopqp@abc.net>... User unknown Dec 6 19:18:15 white sm-mta[25977]: iB70IF6n025977: <lopqp@abc.net>... User unknown Dec 6 19:18:16 white sm-mta[25976]: iB70IF6n025976: from=<>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182] Dec 6 19:18:16 white sm-mta[25977]: iB70IF6n025977: from=<>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68] Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: <lopqp@abc.net>... User unknown Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: <lopqp@abc.net>... User unknown Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: lost input channel from sc006pub.verizon.net [206.46.170.182] to MTA after rcpt Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: from=<antispam245967@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182] Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: lost input channel from sc019pub.verizon.net [206.46.170.68] to MTA after rcpt Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: from=<antispam830748@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68]
What happens when verizon tries to send email to somebody who does the same type of check, does this not create an infinite loop?
Not if Verizon treats the antispam[0-9]+ mailboxes in a special manner and answers without a check. And they have to answer that the box exists or things are gonna _really_ break. Doing a quick test using the last antispam[0-9]+ address in my SMTP logs, I got all 250 responses without a more recent call back. -- Crist J. Clark crist.clark@globalstar.com Globalstar Communications (408) 933-4387
On Fri, Dec 10, 2004 at 06:03:11PM -0500, Krzysztof Adamski wrote:
It does not appear that they are caching it, here is a sample from my log file: ...
Well when I tested it (3 hours ago) I connected to them manually while watching my incoming milter log. Indeed they visited immediate and verified my address (with an rcpt to command) and when that succeeded, I got the "sender ok" in my telnet session. I then broke the connection and tried again. The second time they did not contact my server and gave me a "sender ok". It is now about three hours later and I just tried again and they did not contact my server. Now this is the "success" case where the mailbox exists. I don't know if they cache the non-existence of a mailbox. -Jeff -- ============================================================================= Jeffrey I. Schiller MIT Network Manager Information Services and Technology Massachusetts Institute of Technology 77 Massachusetts Avenue Room W92-190 Cambridge, MA 02139-4307 617.253.0161 - Voice jis@mit.edu ============================================================================
On Fri, 10 Dec 2004, Rich Kulawiec wrote:
Verizon has put in place an exceedingly stupid "anti-spam" system which does not work, which facilitates DoS attacks, and which provides active assistance to spammers.
The technique discussed is called callback verification and I do not agree that the technique stupid or provides assistance to spammers. I do agree that some of the aspects in how this was implemented by Verizon is not correct and causing problems.
Details:
When an incoming SMTP connection is made to one of Verizon's MX's, they allow it to proceed until the putative sender is specified, i.e. they wait for this part of the SMTP transaction:
MAIL From:<blah@example.com>
Then they pause the incoming connection. And then they start up an outbound SMTP connection from somewhere else on Verizon's network, back to one of the MX's for example.com. They then attempt to verify that "blah" is a valid, deliverable address there. Since most people have long since disabled SMTP VRFY, they actually construct a fake message and attempt delivery with RCPT. If delivery looks like it's going to succeed, they hang up this connection (which is rude)
If I were them I'd do RSET first and then QUIT instead of direct hangup
, and un-pause the incoming one, and allow it to proceed. If delivery looks like it's going to fail, then they also hang up their outbound connection (still rude), un-pause the incoming one, and reject the traffic.
Rejection of traffic is not approriate in this case, the greylisting technique should instead be used and they should reply back with 4xx error code (most likely 421 or 451), this allows for retry of email later on instead of outright denial.
This also means that if the MX they try to connect to is (a) busy (b) down (c) unaware of all the deliverable addresses (d) something else, that they'll refuse the incoming message.
All good reasons explaining why error code should 4xx instead of 5xx as I noted above.
It also means that if the address that's trying to send mail to Verizon is something like "support@thuleracks.com", which is the address that the people at Thule Racks emit support traffic from, but which doesn't accept traffic, that Verizon will deny the message. (Yeah, this isn't very bright on Thule's part, either.)
They are correct in this case. The address entered in RFC2821 MAIL FROM is "Bounces-To" address and it must accept bounced email and as such it must accept incoming emails. If the address does not accept traffic as you indicated should not be used in MAIL FROM and different adddress from local machine should be used. Please read email RFCs and then you'll understand that Envelope MAIL FROM (despite its name) is not an address that indicate source of email, the source of email address is actually "From:" header (or in some cases Sender header - which one to use will depend on your situation)
This is bad for a whole bunch of reasons: two of the more obvious ones are (a) it's a pathetic "anti-spam" measure because ANY forged address ANYWHERE will do I agree that it requires verification that connecting machine has right to use given mail-from address as part of its sending email. See SPF.
But for current situation it does work just fine and causes number of emails with randomly generated emails to be stopped.
, and (b) it doesn't scale.
The scalability depends on implementation. Since we have Verizon implementing it, I'm guessing it scales just fine based on the size of their email network. I'll also note that a lot of email is sent from the same address and as long as they cache results of verification it'll not be necessary to do verification each and every time.
Add to that (c) it abuses RCPT because apparently Verizon is unwilling to use VRFY and to accept the decision of many mail server operators to disable it. Oh, and (d) the behavior of their probe systems is nearly indistinguishable from that of spam-spewing zombies, which don't obey the SMTP protocol either.
Sometimes spammers probe various address before sending email to compile new list of email addresses and they use similar technique, however those are in fact distinguishable because you'll see them probing multiple non-existing address usually in alphabetical order. If email is sent from real working addresses, the callback verification would not look like a spammer probing technique.
[ (b) is also how it lends itself to DoS attacks. Sure, Verizon could rate-limit the rate at which they make outbound connections, but then attacker X could impose significant delay on mail from domain Y just by forging a boatload of messages purporting to be from addresses in Y to Verizon. If Verizon rate-limits their outbound connections, then any real messages from Y will be stuck in the verification queue along with a kazillion forgeries.
As with many techniques trying to address problems of current email infrastructure with temporary solutions, it has weaknesses that made those using vulnerable to specialized DoS attacks. This is nothing new and that is why IETF requires "Security Considerations" section for every draft. Obviously addressing possible security weaknesses is a task for implementor of the technique and as you point out Verizon could use rate-limiting to protect against ongoing attack. I'll also note that rate-limiting is used to protect many kind of attacks (including ip and web DoS) and in all cases it causes some legitimate requests to be delayed along with large number of attacks sources. This is considered to be acceptable temporary situation in order to protect against larger problem.
General principle: any "anti-spam" measure that generates more junk SMTP traffic at a time when we're drowning in it is probably a bad idea. ]
Callback verification if properly implemented will never generate more junk SMTP traffic as DATA part of SMTP transmission never happens. --- William Leibzon, Elan Networks: mailto: william@elan.net Anti-Spam and Email Security Research Worksite: http://www.elan.net/~william/emailsecurity/
on Fri, Dec 10, 2004 at 12:36:12PM -0800, william(at)elan.net wrote:
On Fri, 10 Dec 2004, Rich Kulawiec wrote:
Verizon has put in place an exceedingly stupid "anti-spam" system which does not work, which facilitates DoS attacks, and which provides active assistance to spammers.
The technique discussed is called callback verification and I do not agree that the technique stupid or provides assistance to spammers. I do agree that some of the aspects in how this was implemented by Verizon is not correct and causing problems.
<snip>
But for current situation it does work just fine and causes number of emails with randomly generated emails to be stopped.
Erm. Yeah, it stops them from being delivered to Verizon by shifting half the cost of verification onto the victims.
, and (b) it doesn't scale.
The scalability depends on implementation. Since we have Verizon implementing it, I'm guessing it scales just fine based on the size of their email network.
See above. It doesn't scale when /everyone else/ starts doing it.
Callback verification if properly implemented will never generate more junk SMTP traffic as DATA part of SMTP transmission never happens.
By the time Verizon's callback servers hang up on us they've already generated more junk SMTP traffic, wasted our resources to protect their customers, and aided spammers doing list validation. Your claim that dictionary attacks are always alphabetical is pretty weak and brings nothing to bear on the actual problem - that by rejecting mail from a given address because of (possibly spurious) "verification", they are actually giving the spammers a tool they can use to cull bad addresses from their own lists. The only positive thing I have to say about Verizon's callback scheme is that so far it has not been seen here more than 6 times in a single day in the past two months. So they must be doing some caching, given that at least one of the domains we host has been under joe job outscatter attack for several months running now. -- hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com join us! http://hesketh.com/about/careers/account_manager.html join us!
On Fri, 10 Dec 2004 12:36:12 PST, "william(at)elan.net" said:
They are correct in this case. The address entered in RFC2821 MAIL FROM is "Bounces-To" address and it must accept bounced email and as such it must accept incoming emails. If the address does not accept traffic as you indicated should not be used in MAIL FROM and different adddress from local machine should be used. Please read email RFCs and then you'll
Yes - *you* know that the address you put in the MAIL FROM: should be one that you know is willing to accept any bounces you may be about to generate, and *I* know that. The problem is that the spammer (I include A/V companies that still generate rejection notices) knows that as well - and doesn't feel the same need to follow the rules and be a good netizen. Yes, it's a royal pain trying to follow a protocol when you know up front that there's a 70% to 90% chance that the other person is intentionally failing to follow it....
Reply (*long* reply) being sent off-list. If anyone else wants to see it, rattle my cage. ---Rsk
On Friday 10 Dec 2004 5:26 pm, Rich Kulawiec wrote:
When an incoming SMTP connection is made to one of Verizon's MX's, they allow it to proceed until the putative sender is specified, i.e. they wait for this part of the SMTP transaction:
I don't think this is the issue, as we aren't getting that far in the transaction. Inspection suggests that the anycast announcements in the UK were pointing to a server that wasn't accepting email. I believe here the problem is using anycast, and not providing a backup system not using anycast. The previous case I'm aware of was when bits of the NE USA lost ".org" because they only had anycast DNS servers (and still do AFAIK), and the announcement messed up. Whilst I plead ignorant of the technical details of anycast, strikes me that it is clearly more complex, and thus more prone to failure, and these failures are potentially less obvious.
On Mon, 13 Dec 2004 08:50:20 +0000, Simon Waters <simonw@zynet.net> wrote:
Inspection suggests that the anycast announcements in the UK were pointing to a server that wasn't accepting email.
First I've heard of Verizon using anycast Or do you mean anycast lookups of .net were broken from your part of the internet? --srs -- Suresh Ramasubramanian (ops.lists@gmail.com)
On Mon, 13 Dec 2004, Simon Waters wrote:
Inspection suggests that the anycast announcements in the UK were pointing to a server that wasn't accepting email.
I believe here the problem is using anycast, and not providing a backup system not using anycast. The previous case I'm aware of was when bits of the NE USA lost ".org" because they only had anycast DNS servers (and still do AFAIK), and the announcement messed up.
Whilst I plead ignorant of the technical details of anycast, strikes me that it is clearly more complex, and thus more prone to failure, and these failures are potentially less obvious.
(for anybody reading this who doesn't know, anycast is multiple servers in multiple locations announcing routes and accepting connections to the same IP address). Off the top of my head, there are two major forms of DNS failures: - DNS servers stop responding - DNS servers respond with incorrect answers. Both of these are problems that can affect Anycast systems, but neither is unique to Anycast. In the first case, the DNS protocol has its own failover mechanism, and good anycast implementations also provide a failover mechanism. If a DNS server doesn't get a response from the first IP address it sends a DNS query to, it will send its query to another IP address listed as a DNS server for the domain it's trying to query. This process gets repeated until a working DNS server is found, or until it runs out of IP addresses of DNS servers to try. Anycast adds a second failover mechanism, whereby if a DNS server knows it's stopped responding, it can withdraw the announcement of its IP address, and routing protocols will redirect queries that it would have rejected to a different (hopefully working) server. Since the anycast routing protocol based failover mechanism can fail (for example, if a server stops answering queries but doesn't withdraw its routing announcements), it is important for there to be multiple IP addresses to query when looking for information on a domain. This allows DNS's internal failover mechanism to be used. However, this doesn't mean that the other listed DNS server IP addresses for a domain shouldn't also be addresses of anycast networks; they just shouldn't be part of the same anycast network. If .org has had failures (I don't know whether it has, and it's not an argument I want to wander into the middle of), it may be instructive to look at the differences between .org's anycast setup and that of the root servers. .org has two IP addresses listed, which I believe are for two different anycast clouds. That means that to get .org responses from any given location, at least one of two DNS servers must be working properly. There are thirteen root server IP addresses, most of which are at least somewhat anycasted. So, to get a response from the roots from any given location, at least one of thirteen servers needs to be doing the right thing. Again, that's not an issue of anycast vs. non-anycast. It's an issue of whether n+1 redundancy works as well as n+12 redundancy. The case being talked about here was the other sort of DNS failure -- a DNS server responding with incorrect information. That's a problem that becomes harder to notice as the number of DNS servers for a domain increases, but has nothing to do with whether those servers are in a anycast configuration. Indeed, Mr. Waters' suggested non-anycast backup would just add yet another server that could potentially get out of sync. Instead, this is the standard monitoring issue for big networks -- when a network gets big enough that failures won't be immediately obvious to a network's operator, there needs to be a monitoring system to detect failures. In the DNS case, the monitoring system should probably be trying DNS queries against all the servers, and making sure it gets consistent results. -Steve
On 13 Dec 2004, at 15:27, Steve Gibbard wrote:
On Mon, 13 Dec 2004, Simon Waters wrote:
Inspection suggests that the anycast announcements in the UK were pointing to a server that wasn't accepting email.
I believe here the problem is using anycast, and not providing a backup system not using anycast. The previous case I'm aware of was when bits of the NE USA lost ".org" because they only had anycast DNS servers (and still do AFAIK), and the announcement messed up.
Whilst I plead ignorant of the technical details of anycast, strikes me that it is clearly more complex, and thus more prone to failure, and these failures are potentially less obvious.
(for anybody reading this who doesn't know, anycast is multiple servers in multiple locations announcing routes and accepting connections to the same IP address).
Distribution of a service (whether by anycast or by some other means) is bound to introduce complexity over that incurred by a single instance of a service running in just one place. In some cases, the cost of that complexity is offset by reduced costs (or risk) elsewhere, and anycast makes sense. For a discussion of some of the issues surrounding service distribution using anycast, see: http://www.ietf.org/internet-drafts/draft-kurtis-anycast-bcp-00.txt Flames and projectiles relating to that draft would be very gratefully received (either directly or on the GROW list, but probably not on NANOG). Joe
On Mon, 13 Dec 2004, Simon Waters wrote:
I believe here the problem is using anycast, and not providing a backup system not using anycast. The previous case I'm aware of was when bits of the NE USA lost ".org" because they only had anycast DNS servers
If it's the same incident I pointed out here, prompting denial and finger pointing from the folks in charge, much of the eastern half of the US did. (I'm in Atlanta, GA, with seven backbone providers at my disposal, and all but one of those were affected.)
Whilst I plead ignorant of the technical details of anycast, strikes me that it is clearly more complex, and thus more prone to failure, and these failures are potentially less obvious.
That's definitely true, though it can be used successfully -- if there's a very reliable kill-switch to withdraw the advertisement in a moment, or some kind of fallback mechanism in place to handle gross failures. -- -- Todd Vierling <tv@duh.org> <tv@pobox.com>
On 14.12 09:39, Todd Vierling wrote:
That's definitely true, though it can be used successfully -- if there's a very reliable kill-switch to withdraw the advertisement in a moment, or some kind of fallback mechanism in place to handle gross failures.
Using this as the *only* remedy for unavailability of an anycast instance is insufficient given the speed at which bad news travels in BGP. You want to have the service available at multiple addresses with each of those engineered as differently as possible. Daniel
On 16-dec-04, at 12:52, Daniel Karrenberg wrote:
That's definitely true, though it can be used successfully -- if there's a very reliable kill-switch to withdraw the advertisement in a moment, or some kind of fallback mechanism in place to handle gross failures.
Using this as the *only* remedy for unavailability of an anycast instance is insufficient given the speed at which bad news travels in BGP. You want to have the service available at multiple addresses with each of those engineered as differently as possible.
And that's exactly why UltraDNS' treatment of .org is evil. I really don't understand why people with .org domains aren't complaining louder about this. There are also other risks associated with anycast. See discussions on the IETF list earlier this year: about anycast + per packet load balancing ("[dnsop] Re: Root Anycast (fwd)", early october) and about root anycast ("13 Root Server Limitation", may).
On Thu, 16 Dec 2004, Iljitsch van Beijnum wrote: And that's exactly why UltraDNS' treatment of .org is evil. I really don't understand why people with .org domains aren't complaining louder about this. Instead of re-starting this particular perennial thread, can we please just abbreviate it to an URL such as ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-kook-anycast-is-evil-01.txt and be done with it? Look. Some folks think that $technology is a good solution for $application. Some don't. The great thing about teh internat is that differing solutions to common problems are embraced. Better solutions reap their rewards, and generally survive. I wonder how many folks perpetually arguing this point have ever actually implemented anycasted DNS service? In any case, I cry uncle. Can we just agree to disagree? matt ghali --matt@snark.net------------------------------------------<darwin>< The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
On 16-dec-04, at 19:33, just me wrote:
The great thing about teh internat is that differing solutions to common problems are embraced.
Better solutions reap their rewards, and generally survive.
I wonder how many folks perpetually arguing this point have ever actually implemented anycasted DNS service?
In any case, I cry uncle. Can we just agree to disagree?
No, we can't. Reasonable people can and disagree on how much anycasting is exactly right. This is somewhat important, as there is a tradeoff between normal case performance and the likeliness of certain exceptional failure modes. However, having two IP addresses that are authoritative for a TLD and then heavily anycasting those is just plain wrong because these failure modes aren't very exceptional any more this way. This would be easily fixed by adding more authoritative addresses and making their anycasting different from the other addresses. Having just two addresses is the main problem, the fact that they're also anycast just makes it even worse under certain circumstances.
Steve Gibbard wrote:
On Thu, 16 Dec 2004, Iljitsch van Beijnum wrote:
Having just two addresses is the main problem, the fact that they're also anycast just makes it even worse under certain circumstances.
How does anycast make it worse?
If both anycast routes converges to the same broken pod somehow(damping?). And troublshooting that when it only happens in ASXXXX sounds like it would be a bit more difficult.
-Steve
> If both anycast routes converges to the same broken pod somehow(damping?). > And troublshooting that when it only happens in ASXXXX sounds like it > would be a bit more difficult. That's not an anycast problem, that's just a misconfiguration. -Bill
On Thu, Dec 16, 2004 at 01:43:25PM -0800, Bill Woodcock wrote:
> If both anycast routes converges to the same broken pod somehow(damping?). > And troublshooting that when it only happens in ASXXXX sounds like it > would be a bit more difficult.
That's not an anycast problem, that's just a misconfiguration.
i've been wondering, since most people aren't using a 25xx class router for bgp anymore, and the forwarding planes are able to cope more when 'bad things(tm)' happen, what the value of dampening is these days. ie: does dampening cause more problems than it tries to solve/avoid these days. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
i've been wondering, since most people aren't using a 25xx class router for bgp anymore, and the forwarding planes are able to cope more when 'bad things(tm)' happen, what the value of dampening is these days.
ie: does dampening cause more problems than it tries to solve/avoid these days.
- jared
I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it. I don't know about the rest of the network operators, but the threat of ever being dampened makes me much more careful with my network in general. My general practice is that if a transit line has problems, it's BGP session is put in an admin down state ASAP until the problems can be isolated and corrected. It is my humble little opinion that the threat of being damped for some period of time is a great hammer to hold over network operator's heads. I think it's a 'good thing(tm)' for automatic penalties to go in to effect for those who are irresponsible with the global routing table. Even if that means that though no fault of my own, one of my transit lines flaps, and gets me somewhat damped for an hour. -Jerry
On 17-dec-04, at 0:21, Jerry Pasker wrote:
ie: does dampening cause more problems than it tries to solve/avoid these days.
I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it.
I generally don't use dampening in most setups, and continuous churning is rare these days, as far as I can tell. I seem to remember that it was mostly caused by bad implementations in the days that it was a big issue. The trouble with dampening is that it only works on stuff that happens beyond the routers your AS talks to. When your neighbors or your own stuff flap you don't get to dampen that. So I guess it's still useful for large networks that have a significant number of views on the same stuff, but it's not really worth the trouble for smaller networks. One reason to be careful with dampening is that flaps can be multiplied. (Connect to routeviews and see the different flap counts under different peers for the same flap at your end to observe this.)
On Fri, Dec 17, 2004 at 12:42:21AM +0100, Iljitsch van Beijnum wrote:
On 17-dec-04, at 0:21, Jerry Pasker wrote:
ie: does dampening cause more problems than it tries to solve/avoid these days.
I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it.
One reason to be careful with dampening is that flaps can be multiplied. (Connect to routeviews and see the different flap counts under different peers for the same flap at your end to observe this.)
There have been numerous people who have spoken and released research on this topic. I think with the "better" routing code out there these days, that most people can quickly handle a large number of next-hop changes, etc.. in their hw/sw that disabling dampening would allow the networks to reconverge fairly quickly without (much) trouble. (going to respond to the streaming video/audio/whatnot issue seperately). - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Thu, Dec 16, 2004 at 11:43:12PM -0500, Jared Mauch wrote:
On Fri, Dec 17, 2004 at 12:42:21AM +0100, Iljitsch van Beijnum wrote:
On 17-dec-04, at 0:21, Jerry Pasker wrote:
ie: does dampening cause more problems than it tries to solve/avoid these days.
I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it.
One reason to be careful with dampening is that flaps can be multiplied. (Connect to routeviews and see the different flap counts under different peers for the same flap at your end to observe this.)
There have been numerous people who have spoken and released research on this topic.
I think with the "better" routing code out there these days, that most people can quickly handle a large number of next-hop changes, etc.. in their hw/sw that disabling dampening would allow the networks to reconverge fairly quickly without (much) trouble. (going to respond to the streaming video/audio/whatnot issue seperately).
oops, i thought their reply was cc:ed to the list, but i guess not. i'm going to summarize a private reply i got: a streaming video provider saw that when dampening was enabled, it reduced the number of complaints related to rebuffering and other similar issues by 15%. I think that having a more stable/reliable network switchover to a forwarding path should happen at a more moderate rate.. Obviously, if a prefix is flapping badly via one connection, it should be less prefered, but if it's the only path to that prefix, what harm is caused by reinstalling it fairly quickly? (eg: after 60 seconds of stability instead of a lot longer). This would possibly create a few transient cases of lack of reachability that would be a bit harder to diagnose if max dampening timers were a lot lower and a route was still oscilating.. In the case of lack of reachability to an anycast prefix (i know that .org and ultradns were being picked on, so i'll use them as an example.. sorry Rodney :) ) having the shorter times would increase availability. Obviously the ideal case is that you would exempt these prefixes from dampening, but creating such policies may actually be more work for the router in the long term than just disabling dampening totally. you might see more transient cpu usage as the network goes flappity-flap, but the cost of not evaluating a more complex dampening policy might see some savings as well. (considering the cpu time spent to evaluate it, allocate memory to store state, count flaps, etc..) - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Jerry,
i've been wondering, since most people aren't using a 25xx class router for bgp anymore, and the forwarding planes are able to cope more when 'bad things(tm)' happen, what the value of dampening is these days.
ie: does dampening cause more problems than it tries to solve/avoid these days.
- jared
I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it.
another point to consider is the number of affected routers. Yakov.
On Thu, 16 Dec 2004 10:33:27 PST, just me said:
and be done with it? Look. Some folks think that $technology is a good solution for $application. Some don't. The great thing about teh internat is that differing solutions to common problems are embraced.
Better solutions reap their rewards, and generally survive.
So the competing .org provider deploys their better solution and survives, how, exactly?
On Thu, 16 Dec 2004 Valdis.Kletnieks@vt.edu wrote: On Thu, 16 Dec 2004 10:33:27 PST, just me said:
and be done with it? Look. Some folks think that $technology is a good solution for $application. Some don't. The great thing about teh internat is that differing solutions to common problems are embraced.
Better solutions reap their rewards, and generally survive.
So the competing .org provider deploys their better solution and survives, how, exactly? Are there not a variety of other registries? matt ghali --matt@snark.net------------------------------------------<darwin>< The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
On Thu, 16 Dec 2004 Valdis.Kletnieks@vt.edu wrote: On Thu, 16 Dec 2004 12:24:56 PST, just me said:
So the competing .org provider deploys their better solution and survives, how, exactly?
Are there not a variety of other registries?
It's not a registry problem. % dig org. ns and ponder all the competition. is org the sole delegation from . --matt@snark.net------------------------------------------<darwin>< The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
On Thu, 16 Dec 2004 12:35:09 PST, just me said:
is org the sole delegation from .
If you're trying to register in .org, yes. If you want to claim "but the organization looking to register under .org can go register under .com or .net or .biz", ask yourself why we bother having TLD's at all? Why don't we all just register our-name.%{3randchars} and be done with it? Or just register our-name. and be done with it?
Thank you for mutating this into yet another interminable topic. matt ghali On Thu, 16 Dec 2004 Valdis.Kletnieks@vt.edu wrote: On Thu, 16 Dec 2004 12:35:09 PST, just me said:
is org the sole delegation from .
If you're trying to register in .org, yes. If you want to claim "but the organization looking to register under .org can go register under .com or .net or .biz", ask yourself why we bother having TLD's at all? Why don't we all just register our-name.%{3randchars} and be done with it? Or just register our-name. and be done with it? --matt@snark.net------------------------------------------<darwin>< The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
Or just register our-name. and be done with it?
Not so farfetched http://www.2idi.com Seems to me that DNS already has n+12 redundancy available. But the world is big enough to believe that perhaps 12 extra servers is not enough for everyone everywhere and that it may not be a trivial problem to figure out the right placement of those 12 servers within the topology. So, anycast provides the ability to move from n+12 to n+x. But we can not only increase the number of extra servers, we can also move them around trying to find the right number and the right places, without impacting any DNS configuration. Seems to me that anycast is a worthwhile thing and certainly justifies moving from n+12 to n+11 or n+10 in the DNS. But anycast should not be considered a replacement for the tried and tested DNS protocol redundancy. Maybe in 5 years or so, but not today. --Michael Dillon
Aside from the general discussion I seem to have provoked - most of which may not be relevant to my problem anyway :( Has anyone any idea if @verizon.net email accounts will accept email from the UK ever again? Enquiring postmasters want to know. Still seeing intermittent issues with hotmail and BTConnect, but at least email is flowing to them most of the time.
participants (27)
-
abuse＠cabal.org.uk -
Bill Woodcock -
Christopher X. Candreva -
Crist Clark -
Daniel Karrenberg -
Iljitsch van Beijnum -
Jared Mauch -
Jeffrey I. Schiller -
Jerry Pasker -
Joe Abley -
Joe Maimon -
just me -
Krzysztof Adamski -
Michael Loftis -
Michael.Dillon＠radianz.com -
Paul G -
Paul Trebilco -
Rich Kulawiec -
Roy -
Simon Waters -
Steve Gibbard -
Steven Champeon -
Suresh Ramasubramanian -
Todd Vierling -
Valdis.Kletnieks＠vt.edu -
william(at)elan.net -
Yakov Rekhter