Verizon FiOS outbound mail TLS problem - Superpages people here?
Anyone on the list who does outbound delivery for Verizon (which I think is actually Superpages)? A client has smart-hosted outbounds to *one* of his customers bouncing suddenly with Deferred: 403 4.7.0 TLS handshake failed. *My* inclination is to think that a cert expired somewhere, but his non-tech contact there tells him that the tech people think things are ok. I'm trying to get a mailer log fragment from them. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
I have no relation, but as a mail server operator I can say that I wouldn't be surprised if this is actually a TLS version mismatch or intolerance problem. I would suggest ensuring that both ends support TLS 1.0, 1.1, and 1.2 and use version tolerant TLS implementations. Next on the short list would be not having compatible cyphers between the two servers. Either way, since the error was a 403 error, the expected behavior would be to queue and retry in plain text; Sounds like a broken MTA implementation or misconfiguration if the sending servers do not revert to plain text. --Blake Jay Ashworth wrote on 6/4/2015 11:15 AM:
Anyone on the list who does outbound delivery for Verizon (which I think is actually Superpages)? A client has smart-hosted outbounds to *one* of his customers bouncing suddenly with
Deferred: 403 4.7.0 TLS handshake failed.
*My* inclination is to think that a cert expired somewhere, but his non-tech contact there tells him that the tech people think things are ok.
I'm trying to get a mailer log fragment from them.
Cheers, -- jra
We had a similar issue around November last year where an upgrade on our PostFix MTA to a current version of OpenSSL, which has Mandatory TLS enabled for certain recipient domains, suddenly started generating the same errors with just one recipient domain. We eventually figured out that the problem was they were running an outdated version of the AsyncOS on their Cisco IronPorts. Firmware versions prior to 8.02 had several problems with TLS and one of them was an inability to interoperate with senders who used a newer version of OpenSSL. Their IronPort logs in fact showed a TLS connection was established when it wasn't. (We had switched them to Opportunistic TLS to be able to send emails but their logs still showed TLS while a PCAP showed clear text SMTP.) As soon as that company updated their IronPorts to a v8.5 variant the problem went away. They would not tell us what version they used to run but did confirm it was prior to v8.02. Interestingly, www.checktls.com said they were OK. The admins at Check TLS confirmed that, at that time (the end of 2014), they were running a version of OpenSSL on their website that was still compatible with the older AsyncOS version. FWIW, Ray
Date: Thu, 4 Jun 2015 11:46:35 -0500 From: blake@ispn.net To: nanog@nanog.org Subject: Re: Verizon FiOS outbound mail TLS problem - Superpages people here?
I have no relation, but as a mail server operator I can say that I wouldn't be surprised if this is actually a TLS version mismatch or intolerance problem. I would suggest ensuring that both ends support TLS 1.0, 1.1, and 1.2 and use version tolerant TLS implementations. Next on the short list would be not having compatible cyphers between the two servers.
Either way, since the error was a 403 error, the expected behavior would be to queue and retry in plain text; Sounds like a broken MTA implementation or misconfiguration if the sending servers do not revert to plain text.
--Blake
Jay Ashworth wrote on 6/4/2015 11:15 AM:
Anyone on the list who does outbound delivery for Verizon (which I think is actually Superpages)? A client has smart-hosted outbounds to *one* of his customers bouncing suddenly with
Deferred: 403 4.7.0 TLS handshake failed.
*My* inclination is to think that a cert expired somewhere, but his non-tech contact there tells him that the tech people think things are ok.
I'm trying to get a mailer log fragment from them.
Cheers, -- jra
Oh, and the way we narrowed it down was somewhat oblique. Because their logs said a TLS connection was established we had a hard time convincing them it wasn't. They were convinced it was us who was broke. We had to send them a PCAP and then they ran one and got the same results. We were communicating via their IronPort "secure email" system and I noticed that the Cisco copyright notice on their messages was from 2012. That put me on the path to look at the Cisco release notes. Once I pointed out that they seemed to be a bit behind and there were fixes in later versions, the conversation went in a different direction. :-)
From: sixsigma44@hotmail.com To: blake@ispn.net; nanog@nanog.org Subject: RE: Verizon FiOS outbound mail TLS problem - Superpages people here? Date: Sat, 6 Jun 2015 19:13:38 -0400
We had a similar issue around November last year where an upgrade on our PostFix MTA to a current version of OpenSSL, which has Mandatory TLS enabled for certain recipient domains, suddenly started generating the same errors with just one recipient domain.
We eventually figured out that the problem was they were running an outdated version of the AsyncOS on their Cisco IronPorts. Firmware versions prior to 8.02 had several problems with TLS and one of them was an inability to interoperate with senders who used a newer version of OpenSSL. Their IronPort logs in fact showed a TLS connection was established when it wasn't. (We had switched them to Opportunistic TLS to be able to send emails but their logs still showed TLS while a PCAP showed clear text SMTP.)
As soon as that company updated their IronPorts to a v8.5 variant the problem went away. They would not tell us what version they used to run but did confirm it was prior to v8.02.
Interestingly, www.checktls.com said they were OK. The admins at Check TLS confirmed that, at that time (the end of 2014), they were running a version of OpenSSL on their website that was still compatible with the older AsyncOS version.
FWIW,
Ray
Date: Thu, 4 Jun 2015 11:46:35 -0500 From: blake@ispn.net To: nanog@nanog.org Subject: Re: Verizon FiOS outbound mail TLS problem - Superpages people here?
I have no relation, but as a mail server operator I can say that I wouldn't be surprised if this is actually a TLS version mismatch or intolerance problem. I would suggest ensuring that both ends support TLS 1.0, 1.1, and 1.2 and use version tolerant TLS implementations. Next on the short list would be not having compatible cyphers between the two servers.
Either way, since the error was a 403 error, the expected behavior would be to queue and retry in plain text; Sounds like a broken MTA implementation or misconfiguration if the sending servers do not revert to plain text.
--Blake
Jay Ashworth wrote on 6/4/2015 11:15 AM:
Anyone on the list who does outbound delivery for Verizon (which I think is actually Superpages)? A client has smart-hosted outbounds to *one* of his customers bouncing suddenly with
Deferred: 403 4.7.0 TLS handshake failed.
*My* inclination is to think that a cert expired somewhere, but his non-tech contact there tells him that the tech people think things are ok.
I'm trying to get a mailer log fragment from them.
Cheers, -- jra
participants (3)
-
Blake Hudson -
Jay Ashworth -
Ray