Gmail (thus Nanog) rejecting ipv6 email
I've been seeing a long thread about why ipv6 adoption isn't there yet. This is half a "paging someone with clue" post and half a "...really, guys?" Picard-facepalm post. I just (earlier this week) had to disable ipv6 outbound on one of $dayjob's MX servers, because Gmail, who hosts nanog.org, was rejecting our mail due to "our domain's very low reputation". (In this parlance, "Very Low" is an actual indicative metric.) Dayjob is the people who make BIND and run a root DNS server. Totally disreputable, I'm sure. I don't see anything indicating this in our postmaster tools. I am certain this action is happening completely transparently and invisibly to NANOG, unless others have complained. Whatever UI google gives them to manage their domain will not show this. There are no logs they can grep. I'm told that "gmail's filters for ipv6 are way tighter than ipv4" but that's from a non-canonical source. If this is the case, it does very little to further ipv6 adoption, that's for sure. I've posted over on mailop, and was given a contact (Brandon), but haven't heard back. Gmail's a black box. I've reached out to a few other people, but if anyone here can loan a bat-phone, please let me know. I'm loathe to randomly re-enable ipv6 without contact from someone saying why this happened, and how it's been fixed. -Dan (Who actually operates my own network) -- --------Dan Mahoney-------- Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC FB: fb.com/DanielMahoneyIV LI: linkedin.com/in/gushi Site: http://www.gushi.org ---------------------------
Hi Dan, Hope the rest of the world is treating you decently! There are a lot of bits and bobs that one has to get right for mail to flow, amongst which: - IP -> PTR lookup -> that hostname lookup, and match to IP again (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS) - SPF - DKIM - DMARC - ARC (for mailinglists) - SRS (When forwarding, rewrite the From and resign DKIM, and then ARC-sign that) - Decent TLS - MTA-STS And that list grows and grows... and grows and grows. It is kinda a test if one has actually bothered to configure a setup, and not just are randomly sending an email by just telneting from a random server. Of course the large spam outfits have this fully automated and configured, so that their spam^Wadvertising comes through. A wee little test tells that there are a few improvements to be made at minimum: https://internet.nl/mail/isc.org/ • Not all authenticity marks against email phishing (DMARC, DKIM and SPF) • Failed :Mail server connection not or insufficiently secured (STARTTLS and DANE) Greets, Jeroen (who also runs his own full net... and had jeroen@isc for a few years... ;) )
On 4/2/22 3:23 PM, Jeroen Massar via NANOG wrote:
Hi Dan,
Hope the rest of the world is treating you decently!
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS) - SPF - DKIM - DMARC - ARC (for mailinglists)
Seriously spend zero time on ARC. It doesn't work as advertised and is basically snake oil. Mike
On 3 Apr 2022, at 00:29, Michael Thomas <mike@mtcc.com> wrote:
On 4/2/22 3:23 PM, Jeroen Massar via NANOG wrote:
Hi Dan,
Hope the rest of the world is treating you decently!
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS) - SPF - DKIM - DMARC - ARC (for mailinglists)
Seriously spend zero time on ARC. It doesn't work as advertised... [snip, see below]
Unless one works at the large ESPs, hard to tell what they really care about and verify. Google at least adds ARC headers in Gmail, and did the editing of RFC8617. MS seems to do something with it: https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/... and https://prodmarc.com/knowledge/authenticated-received-chain/ states: 8<---- Who has adopted ARC? Google has added ARC verification and sealing to their email services (Gmail, G Suite, and Google Groups). The popular Mailing List Manager (MLM) software Sympa incorporated ARC in v6.2.38, and ARC is being incorporated into the next release of the Mailman MLM – ARC configuration directives are already in the online documentation. The commercial MTAs Halon and MailerQ incorporate ARC, and the milters authentication_milter and OpenARC can be used to deploy ARC with the Postfix, Oracle Communications Messaging Server, and Sendmail MTAs. Several open-source libraries and modules are already available for those who need to integrate ARC functions into their systems. ----->8 thus there is at least that for ARC. For one project that sends a rather decent amount of email, adopting DMARC/ARC and @via rewriting made all mail go through (at least all the google reception works), though there might be other factors at work: unless you work in the closed corp and on that project, impossible to know why your mail really gets rejected.
... and is basically snake oil.
Unfortunately it is April 3rd, so two days late, but you are thinking of another acronym: BIMI -- https://bimigroup.org Now, THAT is snakeoil, or well, a scam is more like it: if you can pay and they like you, you get a logo, anybody else is out... marketing companies of the world (and the once earning money for bits ala domains and worse EV SSL certs... rejoice) At least they are 'honest' about the scam: https://bimigroup.org/vmcs-arent-a-golden-ticket-for-bimi-logo-display/ but the big ones support it too .... https://support.google.com/a/answer/10911432?hl=en but https://bimigroup.org/bimi-generator/ BIMI record not found for gmail.com. BIMI record not found for google.com. BIMI record not found for yahoo.com. BIMI record not found for microsoft.com. Interesting as https://bimigroup.org/bimi-infographic/ claims they 'support' it... view only maybe? but from where? At least there is: BIMI record found for bimigroup.org, and is BIMI compliant v=BIMI1; l=https://bimigroup.org/bimi-sq.svg; a= https://bimigroup.org/bimi-sq.svg Oh well, 3rd of April, not the 1st... yet another Internet money printing thing... Greets, Jeroen
On 4/2/22 3:56 PM, Jeroen Massar wrote:
On 3 Apr 2022, at 00:29, Michael Thomas <mike@mtcc.com> wrote:
On 4/2/22 3:23 PM, Jeroen Massar via NANOG wrote:
Hi Dan,
Hope the rest of the world is treating you decently!
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS) - SPF - DKIM - DMARC - ARC (for mailinglists) Seriously spend zero time on ARC. It doesn't work as advertised... [snip, see below] Unless one works at the large ESPs, hard to tell what they really care about and verify.
Google at least adds ARC headers in Gmail, and did the editing of RFC8617.
ARC resolves into a previously unsolved problem: reputation. You could do reputation with plain old DKIM too, so I don't see why changing the name of the header changes anything on the ground. And nobody could give me an answer of why signing previous Authentication-Results is useful for toward that end. It's just more magical thinking. Thank goodness it's an experimental RFC so it can go the way of the dodo. Mike
It appears that Michael Thomas <mike@mtcc.com> said:
Google at least adds ARC headers in Gmail, and did the editing of RFC8617.
ARC resolves into a previously unsolved problem: reputation. ...
No, actually it doesn't, as has been repeatedly explained. ARC addreses the problem that mailing lists do a lousy job of spam filtering, A list that usually sends lovely clean mail sometimes doesn't, since a typical list forwards anything with a subscriber's address on the From line including spam from cleverish spammers who take pairs of from/to addresses from stolen mailboxes. ARC lets the recipient system look back and do what we might call retroactive filtering, using info about messages as they arrived at the previous forwarder. While it would be nice if lists did a better job of spam filtering, they don't, and ARC is a reasonable remedy for that. R's, John
On 4/2/22 6:21 PM, John Levine wrote:
It appears that Michael Thomas <mike@mtcc.com> said:
Google at least adds ARC headers in Gmail, and did the editing of RFC8617. ARC resolves into a previously unsolved problem: reputation. ... No, actually it doesn't, as has been repeatedly explained.
ARC addreses the problem that mailing lists do a lousy job of spam filtering, A list that usually sends lovely clean mail sometimes doesn't, since a typical list forwards anything with a subscriber's address on the From line including spam from cleverish spammers who take pairs of from/to addresses from stolen mailboxes.
ARC lets the recipient system look back and do what we might call retroactive filtering, using info about messages as they arrived at the previous forwarder. While it would be nice if lists did a better job of spam filtering, they don't, and ARC is a reasonable remedy for that.
I'll be eager to see the papers substantiating this. Until then I remain completely skeptical. It's an experimental RFC for a reason. Let's see the data. I'd also like to see a paper substantiating your claim that mailing lists do a bad job of spam filtering. In my experience it is a non-problem. Mike
It appears that Michael Thomas <mike@mtcc.com> said:
ARC lets the recipient system look back and do what we might call retroactive filtering, using info about messages as they arrived at the previous forwarder. While it would be nice if lists did a better job of spam filtering, they don't, and ARC is a reasonable remedy for that.
I'll be eager to see the papers substantiating this. Until then I remain completely skeptical. It's an experimental RFC for a reason. Let's see the data.
I'd also like to see a paper substantiating your claim that mailing lists do a bad job of spam filtering. In my experience it is a non-problem.
People from Google have told me that is the specific reason that they need all the complexity of ARC rather than just whitelisting mailing lists. If you think they're lying, or you know more about their mail stream than they do, not much we can do about that. R's, John
On 4/2/22 8:01 PM, John Levine wrote:
ARC lets the recipient system look back and do what we might call retroactive filtering, using info about messages as they arrived at the previous forwarder. While it would be nice if lists did a better job of spam filtering, they don't, and ARC is a reasonable remedy for that. I'll be eager to see the papers substantiating this. Until then I remain completely skeptical. It's an experimental RFC for a reason. Let's see the data.
I'd also like to see a paper substantiating your claim that mailing lists do a bad job of spam filtering. In my experience it is a non-problem. People from Google have told me that is the specific reason that they need all the complexity of ARC rather than just whitelisting mailing
It appears that Michael Thomas <mike@mtcc.com> said: lists. If you think they're lying, or you know more about their mail stream than they do, not much we can do about that.
Then they should publish it since it's an IETF document it so everybody can evaluate it. Otherwise it's just a private vanity project. I've seen absolutely nothing to conclude it is not. And impugning me about "lying" is an ad hominem and against NANOG's rules. Mike
On Sat, 2 Apr 2022, Michael Thomas wrote:
On 4/2/22 6:21 PM, John Levine wrote:
It appears that Michael Thomas <mike@mtcc.com> said:
I'll be eager to see the papers substantiating this. Until then I remain completely skeptical. It's an experimental RFC for a reason. Let's see the data.
ARC is not mentioned here: https://support.google.com/mail/answer/81126?hl=en But nor are mailing lists/listservs. Most of the guidance on "lists" seems to be related to marketing lists (which I hate way more, but gmail seems to be quite forgiving of), vs discussion lists. Also, the error message we're getting speaks to the reputation of "our domain", not our IP block. Otherwise, I would think v4 mail would bounce as well. Now, if that's caused by our staff posting to *other* mailing lists that do not do ARC, we have zero control over that. If it's being implied that gmail is ranking us (i.e. dkim-signed and spf-compliant mail from Mark Andrews to *this list*) with a "very low" reputation because *our* mailman lists don't presently do arc-sealing, then could someone from google please tell me that canonically? -Dan -- --------Dan Mahoney-------- Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC FB: fb.com/DanielMahoneyIV LI: linkedin.com/in/gushi Site: http://www.gushi.org ---------------------------
It appears that Michael Thomas <mike@mtcc.com> said:
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again - SPF - DKIM - DMARC
Yup. Gmail has made it quite clear that they will not accept v6 mail that isn't SPF or DKIM authenticated. DKIM is more work but works more reliably.
- ARC (for mailinglists)
Seriously spend zero time on ARC. It doesn't work as advertised ...
Please, not this again. ARC does what it does, even if it doesn't do what you might wish it did instead. It's certainly not a magic ticket into an inbox but it is slowly helping undo DMARC mailing list damage. It's not important unless you forward mail like a mailing list does. R's, John
On 4/2/22 6:16 PM, John Levine wrote:
It appears that Michael Thomas <mike@mtcc.com> said:
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again - SPF - DKIM - DMARC Yup. Gmail has made it quite clear that they will not accept v6 mail that isn't SPF or DKIM authenticated. DKIM is more work but works more reliably.
- ARC (for mailinglists) Seriously spend zero time on ARC. It doesn't work as advertised ... Please, not this again. ARC does what it does, even if it doesn't do what you might wish it did instead.
I does what it does which is DKIM. That's it.
It's certainly not a magic ticket into an inbox but it is slowly helping undo DMARC mailing list damage. It's not important unless you forward mail like a mailing list does.
No it doesn't. It requires the previously unsolved problem of reputation which manifestly incapable of being solved. DMARC is not the problem, ancient mailing list technology which came before security requirements is the problem. Mike
On 2 Apr 2022, at 6:23 PM, Jeroen Massar via NANOG <nanog@nanog.org> wrote:
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS) - SPF - DKIM - DMARC - ARC (for mailinglists) - SRS (When forwarding, rewrite the From and resign DKIM, and then ARC-sign that) - Decent TLS - MTA-STS
Jeroen - It is indeed amazing how many protocols we can spin up to address the same underlying problem, time and time again... If anyone can anonymously join the mail-sending club and send some email [until bad reputation precludes such], and achieving bad reputation results has no real-world implications, and a new network persona (e.g. domain name) is always available, then the problem could be considered intractable by initial conditions – and no amount of anti-spam protocols (no matter how brilliantly designed and engineered) should be expected to durably address the problem. (It might, however, be interesting to do a regression analysis on the spam mitigation protocol introduction dates – it’d be interesting to know if the expected number protocols that will need proper setup in 10, 20, 40 years…!) <chuckle> /John Disclaimer(s): my views alone. This email composed of 100% recycled electrons.
On 4/2/22 4:05 PM, John Curran wrote:
On 2 Apr 2022, at 6:23 PM, Jeroen Massar via NANOG <nanog@nanog.org> wrote:
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS) - SPF - DKIM - DMARC - ARC (for mailinglists) - SRS (When forwarding, rewrite the From and resign DKIM, and then ARC-sign that) - Decent TLS - MTA-STS
Jeroen -
It is indeed amazing how many protocols we can spin up to address the same underlying problem, time and time again...
If anyone can anonymously join the mail-sending club and send some email [until bad reputation precludes such], and achieving bad reputation results has no real-world implications, and a new network persona (e.g. domain name) is always available, then the problem could be considered intractable by initial conditions – and no amount of anti-spam protocols (no matter how brilliantly designed and engineered) should be expected to durably address the problem.
(It might, however, be interesting to do a regression analysis on the spam mitigation protocol introduction dates – it’d be interesting to know if the expected number protocols that will need proper setup in 10, 20, 40 years…!)
That's why I wrote this: https://rip-van-webble.blogspot.com/2020/12/are-mailing-lists-toast.html Trust me, it wasn't for lack of trying on my part. Mike
On Sun, 3 Apr 2022, Jeroen Massar wrote:
Hi Dan,
Hope the rest of the world is treating you decently!
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS) - SPF - DKIM - DMARC - ARC (for mailinglists) - SRS (When forwarding, rewrite the From and resign DKIM, and then ARC-sign that) - Decent TLS - MTA-STS
And that list grows and grows... and grows and grows. It is kinda a test if one has actually bothered to configure a setup, and not just are randomly sending an email by just telneting from a random server. Of course the large spam outfits have this fully automated and configured, so that their spam^Wadvertising comes through.
A wee little test tells that there are a few improvements to be made at minimum:
https://internet.nl/mail/isc.org/
• Not all authenticity marks against email phishing (DMARC, DKIM and SPF)
We have SPF, DKIM signing, and a DMARC policy that sets p=none. We're not setting p=reject, considering the number of mailing lists our users are on that are outdated or based on EOL software (including this one which depends on python 2.7, and including our own which have the same problem). It's impossible to know, from the outside, how mailing lists are configured. Mailman3 is...special. That's a rant for another time. We get about an email a week from someone emailing security-officer@ trying to get a bug bounty telling us we should set p=reject. There's an ecosystem for this stuff. I don't think this affects our domain's "reputation".
• Failed :Mail server connection not or insufficiently secured (STARTTLS and DANE)
This has little to do with what ciphers we support outbound, and little to do with our reputation. Unlike HTTPS, the failback to startTLS not working is plain-text. Setting a stricter cipher requirement would result in more mail being delivered in the clear. This is a somewhat broken test. -Dan -- --------Dan Mahoney-------- Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC FB: fb.com/DanielMahoneyIV LI: linkedin.com/in/gushi Site: http://www.gushi.org ---------------------------
I’ve not experienced this problem sending emails via IPv6 to gmail destinations from my personal domain. (delong.com <http://delong.com/>) Likely this email will, in fact, get sent to GMAIL via IPv6. I do have good SPF and DKIM records and signing and a reasonable DMARC policy set up. If ISC doesn’t have that yet, it might be a better alternative than turning off IPv6. If that doesn’t solve it, I can reach out to someone at Google who can likely get the right parties involved. Owen
On Apr 2, 2022, at 15:23, Jeroen Massar via NANOG <nanog@nanog.org> wrote:
Hi Dan,
Hope the rest of the world is treating you decently!
There are a lot of bits and bobs that one has to get right for mail to flow, amongst which:
- IP -> PTR lookup -> that hostname lookup, and match to IP again (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS) - SPF - DKIM - DMARC - ARC (for mailinglists) - SRS (When forwarding, rewrite the From and resign DKIM, and then ARC-sign that) - Decent TLS - MTA-STS
And that list grows and grows... and grows and grows. It is kinda a test if one has actually bothered to configure a setup, and not just are randomly sending an email by just telneting from a random server. Of course the large spam outfits have this fully automated and configured, so that their spam^Wadvertising comes through.
A wee little test tells that there are a few improvements to be made at minimum:
https://internet.nl/mail/isc.org/
• Not all authenticity marks against email phishing (DMARC, DKIM and SPF) • Failed :Mail server connection not or insufficiently secured (STARTTLS and DANE)
Greets, Jeroen (who also runs his own full net... and had jeroen@isc for a few years... ;) )
On 2022-04-03 07:18, Owen DeLong via NANOG wrote:
I’ve not experienced this problem sending emails via IPv6 to gmail destinations from my personal domain.
(delong.com <http://delong.com>)
Likely this email will, in fact, get sent to GMAIL via IPv6.
I do have good SPF and DKIM records and signing and a reasonable DMARC policy set up.
If ISC doesn’t have that yet, it might be a better alternative than turning off IPv6.
If that doesn’t solve it, I can reach out to someone at Google who can likely get the right parties involved.
Owen
I think it has been argued before that having a different email acceptance policy over IPv4 vs IPv6 is essentially a layering violation. I'm sympathetic to that argument. More to the point: *you* could do this and there are a number of other clueful people who can make this work today. And when Google changes their rules (that you'll have to learn about once you hit the next wall), then you adjust. And you keep on doing this whack-a-mole game. Of course there's an argument that say "mom and pop should not run their own mailserver, there are professionals for that!" but at the end of the day what this really serves is deliberate and pre-mediated centralisation, slowly but steadily stamping out small players. Robert
On Apr 4, 2022, at 08:13 , Robert Kisteleki <robert@ripe.net> wrote:
On 2022-04-03 07:18, Owen DeLong via NANOG wrote:
I’ve not experienced this problem sending emails via IPv6 to gmail destinations from my personal domain. (delong.com <http://delong.com>) Likely this email will, in fact, get sent to GMAIL via IPv6. I do have good SPF and DKIM records and signing and a reasonable DMARC policy set up. If ISC doesn’t have that yet, it might be a better alternative than turning off IPv6. If that doesn’t solve it, I can reach out to someone at Google who can likely get the right parties involved. Owen
I think it has been argued before that having a different email acceptance policy over IPv4 vs IPv6 is essentially a layering violation. I'm sympathetic to that argument.
The problem with that argument is that it ignores the fact that IP reputation services are available for IPv4 and impractical for IPv6.
More to the point: *you* could do this and there are a number of other clueful people who can make this work today. And when Google changes their rules (that you'll have to learn about once you hit the next wall), then you adjust. And you keep on doing this whack-a-mole game.
It hasn’t been all that much whack-a-mole. Frankly, I’ve had more difficulty playing whack-a-mole with Apple’s changes in what they require for a CA to be accepted by an iPhone so that I can access my own IMAP server than anything Google has done to IPv6 mail acceptance. Bottom line, if you’re running an MTA, then there is a changing landscape of BCPs that you have to adapt to. Google may be one of the first to get strict about some of those BCPs, they are also likely the first one many sites will trip over due to the high volume of email headed their way and the large user base they have, but there are definitely others that you will also trip over. You can’t run an MTA in the modern internet without this whack-a-mole game and I suspect it will eventually hit v4 just as hard as it currently hits v6 because I think that v4 reputation services will fail to cope with CGNAT in much the same way that they currently can’t cope with IPv6.
Of course there's an argument that say "mom and pop should not run their own mailserver, there are professionals for that!" but at the end of the day what this really serves is deliberate and pre-mediated centralisation, slowly but steadily stamping out small players.
As pop running his own mail server, I don’t buy that first argument at all. However, I will say that if you are going to run an MTA on the greater internet, then you have inherently as part of the social contract, accepted the obligation to run it in accordance with the current form of BCP and the further obligation to keep up with the current definition of current BCP.
Robert
Owen
On Tue, 5 Apr 2022, Owen DeLong via NANOG wrote:
Of course there's an argument that say "mom and pop should not run their own mailserver, there are professionals for that!" but at the end of the day what this really serves is deliberate and pre-mediated centralisation, slowly but steadily stamping out small players.
As pop running his own mail server, I don’t buy that first argument at all. However, I will say that if you are going to run an MTA on the greater internet, then you have inherently as part of the social contract, accepted the obligation to run it in accordance with the current form of BCP and the further obligation to keep up with the current definition of current BCP.
Let's talk about professionals? Even assuming these grep's aren't perfect: Number of gmail from/to messages in our last logs: # bzgrep gmail /var/log/maillog* | grep google.com | wc -l 1514 Number that gmail sent us that we flagged as spam and dropped at SMTP time: # bzgrep gmail /var/log/maillog* | grep google.com | grep -i spamassassin | wc -l 641 Number that gmail users are sending, regularly, en-masse, to dead contacts (mutually exclusive with above): # bzgrep gmail /var/log/maillog* | grep google.com | grep -i "User unknown" | wc -l 785 Number of messages we've sent to gmail that have been rejected: # bzgrep gmail /var/log/maillog* | grep google.com | grep -i 188131 | wc -l 9 Number of reported spams that have made it to actual abuse contacts at google: 0 (they /dev/null abuse@, let me know how that jives with your BCP's.) If you're sending them volumes, they offer you a feedback loop so they can report how spammy you look to them. The inverse does not exist. They have no documented SMTP error code that you can set that sigils to them that a user is spamming and should be rate-limited. What gmail has here is a problem at scale. There are large masses of people writing in bad english signing up for accounts and who will take payment to complete a captcha every single time if required, the same way every robocall you get will connect you with an actual human. And gmail are, largely, "too big to block". -Dan Cranky Sysadmin --
* danm@prime.gushi.org (Dan Mahoney (Gushi)) [Sun 03 Apr 2022, 00:11 CEST]:
I've been seeing a long thread about why ipv6 adoption isn't there yet. This is half a "paging someone with clue" post and half a "...really, guys?" Picard-facepalm post.
I just (earlier this week) had to disable ipv6 outbound on one of $dayjob's MX servers, because Gmail, who hosts nanog.org, was rejecting our mail due to "our domain's very low reputation". (In this parlance, "Very Low" is an actual indicative metric.) Dayjob is the people who make BIND and run a root DNS server. Totally disreputable, I'm sure.
I don't see anything indicating this in our postmaster tools.
I am certain this action is happening completely transparently and invisibly to NANOG, unless others have complained. Whatever UI google gives them to manage their domain will not show this. There are no logs they can grep.
I'm told that "gmail's filters for ipv6 are way tighter than ipv4" but that's from a non-canonical source. If this is the case, it does very little to further ipv6 adoption, that's for sure.
I've posted over on mailop, and was given a contact (Brandon), but haven't heard back. Gmail's a black box. I've reached out to a few other people, but if anyone here can loan a bat-phone, please let me know.
I'm loathe to randomly re-enable ipv6 without contact from someone saying why this happened, and how it's been fixed.
-Dan (Who actually operates my own network)
I also run my own mail server. I had to firewall off Google's MXes for this exact reason: silent and not-so-silent email rejection when offered over IPv6. Every now and then they rotate their IP addresses, which causes mail to get dropped for a while. There is no other conclusion possible than that Gmail is actively anti-email at this point. I'm pretty sure I receive more spam from them than I send to them, despite forwarding all emails for a few family members' domains. -- Niels.
On Sun, 3 Apr 2022, Niels Bakker wrote:
I also run my own mail server. I had to firewall off Google's MXes for this exact reason: silent and not-so-silent email rejection when offered over IPv6.
Every now and then they rotate their IP addresses, which causes mail to get dropped for a while.
There is no other conclusion possible than that Gmail is actively anti-email at this point. I'm pretty sure I receive more spam from them than I send to them, despite forwarding all emails for a few family members' domains.
I too have encountered this. This comes up on mailop periodically. It kind of makes me want to drop entries for the various gmail.com MXes in /etc/hosts, because while postfix gives me a way to override the one domain (say, gmail.com) it's whack-a-mole with the various gmail-hosted-domains. Bind9 has a filter-aaaa feature, but it doesn't quite work this way, easily, and of course it breaks DNSSEC. It's my opinion (not that of my employer, necessarily), that gmail is to email as old-school AOL is to the internet. <beard color="#808080"> And it's september. </beard> -Dan -- --------Dan Mahoney-------- Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC FB: fb.com/DanielMahoneyIV LI: linkedin.com/in/gushi Site: http://www.gushi.org ---------------------------
It appears that Niels Bakker <niels=nanog@bakker.net> said:
I also run my own mail server. I had to firewall off Google's MXes for this exact reason: silent and not-so-silent email rejection when offered over IPv6.
I run my own mail server and have no trouble at all delivering mail to Gmail over IPv6. I do have SPF, DKIM, DNSSEC and DANE on my mail servers. My DMARC policy is p=none. If it matters, the MTA is a heavily hacked version of qmail. While I believe that Gmail rejects some people's mail, every time when I have looked in detail, I have found that their mail authentication isn't working properly. I'd suggest starting there. R's, John
I didn't know anyone still cared? Google has been trying to move away from Internet email for many years now. Just let them. There is no way you can "fix" that problem on your side. If you care about specific recipients, then inform them that Google randomly throws away some of their legitimate email. Send a paper mail or phone them if necessary. That's pretty much all you can do. If those recipients continue to use gmail, then that's their decision and problem. I assume NANOG is informed about this now. Bjørn
It appears that Bjørn Mork <bjorn@mork.no> said:
Google has been trying to move away from Internet email for many years now. Just let them. There is no way you can "fix" that problem on your side.
Don't be silly. Gmail has over a billion users and hosts mail for vast numbers of businesses large and small. I agree that they are stricter than many others at mail authentication but considering how big they are, they do a very good job of doing what the standards say. Way better than Y**o* ot M*****o**. R's, John -- Regards, John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly
i try to keep a list of goog's ipv6 email space and don't deliver to it; rather using ipv4 instead. unfortunately, goog does not cooperate with dnswl.org, so this can not be automated. this is mildly damaging to the ipv6 religion, but i don't let that spoil my coffee. their lack of cooperation with the dns good list means inbound from them gets dropped when one of their outbound smtp senders gets badlisted, which they often do. i do not let that spoil my coffee either. i would not want to work for goog's email service; too much pain. randy
Randy Bush <randy@psg.com> writes:
i try to keep a list of goog's ipv6 email space and don't deliver to it; rather using ipv4 instead. unfortunately, goog does not cooperate with dnswl.org, so this can not be automated.
How about using their SPF records as automation input? Their MXes are inside those blocks right now at least. Bjørn
On a slightly related subject... This DKIM failure surprised me, but at least I verified that many NANOG subscribers have mailservers returning DMARC failure reports ;-) Bjørn Mork <bjorn@mork.no> writes:
Authentication-Results: mx.google.com; dkim=fail header.i=@mork.no header.s=b header.b=NB0BT8Ez; spf=pass (google.com: best guess record for domain of bjorn@miraculix.mork.no designates 2001:41c8:51:8a:feff:ff:fe00:e5 as permitted sender) smtp.mailfrom=bjorn@miraculix.mork.no; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=mork.no Received: from canardo.dyn.mork.no ([IPv6:2a01:799:c9f:8600:0:0:0:1]) (authenticated bits=0) by louie.mork.no (8.15.2/8.15.2) with ESMTPSA id 233IGnGC342047 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 3 Apr 2022 19:16:50 +0100 Received: from miraculix.mork.no ([IPv6:2a01:799:c9f:8602:8cd5:a7b0:d07:d516]) (authenticated bits=0) by canardo.dyn.mork.no (8.15.2/8.15.2) with ESMTPSA id 233IGnKb1147676 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 3 Apr 2022 20:16:49 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mork.no; s=b; t=1649009809; bh=ZByFGHIiZPQYmJjQnCv16CXFZhKG8U3fTayR+Mx3piY=; h=From:To:Cc:Subject:References:Date:Message-ID:From; b=NB0BT8EzJBl2E3jzDaz7QY4C/utMGKFF+HCs8qjQFoHA4JHTD21ZkTk34jp2VOiJ0 pYWHUNXCNaEBK44Hr4U96h5pfXor+dqo0cSuRPTLNnRsoLAQg2kqmQkvylagdeezZc 4p+jQEQv5La2KbjzEIvW6iSGwwe4ltT9hu7h0H8U= Received: (nullmailer pid 389787 invoked by uid 1000); Sun, 03 Apr 2022 18:16:48 -0000 From: =?utf-8?Q?Bj=C3=B8rn_Mork?= <bjorn@mork.no> To: Randy Bush <randy@psg.com> Cc: John Levine <johnl@iecc.com>, "North American Network Operators' Group" <nanog@nanog.org> Subject: Re: Gmail (thus Nanog) rejecting ipv6 email Organization: m References: <875ynqcvsl.fsf@miraculix.mork.no> <20220403164123.4CE413A4B99C@ary.qy> <m28rsmjbu9.wl-randy@psg.com> Date: Sun, 03 Apr 2022 20:16:48 +0200 In-Reply-To: <m28rsmjbu9.wl-randy@psg.com> (Randy Bush's message of "Sun, 03 Apr 2022 10:50:06 -0700") Message-ID: <87v8vqav73.fsf@miraculix.mork.no>
Did a little testing, and it looks like opendkim create a bogus signature if a quoted-string diplay name in a To or Cc headers contains an apostrophe. Not good at all. Bjørn
On 4/3/22 12:12 PM, Bjørn Mork wrote:
On a slightly related subject... This DKIM failure surprised me, but at least I verified that many NANOG subscribers have mailservers returning DMARC failure reports ;-)
Oh wow, you should report that to Murray. Mike
Bjørn Mork <bjorn@mork.no> writes:
Authentication-Results: mx.google.com; dkim=fail header.i=@mork.no header.s=b header.b=NB0BT8Ez; spf=pass (google.com: best guess record for domain of bjorn@miraculix.mork.no designates 2001:41c8:51:8a:feff:ff:fe00:e5 as permitted sender) smtp.mailfrom=bjorn@miraculix.mork.no; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=mork.no Received: from canardo.dyn.mork.no ([IPv6:2a01:799:c9f:8600:0:0:0:1]) (authenticated bits=0) by louie.mork.no (8.15.2/8.15.2) with ESMTPSA id 233IGnGC342047 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 3 Apr 2022 19:16:50 +0100 Received: from miraculix.mork.no ([IPv6:2a01:799:c9f:8602:8cd5:a7b0:d07:d516]) (authenticated bits=0) by canardo.dyn.mork.no (8.15.2/8.15.2) with ESMTPSA id 233IGnKb1147676 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 3 Apr 2022 20:16:49 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mork.no; s=b; t=1649009809; bh=ZByFGHIiZPQYmJjQnCv16CXFZhKG8U3fTayR+Mx3piY=; h=From:To:Cc:Subject:References:Date:Message-ID:From; b=NB0BT8EzJBl2E3jzDaz7QY4C/utMGKFF+HCs8qjQFoHA4JHTD21ZkTk34jp2VOiJ0 pYWHUNXCNaEBK44Hr4U96h5pfXor+dqo0cSuRPTLNnRsoLAQg2kqmQkvylagdeezZc 4p+jQEQv5La2KbjzEIvW6iSGwwe4ltT9hu7h0H8U= Received: (nullmailer pid 389787 invoked by uid 1000); Sun, 03 Apr 2022 18:16:48 -0000 From: =?utf-8?Q?Bj=C3=B8rn_Mork?= <bjorn@mork.no> To: Randy Bush <randy@psg.com> Cc: John Levine <johnl@iecc.com>, "North American Network Operators' Group" <nanog@nanog.org> Subject: Re: Gmail (thus Nanog) rejecting ipv6 email Organization: m References: <875ynqcvsl.fsf@miraculix.mork.no> <20220403164123.4CE413A4B99C@ary.qy> <m28rsmjbu9.wl-randy@psg.com> Date: Sun, 03 Apr 2022 20:16:48 +0200 In-Reply-To: <m28rsmjbu9.wl-randy@psg.com> (Randy Bush's message of "Sun, 03 Apr 2022 10:50:06 -0700") Message-ID: <87v8vqav73.fsf@miraculix.mork.no>
Did a little testing, and it looks like opendkim create a bogus signature if a quoted-string diplay name in a To or Cc headers contains an apostrophe. Not good at all.
Bjørn
It appears that Michael Thomas <mike@mtcc.com> said:
On 4/3/22 12:12 PM, Bjørn Mork wrote:
On a slightly related subject... This DKIM failure surprised me, but at least I verified that many NANOG subscribers have mailservers returning DMARC failure reports ;-)
Oh wow, you should report that to Murray.
It's on Github, so you can open an issue and if you're feeling inspired a fork and a patch. There's currently 67 open issues and 15 pull requests so don't hold your breath. https://github.com/trusteddomainproject/OpenDKIM R's, John
Bjørn Mork <bjorn@mork.no> writes:
Authentication-Results: mx.google.com; dkim=fail header.i=@mork.no header.s=b header.b=NB0BT8Ez; spf=pass (google.com: best guess record for domain of bjorn@miraculix.mork.no designates 2001:41c8:51:8a:feff:ff:fe00:e5 as permitted sender) smtp.mailfrom=bjorn@miraculix.mork.no; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=mork.no Received: from canardo.dyn.mork.no ([IPv6:2a01:799:c9f:8600:0:0:0:1]) (authenticated bits=0) by louie.mork.no (8.15.2/8.15.2) with ESMTPSA id 233IGnGC342047 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 3 Apr 2022 19:16:50 +0100 Received: from miraculix.mork.no ([IPv6:2a01:799:c9f:8602:8cd5:a7b0:d07:d516]) (authenticated bits=0) by canardo.dyn.mork.no (8.15.2/8.15.2) with ESMTPSA id 233IGnKb1147676 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sun, 3 Apr 2022 20:16:49 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mork.no; s=b; t=1649009809; bh=ZByFGHIiZPQYmJjQnCv16CXFZhKG8U3fTayR+Mx3piY=; h=From:To:Cc:Subject:References:Date:Message-ID:From; b=NB0BT8EzJBl2E3jzDaz7QY4C/utMGKFF+HCs8qjQFoHA4JHTD21ZkTk34jp2VOiJ0 pYWHUNXCNaEBK44Hr4U96h5pfXor+dqo0cSuRPTLNnRsoLAQg2kqmQkvylagdeezZc 4p+jQEQv5La2KbjzEIvW6iSGwwe4ltT9hu7h0H8U= Received: (nullmailer pid 389787 invoked by uid 1000); Sun, 03 Apr 2022 18:16:48 -0000 From: =?utf-8?Q?Bj=C3=B8rn_Mork?= <bjorn@mork.no> To: Randy Bush <randy@psg.com> Cc: John Levine <johnl@iecc.com>, "North American Network Operators' Group" <nanog@nanog.org> Subject: Re: Gmail (thus Nanog) rejecting ipv6 email Organization: m References: <875ynqcvsl.fsf@miraculix.mork.no> <20220403164123.4CE413A4B99C@ary.qy> <m28rsmjbu9.wl-randy@psg.com> Date: Sun, 03 Apr 2022 20:16:48 +0200 In-Reply-To: <m28rsmjbu9.wl-randy@psg.com> (Randy Bush's message of "Sun, 03 Apr 2022 10:50:06 -0700") Message-ID: <87v8vqav73.fsf@miraculix.mork.no>
Did a little testing, and it looks like opendkim create a bogus signature if a quoted-string diplay name in a To or Cc headers contains an apostrophe. Not good at all.
"John Levine" <johnl@iecc.com> writes:
It appears that Michael Thomas <mike@mtcc.com> said:
On 4/3/22 12:12 PM, Bjørn Mork wrote:
On a slightly related subject... This DKIM failure surprised me, but at least I verified that many NANOG subscribers have mailservers returning DMARC failure reports ;-)
Oh wow, you should report that to Murray.
It's on Github, so you can open an issue and if you're feeling inspired a fork and a patch. There's currently 67 open issues and 15 pull requests so don't hold your breath.
There is absolutely nothing wrong with opendkim. Sorry for this off-topic noise. opendkim is an excellent tool, which helped me find the real problem with a simple "Diagnostics yes" in the config file. My problem was caused by bad interaction between nullmailer and sendmail. Turns that out nullmailer removes quotes around the display-name unless required, while sendmail adds quotes it consider necessary. The end-result is a Cc header looking exacly like the one I sent. Only problem is that it wasn't that header opendkim got. 1) I submitted this to nullmailer: Cc: John Levine <johnl@iecc.com>, "North American Network Operators' Group" <nanog@nanog.org> 2) nullmailer forwarded this to sendmail: Cc: John Levine <johnl@iecc.com>, North American Network Operators' Group <nanog@nanog.org> 3) opendkim signed the mail using the unquoted Cc header 4) sendmail added quotes and forwarded this: Cc: John Levine <johnl@iecc.com>, "North American Network Operators' Group" <nanog@nanog.org> 5) validation failed since the header signature was based on the unquoted version. The header modifications in transit is the real bug. IMHO neither nullmailer nor sendmail should change the Cc header here. They should rather reject the mail if they don't like the headers. But I can't see any reasons for that. Both the quoted and the unquoted versions are fine according to my understanding of RFC5322. Any hints on how to configure sendmail to avoid this are appreciated. I can always patch nullmailer. But the same problem can be triggerd by any client submitting an unquoted display-name with an apostrophe to sendmail. Possibly also other characters which are allowed in an atom. I do understand why most people just go with gmail... Bjørn
Bjørn Mork <bjorn@mork.no> writes:
Any hints on how to configure sendmail to avoid this are appreciated.
This turned out to be simple. Should have looked first, as usual. The sendmail config problem was Debian specific, caused by dnl # add .' to mustquote chars (and match the binary default) changequote([, ])dnl define([confMUST_QUOTE_CHARS], [.'])dnl changequote(`, ')dnl in /usr/share/sendmail/cf/ostype/debian.m4 Bjørn
On Mon, 4 Apr 2022, Bjørn Mork wrote:
"John Levine" <johnl@iecc.com> writes:
It appears that Michael Thomas <mike@mtcc.com> said:
On 4/3/22 12:12 PM, Bjørn Mork wrote:
On a slightly related subject... This DKIM failure surprised me, but at least I verified that many NANOG subscribers have mailservers returning DMARC failure reports ;-)
Oh wow, you should report that to Murray.
It's on Github, so you can open an issue and if you're feeling inspired a fork and a patch. There's currently 67 open issues and 15 pull requests so don't hold your breath.
There is absolutely nothing wrong with opendkim.
I wouldn't go that far. It definitely needs some love. John's comment about not holding your breath was perhaps a bit cutting, but we're working on it. -Dan -- --------Dan Mahoney-------- Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC FB: fb.com/DanielMahoneyIV LI: linkedin.com/in/gushi Site: http://www.gushi.org ---------------------------
On Apr 3, 2022, at 9:41 AM, John Levine <johnl@iecc.com> wrote:
It appears that Bjørn Mork <bjorn@mork.no> said:
Google has been trying to move away from Internet email for many years now. Just let them. There is no way you can "fix" that problem on your side.
Don't be silly. Gmail has over a billion users and hosts mail for vast numbers of businesses large and small.
I agree that they are stricter than many others at mail authentication but considering how big they are, they do a very good job of doing what the standards say. Way better than Y**o* ot M*****o**.
Accepting mail for delivery, and then either silently dropping it, delaying it for days, or putting mail that in no way resembles spam into a spam folder seems a little worse than “doing what the standards say”. If you’re going to decide, on little or no evidence, that a message is spam or otherwise does not deserve to get delivered, the least you could do is to bounce it so that the sender is aware. No need to generate a bounce mail that could turn into backscatter; just reject the mail during the SMTP exchange. Jim Shankland
On Apr 3, 2022, at 1:40 PM, nanog@shankland.org wrote:
It appears that Bjørn Mork <bjorn@mork.no> said:
Google has been trying to move away from Internet email for many years now. Just let them. There is no way you can "fix" that problem on your side.
Don't be silly. Gmail has over a billion users and hosts mail for vast numbers of businesses large and small.
I agree that they are stricter than many others at mail authentication but considering how big they are, they do a very good job of doing what the standards say. Way better than Y**o* ot M*****o**.
Accepting mail for delivery, and then either silently dropping it, delaying it for days, or putting mail that in no way resembles spam into a spam folder seems a little worse than “doing what the standards say”. If you’re going to decide, on little or no evidence, that a message is spam or otherwise does not deserve to get delivered, the least you could do is to bounce it so that the sender is aware. No need to generate a bounce mail that could turn into backscatter; just reject the mail during the SMTP exchange.
NO FREAKING KIDDING. I’m running into this with clients for whom we do web site work. Mail not being delivered to Gmail accounts. No bounceback, not being delayed, not marked as spam, just black-holed for no discernible reason. Like, clients losing money because sales leads never make it to them. Extremely frustrating. -Andy
Hello, On Sun, Apr 03, 2022 at 07:44:59PM -0500, Andy Ringsmuth wrote:
I’m running into this with clients for whom we do web site work. Mail not being delivered to Gmail accounts. No bounceback, not being delayed, not marked as spam, just black-holed for no discernible reason. Like, clients losing money because sales leads never make it to them.
Over on mailop a Google employee said that gmail never silently discards email, that mail will always be delivered somewhere or else rejected at SMTP time, and that the only exception is that GAFYD customers (and similar) can set up rules themselves to delete email under some circumstances. I too occasionally experience customers saying their invoices and payment reminders were silently discarded by gmail, so I don't know who to believe. As regards the topic at hand, gmail does seem to go through phases of not accepting our email. While they certainly are tighter on IPv6 and absolutely require SPF, DKIM, DMARC there, other than that I haven't seen v6 be any different to v4 and just swallow it. I know others prefer to only deliver to them over v4. Thanks, Andy
On Apr 4, 2022, at 9:39 AM, Andy Smith <andy@strugglers.net> wrote:
On Sun, Apr 03, 2022 at 07:44:59PM -0500, Andy Ringsmuth wrote:
I’m running into this with clients for whom we do web site work. Mail not being delivered to Gmail accounts. No bounceback, not being delayed, not marked as spam, just black-holed for no discernible reason. Like, clients losing money because sales leads never make it to them.
Over on mailop a Google employee said that gmail never silently discards email, that mail will always be delivered somewhere or else rejected at SMTP time, and that the only exception is that GAFYD customers (and similar) can set up rules themselves to delete email under some circumstances.
I’m on mailop as well. Sadly, what that Google employee said is provably false. -Andy
Accepting mail for delivery, and then either silently dropping it, delaying it for days, or putting mail that in no way resembles spam into a spam folder seems a little worse than “doing what the standards say”. If you’re going to decide, on little or no evidence, that a message is spam or otherwise does not deserve to get delivered, the least you could do is to bounce it so that the sender is aware. No need to generate a bounce mail that could turn into backscatter; just reject the mail during the SMTP exchange.
Jim Shankland
I think they have turned some knobs recently (or rather, they continuously do). Yesterday's soft reject (i.e. mail ending up in the spam folder) became a hard reject. I guess it's possible to argue both ways - at least the soft reject could be trained not to categorise real mail as spam. With a hard reject that problem is shifted entirely to the sender. Robert
participants (13)
-
Andy Ringsmuth -
Andy Smith -
Bjørn Mork -
Dan Mahoney (Gushi) -
Jeroen Massar -
John Curran -
John Levine -
Michael Thomas -
nanog＠shankland.org -
Niels Bakker -
Owen DeLong -
Randy Bush -
Robert Kisteleki