What must one do to avoid Gmail's retarded non-spam filtering?
I realize that this is somewhat OT, but I'm sure that others on the list encounter the same issues and that at least some folks might have useful comments. An increasingly large number of our customers are using Gmail or Google Apps and almost all of our OSS/BSS mail is getting spam filtered by Google. Among others, these e-mails include invoices, order confirmations, payment notifications, customer portal logins, and tickets. Almost anything we send to customers on Google ends up in their spam folder. This results in a lot of calls and makes much of our automation pointless, never mind all the lost sales. The problem is compounded by those who use mail clients and do not log in to the webmail at all, since they would never see the contents of the Google spam folder. We have proper A+PTR records on the edge MTAs, proper SPF records for the originating domain, proper Return-Path and other headers, and so on. There isn't anything that I can think of other than the content itself which would be abnormal, and obviously the content is repetitive and can't be changed much. Is there something obvious which we've missed? Aside from the following clearly impractical solutions, what can we do? 1. Asking everyone (including those we don't even know yet) to whitelist all of our addresses, to check their spam folders, and to click on "this is not spam" 2. Providing our own free e-mail service to everyone (including those we don't even know yet) and putting up "don't use Google" ads on all of our customer-facing systems At least this isn't Hotmail where mail is just silently deleted with no NDR after it's accepted by their MTAs. The call volume has been going up instead of down lately and it's gotten to the point where we're sending MTA log extracts to people to prove to them that we really did e-mail them. Would greatly appreciate any advice. Erik
On Tue, Sep 28, 2010 at 4:15 PM, Erik L <erik_list@caneris.com> wrote:
An increasingly large number of our customers are using Gmail or Google Apps and almost all of our OSS/BSS mail is getting spam filtered by Google. Among others, these e-mails include invoices, order confirmations, payment notifications, customer portal logins, and tickets. Almost anything we send to customers on Google ends up in their spam folder.
Erik, Do you send marketing information to your customers from the same IP addresses as you send your invoices? Have you checked the source IPs against the RBLs with, e.g. http://rbls.org/? What about the source IP of the specific machine the originates the invoices et. al? Is the invoicing machine also in SPF? One of the companies I do business with accidentally excluded their billing mailer from the SPF record. Have you sent copies of the various types of documents to a mail server running Spam Assassin in order to see which qualms Spam Assassin flags? Careless mistakes with the headers or the clock can get your message flagged as spam. Even something as simple as ALL CAPS can get you flagged. Have you added your invoicing mailer, the one you can guarantee sends no marketing materials, in to the whitelist at http://www.dnswl.org/? Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Bill, Thanks for the response and excellent ideas. The OSS/BSS box is in our internal network with a RFC1918 address and it relays to outside via another box which sits in the DMZ and which is in SPF records for the domain that appears in From/Return-Path, etc. That DMZ machine does not appear on RBLs nor is it whitelisted anywhere. Google adds headers like these: Received-SPF: pass ... Authentication-Results: mx.google.com; spf=pass ... So the problem is unlikely to be a SPF issue, as mentioned in my first e-mail. We haven't sent any marketing mail from the same IP in months, but I believe that we did previously. We do send out once every few months a network maintenance notice with identical content for all recipients, except their name and To header. Clocks are set correctly everywhere via NTP and Received headers in the messages are >= each of the previous Received headers as you go up. We do use ALL CAPS in some subjects (rare) but more commonly, some customer names and other data in the body does appear in caps. We also have a www.caneris.com link on most e-mails (removing it doesn't change gmail's behaviour). Messages without anything ALL CAPS in them end up in spam as well. I created a new mailbox under a Google Apps domain and sent it one of the typical messages. It went into spam. I also ran the same message through SpamAssassin as per your suggestion and it came out clean. A test message sent directly from the same edge MTA but with a From address in a different domain is not delivered to spam, even though Return-Path there is still an @caneris.com address (but different than our usual one). A test message sent from our corporate mail system (Zimbra box in internal network relaying to the same edge MTA as the one used by the OSS/BSS box) goes to spam. I've received several off-list responses, including those from Google Apps users, asking for sample messages, so will be forwarding those. Ideas: -Message-Id @ "non-existant host" (ones in our internal network) -Received headers showing that the message was relayed, and again from a "non-existant host" Unlikely to be an IP repuation issue given the one message above did go through fine. Erik ----- Original Message ----- From: "William Herrin" <bill@herrin.us> To: "Erik L" <erik_list@caneris.com> Cc: nanog@nanog.org Sent: Tuesday, September 28, 2010 4:44:24 PM Subject: Re: What must one do to avoid Gmail's retarded non-spam filtering? On Tue, Sep 28, 2010 at 4:15 PM, Erik L <erik_list@caneris.com> wrote:
An increasingly large number of our customers are using Gmail or Google Apps and almost all of our OSS/BSS mail is getting spam filtered by Google. Among others, these e-mails include invoices, order confirmations, payment notifications, customer portal logins, and tickets. Almost anything we send to customers on Google ends up in their spam folder.
Erik, Do you send marketing information to your customers from the same IP addresses as you send your invoices? Have you checked the source IPs against the RBLs with, e.g. http://rbls.org/? What about the source IP of the specific machine the originates the invoices et. al? Is the invoicing machine also in SPF? One of the companies I do business with accidentally excluded their billing mailer from the SPF record. Have you sent copies of the various types of documents to a mail server running Spam Assassin in order to see which qualms Spam Assassin flags? Careless mistakes with the headers or the clock can get your message flagged as spam. Even something as simple as ALL CAPS can get you flagged. Have you added your invoicing mailer, the one you can guarantee sends no marketing materials, in to the whitelist at http://www.dnswl.org/? Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Erik L wrote:
Received-SPF: pass ... Authentication-Results: mx.google.com; spf=pass ...
So the problem is unlikely to be a SPF issue, as mentioned in my first e-mail.
http://david.woodhou.se/why-not-spf.html The lack of SPF records should never be the reason to block an email. It's about time SPF is being laid to rest, it's broken, ineffective and a flawed concept. Well, it's effective in legitimising spammers and breaking forwarding for example, but otherwise... -- http://goldmark.org/jeff/stupid-disclaimers/ http://linuxmafia.com/~rick/faq/plural-of-virus.html
I don't believe in SPF, which is why we don't use it to check inbound mail. I do believe in being able to communicate with our customers irrespective of which provider they use, and given that Hotmail in particular is extremely unforgiving with respect to SPF, we have no choice but to publish such records. ----- Original Message ----- From: "Jeroen van Aart" <jeroen@mompl.net> To: "nanog" <nanog@nanog.org> Sent: Wednesday, September 29, 2010 2:11:43 PM Subject: Re: What must one do to avoid Gmail's retarded non-spam filtering? Erik L wrote:
Received-SPF: pass ... Authentication-Results: mx.google.com; spf=pass ...
So the problem is unlikely to be a SPF issue, as mentioned in my first e-mail.
http://david.woodhou.se/why-not-spf.html The lack of SPF records should never be the reason to block an email. It's about time SPF is being laid to rest, it's broken, ineffective and a flawed concept. Well, it's effective in legitimising spammers and breaking forwarding for example, but otherwise... -- http://goldmark.org/jeff/stupid-disclaimers/ http://linuxmafia.com/~rick/faq/plural-of-virus.html
Hi, Have you checked the IronPort reputation scores for your mailserver IPs? Google uses this data as part of it's spam detection method. William On Tue, 2010-09-28 at 16:15 -0400, Erik L wrote:
I realize that this is somewhat OT, but I'm sure that others on the list encounter the same issues and that at least some folks might have useful comments.
An increasingly large number of our customers are using Gmail or Google Apps and almost all of our OSS/BSS mail is getting spam filtered by Google. Among others, these e-mails include invoices, order confirmations, payment notifications, customer portal logins, and tickets. Almost anything we send to customers on Google ends up in their spam folder. This results in a lot of calls and makes much of our automation pointless, never mind all the lost sales.
The problem is compounded by those who use mail clients and do not log in to the webmail at all, since they would never see the contents of the Google spam folder.
We have proper A+PTR records on the edge MTAs, proper SPF records for the originating domain, proper Return-Path and other headers, and so on. There isn't anything that I can think of other than the content itself which would be abnormal, and obviously the content is repetitive and can't be changed much. Is there something obvious which we've missed?
Aside from the following clearly impractical solutions, what can we do? 1. Asking everyone (including those we don't even know yet) to whitelist all of our addresses, to check their spam folders, and to click on "this is not spam" 2. Providing our own free e-mail service to everyone (including those we don't even know yet) and putting up "don't use Google" ads on all of our customer-facing systems
At least this isn't Hotmail where mail is just silently deleted with no NDR after it's accepted by their MTAs.
The call volume has been going up instead of down lately and it's gotten to the point where we're sending MTA log extracts to people to prove to them that we really did e-mail them.
Would greatly appreciate any advice.
Erik
Hi William, I do so for our entire IP space on a regular basis. The edge MTA I mentioned in the reply to Bill shows up as "Neutral" there. Thanks Erik ----- Original Message ----- From: "William Pitcock" <nenolod@systeminplace.net> To: "Erik L" <erik_list@caneris.com> Cc: nanog@nanog.org Sent: Tuesday, September 28, 2010 6:06:49 PM Subject: Re: What must one do to avoid Gmail's retarded non-spam filtering? Hi, Have you checked the IronPort reputation scores for your mailserver IPs? Google uses this data as part of it's spam detection method. William On Tue, 2010-09-28 at 16:15 -0400, Erik L wrote:
I realize that this is somewhat OT, but I'm sure that others on the list encounter the same issues and that at least some folks might have useful comments.
An increasingly large number of our customers are using Gmail or Google Apps and almost all of our OSS/BSS mail is getting spam filtered by Google. Among others, these e-mails include invoices, order confirmations, payment notifications, customer portal logins, and tickets. Almost anything we send to customers on Google ends up in their spam folder. This results in a lot of calls and makes much of our automation pointless, never mind all the lost sales.
The problem is compounded by those who use mail clients and do not log in to the webmail at all, since they would never see the contents of the Google spam folder.
We have proper A+PTR records on the edge MTAs, proper SPF records for the originating domain, proper Return-Path and other headers, and so on. There isn't anything that I can think of other than the content itself which would be abnormal, and obviously the content is repetitive and can't be changed much. Is there something obvious which we've missed?
Aside from the following clearly impractical solutions, what can we do? 1. Asking everyone (including those we don't even know yet) to whitelist all of our addresses, to check their spam folders, and to click on "this is not spam" 2. Providing our own free e-mail service to everyone (including those we don't even know yet) and putting up "don't use Google" ads on all of our customer-facing systems
At least this isn't Hotmail where mail is just silently deleted with no NDR after it's accepted by their MTAs.
The call volume has been going up instead of down lately and it's gotten to the point where we're sending MTA log extracts to people to prove to them that we really did e-mail them.
Would greatly appreciate any advice.
Erik
On 09/28/2010 10:15 PM, Erik L wrote:
I realize that this is somewhat OT, but I'm sure that others on the list encounter the same issues and that at least some folks might have useful comments.
An increasingly large number of our customers are using Gmail or Google Apps and almost all of our OSS/BSS mail is getting spam filtered by Google. Among others, these e-mails include invoices, order confirmations, payment notifications, customer portal logins, and tickets. Almost anything we send to customers on Google ends up in their spam folder. This results in a lot of calls and makes much of our automation pointless, never mind all the lost sales.
The problem is compounded by those who use mail clients and do not log in to the webmail at all, since they would never see the contents of the Google spam folder.
We have proper A+PTR records on the edge MTAs, proper SPF records for the originating domain, proper Return-Path and other headers, and so on. There isn't anything that I can think of other than the content itself which would be abnormal, and obviously the content is repetitive and can't be changed much. Is there something obvious which we've missed?
Have you tried DKIM signing? All email sent from Gmail is DKIM signed, so they probably also support checking it and a valid signature may lower your spam score. Regards, -Lorand Jakab
Aside from the following clearly impractical solutions, what can we do? 1. Asking everyone (including those we don't even know yet) to whitelist all of our addresses, to check their spam folders, and to click on "this is not spam" 2. Providing our own free e-mail service to everyone (including those we don't even know yet) and putting up "don't use Google" ads on all of our customer-facing systems
At least this isn't Hotmail where mail is just silently deleted with no NDR after it's accepted by their MTAs.
The call volume has been going up instead of down lately and it's gotten to the point where we're sending MTA log extracts to people to prove to them that we really did e-mail them.
Would greatly appreciate any advice.
Erik
Have you tried DKIM signing? All email sent from Gmail is DKIM signed, so they probably also support checking it and a valid signature may lower your spam score. DKIM is definitively a must have for gmail.
At least this isn't Hotmail where mail is just silently deleted with no NDR after it's accepted by their MTAs. If Hotmail is doing that to your messages, you're in a quite bad
posture. That's their way to tell you "we don't like what you're sending at all". Did you consider hiring deliverability consultants? I'm afraid there is no magical way to reach inbox without a audit of your process and systems. Benjamin
We have proper A+PTR records on the edge MTAs, proper SPF records for the originating domain, proper Return-Path and other headers, and so on. There isn't anything that I can think of other than the content itself which would be abnormal, and obviously the content is repetitive and can't be changed much. Is there something obvious which we've missed?
What else goes out from that MTA? User forwarded mail? Mailing lists? random customer junk? It is a really good idea to separate your mail streams. These days it means separate IPs for users, transactions, forwards, and such, eventually different DKIM signatures will do the trick. R's, John
Thanks John. This was a common question that was asked off-list. That edge MTA is not used and has never been used by anything/anyone other than us. No customer mail flows or has flowed in or out via it ever. As I mentioned in my follow-up post, the issue at this point is that the domain has been blacklisted. I can send an identical message from the same MTA, changing only the From header, and it will be delivered to Inbox. Only when the From header contains @caneris.com will the message be delivered to spam. Any changes to the MTA IP, content, headers, etc. don't have any effect. Erik ----- Original Message ----- From: "John Levine" <johnl@iecc.com> To: nanog@nanog.org Cc: "erik list" <erik_list@caneris.com> Sent: Wednesday, September 29, 2010 1:58:28 PM Subject: Re: What must one do to avoid Gmail's retarded non-spam filtering?
We have proper A+PTR records on the edge MTAs, proper SPF records for the originating domain, proper Return-Path and other headers, and so on. There isn't anything that I can think of other than the content itself which would be abnormal, and obviously the content is repetitive and can't be changed much. Is there something obvious which we've missed?
What else goes out from that MTA? User forwarded mail? Mailing lists? random customer junk? It is a really good idea to separate your mail streams. These days it means separate IPs for users, transactions, forwards, and such, eventually different DKIM signatures will do the trick. R's, John
On 9/29/2010 11:48, Erik L wrote:
Thanks John. This was a common question that was asked off-list. That edge MTA is not used and has never been used by anything/anyone other than us. No customer mail flows or has flowed in or out via it ever.
As I mentioned in my follow-up post, the issue at this point is that the domain has been blacklisted. I can send an identical message from the same MTA, changing only the From header, and it will be delivered to Inbox. Only when the From header contains @caneris.com will the message be delivered to spam. Any changes to the MTA IP, content, headers, etc. don't have any effect.
Do you let customers have addresses under that domain? ~Seth
No ----- Original Message ----- From: "Seth Mattinen" <sethm@rollernet.us> To: nanog@nanog.org Sent: Wednesday, September 29, 2010 2:51:49 PM Subject: Re: What must one do to avoid Gmail's retarded non-spam filtering? On 9/29/2010 11:48, Erik L wrote:
Thanks John. This was a common question that was asked off-list. That edge MTA is not used and has never been used by anything/anyone other than us. No customer mail flows or has flowed in or out via it ever.
As I mentioned in my follow-up post, the issue at this point is that the domain has been blacklisted. I can send an identical message from the same MTA, changing only the From header, and it will be delivered to Inbox. Only when the From header contains @caneris.com will the message be delivered to spam. Any changes to the MTA IP, content, headers, etc. don't have any effect.
Do you let customers have addresses under that domain? ~Seth
As I mentioned in my follow-up post, the issue at this point is that the domain has been blacklisted.
Most peculiar. Since gmail is driven by algorithms, that would suggest that a lot of your recipients are reporting your mail as spam, which I can assure you people do, no matter how legitimate and opt-in it is. Do you have feedback loops set up? Google doesn't offer one, but AOL, Yahoo, Hotmail, and other large ISPs do, which will let you figure out what sorts of mail people are complaining about so you can perhaps send less of it. R's, John
Hello, I would also recommend to implement the list unsubscribe header, because google is supporting that kind of user feedback to senders. http://www.list-unsubscribe.com/ Regards, André André Görmer Senior Deliverability Manager eCircle P: +49 89 12009-762 | F: +49 89 12009-750 | E: a.goermer@ecircle.com Nymphenburger Str. 86, 80636 München Stay in touch Web: www.ecircle.com/de | Newsletter: www.ecircle.com/index.php?id=63&L=0 Für Hilfe mit dem eC-messenger wenden Sie sich bitte an unseren Support unter Tel +49 (0)89 120 09 600 oder support-de@ecircle.com Neuste Untersuchungen Ein unschlagbares Doppel: E-Mail-Marketing & Webanalyse Download Whitepaper: www.ecircle.com/index.php?id=61&L=0 eCircle GmbH, HRB 184 478, Handelsregister München, Geschäftsführung: Volker Wiewer (Vorsitzender), Alexander Meyer, Thomas Wilke, Vorsitzender des Aufsichtsrates: Dr. Arnold Bahlmann-----Ursprüngliche Nachricht----- Von: John R. Levine [mailto:johnl@iecc.com] Gesendet: Mittwoch, 29. September 2010 21:49 An: Erik L Cc: nanog@nanog.org Betreff: Re: What must one do to avoid Gmail's retarded non-spam filtering?
As I mentioned in my follow-up post, the issue at this point is that the domain has been blacklisted.
Most peculiar. Since gmail is driven by algorithms, that would suggest that a lot of your recipients are reporting your mail as spam, which I can assure you people do, no matter how legitimate and opt-in it is. Do you have feedback loops set up? Google doesn't offer one, but AOL, Yahoo, Hotmail, and other large ISPs do, which will let you figure out what sorts of mail people are complaining about so you can perhaps send less of it. R's, John
Can you please not use the word "retarded" in a pejorative sense? On Tue, Sep 28, 2010 at 3:15 PM, Erik L <erik_list@caneris.com> wrote:
I realize that this is somewhat OT, but I'm sure that others on the list encounter the same issues and that at least some folks might have useful comments.
An increasingly large number of our customers are using Gmail or Google Apps and almost all of our OSS/BSS mail is getting spam filtered by Google. Among others, these e-mails include invoices, order confirmations, payment notifications, customer portal logins, and tickets. Almost anything we send to customers on Google ends up in their spam folder. This results in a lot of calls and makes much of our automation pointless, never mind all the lost sales.
The problem is compounded by those who use mail clients and do not log in to the webmail at all, since they would never see the contents of the Google spam folder.
We have proper A+PTR records on the edge MTAs, proper SPF records for the originating domain, proper Return-Path and other headers, and so on. There isn't anything that I can think of other than the content itself which would be abnormal, and obviously the content is repetitive and can't be changed much. Is there something obvious which we've missed?
Aside from the following clearly impractical solutions, what can we do? 1. Asking everyone (including those we don't even know yet) to whitelist all of our addresses, to check their spam folders, and to click on "this is not spam" 2. Providing our own free e-mail service to everyone (including those we don't even know yet) and putting up "don't use Google" ads on all of our customer-facing systems
At least this isn't Hotmail where mail is just silently deleted with no NDR after it's accepted by their MTAs.
The call volume has been going up instead of down lately and it's gotten to the point where we're sending MTA log extracts to people to prove to them that we really did e-mail them.
Would greatly appreciate any advice.
Erik
On Sep 29, 2010, at 4:08 PM, Ryan Hayes wrote:
Can you please not use the word "retarded" in a pejorative sense?
The word "please" is probably not required, since using that word in this manner is prosecutable hate speech in some jurisdictions.
Le mercredi 29 septembre 2010 à 16:31 -0500, Daniel Seagraves a écrit :
On Sep 29, 2010, at 4:08 PM, Ryan Hayes wrote:
Can you please not use the word "retarded" in a pejorative sense?
The word "please" is probably not required, since using that word in this manner is prosecutable hate speech in some jurisdictions.
You're serious? :) mh
On Sep 29, 2010, at 2:31 PM, Daniel Seagraves wrote:
On Sep 29, 2010, at 4:08 PM, Ryan Hayes wrote:
Can you please not use the word "retarded" in a pejorative sense?
The word "please" is probably not required, since using that word in this manner is prosecutable hate speech in some jurisdictions.
Really? Where? Seems to me equating "retarded" used as an adjective to say, speech of the KKK seems... lame. Also, "retarded" was applied to software, which to date, has not been entered as a protected group. -j
In message <AANLkTikaiBkwc3R2ijKHpYhb=i+ACyn_ht7jgthtHRNz@mail.gmail.com>, Ryan Hayes <ryguillian@gmail.com> wrote:
Can you please not use the word "retarded" in a pejorative sense?
Obviously not a Colbert fan. http://www.huffingtonpost.com/2010/02/09/colbert-sarah-palin-is-a_n_454744.h... Regards, rfg P.S. It's satire.
participants (15)
-
André Görmer -
Benjamin Billon -
Daniel Seagraves -
Erik L -
James Downs -
Jeroen van Aart -
John Levine -
John R. Levine
-
Loránd Jakab -
Michael Hallgren -
Ronald F. Guilmette -
Ryan Hayes -
Seth Mattinen -
William Herrin -
William Pitcock