Recovering from spam resulting from compromised account
Hello, oh knowledgeable NANOG. I am the technical lead for network for Pixar. (Note: I am not the mail admin, he's on vacation.) Yesterday we had an account compromise that resulted in ~2.5M messages being sent through our two MTAs. I have acknowledged/closed the two SpamCop incidents, and mail is starting to flow, slowly, however we are still receiving bounces (some hard!) and I am looking for assistance in getting Pixar's IPs cleared from the blacklists. I was pointed to: http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a12.25.180.66 http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a12.25.180.94 Which shows we're still listed on Backscatterer and SPAM Cannibal. Also had reports that we're still seeing bounces to Gmail, Comcast and Yahoo accounts. What can we do to speed things along? We have a ticket open with Gmail folks since we have a studio who uses Gmail for Corporate mail. Any Comcast or Gmail SMTP contacts on NANOG that can help? Would love to get all out stuck mail out of these folks' MTAs. Or do we need to just remove ourselves from the last two blacklists at mxtoolbox? Thanks, David Sotnick -- Pixar Emeryville, CA
So - 1. backscatterer and spamcannibal are obscure blocklists nobody ever uses. Spamcannibal is actually quite reasonable about removals if you declare the issue fixed 2. Gmail, comcast etc have their own blocklist removal procedures - based on you contacting their postmaster teams. postmaster.comcast.net, etc etc. 3. MXToolbox is merely a search engine for various publicly available blocklists. Gmail etc blocks wont show up there because those dont get exposed outside the provider's servers .. if you get listed on gmail you know because you see your mail bounced or bulk foldered. --srs On Thu, Nov 22, 2012 at 7:23 AM, Dave Sotnick <sotnickd-nanog@ddv.com>wrote:
Hello, oh knowledgeable NANOG.
I am the technical lead for network for Pixar. (Note: I am not the mail admin, he's on vacation.) Yesterday we had an account compromise that resulted in ~2.5M messages being sent through our two MTAs.
I have acknowledged/closed the two SpamCop incidents, and mail is starting to flow, slowly, however we are still receiving bounces (some hard!) and I am looking for assistance in getting Pixar's IPs cleared from the blacklists.
I was pointed to:
http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a12.25.180.66 http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a12.25.180.94
Which shows we're still listed on Backscatterer and SPAM Cannibal.
Also had reports that we're still seeing bounces to Gmail, Comcast and Yahoo accounts.
What can we do to speed things along? We have a ticket open with Gmail folks since we have a studio who uses Gmail for Corporate mail. Any Comcast or Gmail SMTP contacts on NANOG that can help? Would love to get all out stuck mail out of these folks' MTAs.
Or do we need to just remove ourselves from the last two blacklists at mxtoolbox?
Thanks, David Sotnick -- Pixar Emeryville, CA
-- Suresh Ramasubramanian (ops.lists@gmail.com)
On Nov 21, 2012, at 8:53 PM, Dave Sotnick <sotnickd-nanog@ddv.com> wrote:
Also had reports that we're still seeing bounces to Gmail, Comcast and Yahoo accounts.
The best thing to do is to go ahead and look at the bounce messages from the various ISP's, and see if they have any instructions or URL's to contact. If you don't have any of those messages at hand, you can see the bounce codes in the logs of your mailserver. If you don't have any useful messages in the bounce code, then you can probably look at the site for each ISP, and google their postmaster group. Matthew Matthew Barr Technical Architect Snap Interactive mbarr@mbarr.net
Thanks Matthew. Sadly, most of the bounce responses have URLs that point you to a help page that doesn't have further contact information or just tells you to wait it out. e.g. http://postmaster.yahoo.com/421-ts03.html http://www.google.com/mail/help/bulk_mail.html I'll do the requisite digging and start contacting postmasters. -Dave On Wed, Nov 21, 2012 at 6:13 PM, Matthew Barr <mbarr@snap-interactive.com> wrote:
On Nov 21, 2012, at 8:53 PM, Dave Sotnick <sotnickd-nanog@ddv.com> wrote:
Also had reports that we're still seeing bounces to Gmail, Comcast and Yahoo accounts.
The best thing to do is to go ahead and look at the bounce messages from the various ISP's, and see if they have any instructions or URL's to contact.
If you don't have any of those messages at hand, you can see the bounce codes in the logs of your mailserver.
If you don't have any useful messages in the bounce code, then you can probably look at the site for each ISP, and google their postmaster group.
Matthew
Matthew Barr Technical Architect Snap Interactive mbarr@mbarr.net
Hi Dave, Try this page, linked from the google help page you referenced: https://support.google.com/mail/bin/answer.py?hl=en&answer=81126&rd=1 Hope that helps Andrew On 22.11.2012 13:29, Dave Sotnick wrote:
Thanks Matthew. Sadly, most of the bounce responses have URLs that point you to a help page that doesn't have further contact information or just tells you to wait it out.
e.g.
http://postmaster.yahoo.com/421-ts03.html http://www.google.com/mail/help/bulk_mail.html
I'll do the requisite digging and start contacting postmasters.
-Dave
On Wed, Nov 21, 2012 at 6:13 PM, Matthew Barr <mbarr@snap-interactive.com> wrote:
On Nov 21, 2012, at 8:53 PM, Dave Sotnick <sotnickd-nanog@ddv.com> wrote:
Also had reports that we're still seeing bounces to Gmail, Comcast and Yahoo accounts.
The best thing to do is to go ahead and look at the bounce messages from the various ISP's, and see if they have any instructions or URL's to contact.
If you don't have any of those messages at hand, you can see the bounce codes in the logs of your mailserver.
If you don't have any useful messages in the bounce code, then you can probably look at the site for each ISP, and google their postmaster group.
Matthew
Matthew Barr Technical Architect Snap Interactive mbarr@mbarr.net
Wait it out as in - you had better examine your mail queues and purge them of any of the spam that was sent and is still queued up. It'll still take a day or two after that's done for the blocks to subside. On Thu, Nov 22, 2012 at 7:59 AM, Dave Sotnick <sotnickd-nanog@ddv.com>wrote:
Thanks Matthew. Sadly, most of the bounce responses have URLs that point you to a help page that doesn't have further contact information or just tells you to wait it out.
e.g.
http://postmaster.yahoo.com/421-ts03.html http://www.google.com/mail/help/bulk_mail.html
I'll do the requisite digging and start contacting postmasters.
-Dave
On Wed, Nov 21, 2012 at 6:13 PM, Matthew Barr <mbarr@snap-interactive.com> wrote:
On Nov 21, 2012, at 8:53 PM, Dave Sotnick <sotnickd-nanog@ddv.com>
Also had reports that we're still seeing bounces to Gmail, Comcast and Yahoo accounts.
The best thing to do is to go ahead and look at the bounce messages from
wrote: the various ISP's, and see if they have any instructions or URL's to contact.
If you don't have any of those messages at hand, you can see the bounce
codes in the logs of your mailserver.
If you don't have any useful messages in the bounce code, then you can
probably look at the site for each ISP, and google their postmaster group.
Matthew
Matthew Barr Technical Architect Snap Interactive mbarr@mbarr.net
-- Suresh Ramasubramanian (ops.lists@gmail.com)
On 11/21/12, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
Wait it out as in - you had better examine your mail queues and purge them of any of the spam that was sent and is still queued up.
It'll still take a day or two after that's done for the blocks to subside.
The majority of blocking should in most cases, eventually clear up after spamming stops, and you can work out delisting with the common RBLs, using URLs in the bounce response; the general rule is 72 hours, after there is a complete stoppage of bad traffic, and you completed these steps: you wipe all bad messages from queues, make certain spam has completely stopped, ensure dilligent 24 hour monitoring, and then proper delisting is requested from any common blocklists that a lookup was available on. It may be impossible for you to clean out some blocklist entries, or you may have a limited number of "reset requests" available, that take effect after 24+ hours, E.g. CSI. For some blocklists, entries autoexpire after 7 days or longer and don't take manual requests, or some blocklists require a fee for delisting requests, and blocklist entries might otherwise be permanent. You can inspect bounces and raise the issues with blocking providers on a case-by-case basis; it is unlikely you reach someone at Google or Yahoo who will manually intervene. You can also lookup various Hosted spam filtering services, there are some large trusted providers, that will provide an outgoing spam filtering option, by using their servers as a smarthost, you offload mail deliverability issues to your service provider; in exchange, inbound/outbound spam filtering services typically charge something such as $12/mailbox. Changing your outgoing IP address of SMTP mail to your service providers, or rerouting mail towards servers blocking you, through a different local mail relay, may provide a temporary quick fix that is faster than waiting a few days until "spam extermination", on your current mail server is fully acknowledged.
On Thu, Nov 22, 2012 at 7:59 AM, Dave Sotnick <sotnickd-nanog@ddv.com>wrote:
Thanks Matthew. Sadly, most of the bounce responses have URLs that point you to a help page that doesn't have further contact information or just tells you to wait it out.
-- -JH
Hello again, I sincerely appreciate all the suggestions over the past week or so. We are mostly out of the woods. Yahoo is still blocking one of our MXs (12.25.180.94), despite repeated attempts to clear that IP. It appears as though no matter who we contact at Yahoo, they are all sending the same canned response: "While we cannot provide you with specific information, we encourage you to
review some of our recommended best practices for sending to Yahoo! Mail. For assistance with delivery issues to Yahoo! Mail, please visit the Yahoo! Postmaster help site. Your patience during this process is greatly appreciated. Thank you again for contacting Yahoo! Mail."
***If anyone knows of a human at Yahoo who might actually be able to assist, that would be much appreciated.*** We got our way out of this mess by writing to the major Postmasters, explaining the situation and then being patient while things cleared up. Gmail was the most responsive (surprise surprise), and once our mail queue was cleared of all queued SPAM and we _actually_ stopped sending messages, they automatically cleared our name without requiring any human intervention. Oh, and to add insult to injury, an IP address change at AT&T was preventing them from slaving our reverse DNS, which expired and caused a whole mess of further problems to our email. :-( Time to add some _external_ DNS health checks to our monitoring systems. Thanks again, Dave On Wed, Nov 21, 2012 at 5:53 PM, Dave Sotnick <sotnickd-nanog@ddv.com> wrote:
Hello, oh knowledgeable NANOG.
I am the technical lead for network for Pixar. (Note: I am not the mail admin, he's on vacation.) Yesterday we had an account compromise that resulted in ~2.5M messages being sent through our two MTAs.
I have acknowledged/closed the two SpamCop incidents, and mail is starting to flow, slowly, however we are still receiving bounces (some hard!) and I am looking for assistance in getting Pixar's IPs cleared from the blacklists.
I was pointed to:
http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a12.25.180.66 http://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a12.25.180.94
Which shows we're still listed on Backscatterer and SPAM Cannibal.
Also had reports that we're still seeing bounces to Gmail, Comcast and Yahoo accounts.
What can we do to speed things along? We have a ticket open with Gmail folks since we have a studio who uses Gmail for Corporate mail. Any Comcast or Gmail SMTP contacts on NANOG that can help? Would love to get all out stuck mail out of these folks' MTAs.
Or do we need to just remove ourselves from the last two blacklists at mxtoolbox?
Thanks, David Sotnick -- Pixar Emeryville, CA
participants (5)
-
Andrew Jones -
Dave Sotnick -
Jimmy Hess -
Matthew Barr -
Suresh Ramasubramanian